Created Super Large Directories (markdown)

Super-Large-Directories.md

@@ -0,0 +1,47 @@
+# Why this is needed?
+
+If one super large directory has way too many files or sub folders, the file listing itself can be a challenge.
+
+For example, for Cassandra filer store, all the children in a directory are physically stored in one Cassandra data node. This is fine for most cases. However, if there are billions of child entries for one directory, the data node would not be able to query or even store the child list.
+
+This is actually a common case when user name, id, or UUID are used as sub folder or file names. We need a way to spread the data to all data nodes.
+
+# How it works?
+
+This is currently only implemented in Cassandra. In Cassandra, each entry has this schema:
+```
+  CREATE TABLE filemeta (
+     directory varchar,
+     name varchar,
+     meta blob,
+     PRIMARY KEY (directory, name)
+  ) WITH CLUSTERING ORDER BY (name ASC);
+```
+The directory is the clustering key. So the entries with the same directory is clustered to the same data node.
+ 
+For super large directories, the directory is hashed and combined together with name as `<directory hash, name>`, which is used as the partitioning key. This ensures the directory children are evenly distributed to all data nodes.
+
+## The Downside
+
+The consequences are:
+
+* The directory listing for this folder is not supported.
+* The filer meta dada import and export for this folder is not supported. You can still do it for specific child folders though.
+* Once this is configured, it can not be changed back easily. You will need to write code to iterate all sub entries for that.
+
+# How to configure it?
+
+In `filer.toml` for Cassandra, there is an option `superLargeDirectories`. For example, if you will have a lot of user data under `/home/users`
+
+```
+[cassandra]
+...
+superLargeDirectories = [
+  "/home/user",
+]
+
+```
+
+This is assuming the `/home/user` is a new folder. 
+
+As you can see, it supports multiple super large directories. However, never change or remove the entries in `superLargeDirectories` or the data will be lost!