From e5e01e480a73c0ab48715ab97dc0de107d964c26 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 9 Oct 2021 06:30:09 -0700 Subject: [PATCH] Created Filer Redis Setup (markdown) --- Filer-Redis-Setup.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 Filer-Redis-Setup.md diff --git a/Filer-Redis-Setup.md b/Filer-Redis-Setup.md new file mode 100644 index 0000000..778416c --- /dev/null +++ b/Filer-Redis-Setup.md @@ -0,0 +1,26 @@ +Currently there are two Redis implementations, i.g., `redis2` and `redis3`. What is the difference? + +# Storing directory list + +Since Redis is a key-value store, the directory list could not be efficiently iterated by scanning the keys. So the directory list is stored as a [sorted set](https://redis.io/topics/data-types#sorted-sets) in `redis2`: +``` + : +``` + +The sorted set in Redis is fairly efficient. Storing 100K ~ 1 million entries is still fairly fast. In most of the cases, your use case should work very well with a directory this size. + +# Super Large Directories + +In some cases, the directory may have 10s of millions or billions of files or sub directories. The directory list would be too big to fit into one Redis entry. For `redis2`, you can specify which folder is super large, and avoid storing this fat list of names, but giving up the directory listing capability. + +This is where `redis3` can help. The internal data structure is: + +``` + : + : + : + ... +``` +The directory list is stored as a skip list, and the child names are spread into the list items. + +This prevents each Redis entry from being too large and slower to access.