From 058f5a22a55c1417870b0e865e3c42111f203627 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 9 Oct 2021 06:45:30 -0700 Subject: [PATCH] Updated Filer Redis Setup (markdown) --- Filer-Redis-Setup.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/Filer-Redis-Setup.md b/Filer-Redis-Setup.md index 778416c..cd12544 100644 --- a/Filer-Redis-Setup.md +++ b/Filer-Redis-Setup.md @@ -7,11 +7,11 @@ Since Redis is a key-value store, the directory list could not be efficiently it : ``` -The sorted set in Redis is fairly efficient. Storing 100K ~ 1 million entries is still fairly fast. In most of the cases, your use case should work very well with a directory this size. +The sorted set in Redis is fairly efficient. Storing 100K ~ 1 million entries is still fairly fast. In most of the cases, it should work very well with large directories. # Super Large Directories -In some cases, the directory may have 10s of millions or billions of files or sub directories. The directory list would be too big to fit into one Redis entry. For `redis2`, you can specify which folder is super large, and avoid storing this fat list of names, but giving up the directory listing capability. +In some cases, the directory may have 10s of millions or billions of files or sub directories. The directory list would be too big to fit into one Redis entry in one Redis instance. For `redis2`, you can specify which folder is super large, and avoid storing this fat list of names, but giving up the capability to list a directory. This is where `redis3` can help. The internal data structure is: @@ -23,4 +23,9 @@ This is where `redis3` can help. The internal data structure is: ``` The directory list is stored as a skip list, and the child names are spread into the list items. -This prevents each Redis entry from being too large and slower to access. +This prevents each directory entry from being too large and slower to access. + +Compared to `redis2`, there are extra cost to maintain this list: +* Adding or deleting needs one additional lock operation. +* Updating an entry needs to takes O(log(N)) times to access the skip list item first. +So no need to jump to `redis3` in a hurry.