gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Filer Redis Setup (markdown)

Chris Lu 2021-10-09 06:45:30 -07:00
parent e5e01e480a
commit 058f5a22a5
1 changed files with 8 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

11
Filer-Redis-Setup.md

@ -7,11 +7,11 @@ Since Redis is a key-value store, the directory list could not be efficiently it
<directory_path>: <sorted set of names> <directory_path>: <sorted set of names>
``` ```
The sorted set in Redis is fairly efficient. Storing 100K ~ 1 million entries is still fairly fast. In most of the cases, your use case should work very well with a directory this size. The sorted set in Redis is fairly efficient. Storing 100K ~ 1 million entries is still fairly fast. In most of the cases, it should work very well with large directories.
# Super Large Directories # Super Large Directories
In some cases, the directory may have 10s of millions or billions of files or sub directories. The directory list would be too big to fit into one Redis entry. For `redis2`, you can specify which folder is super large, and avoid storing this fat list of names, but giving up the directory listing capability. In some cases, the directory may have 10s of millions or billions of files or sub directories. The directory list would be too big to fit into one Redis entry in one Redis instance. For `redis2`, you can specify which folder is super large, and avoid storing this fat list of names, but giving up the capability to list a directory.
This is where `redis3` can help. The internal data structure is: This is where `redis3` can help. The internal data structure is:
@ -23,4 +23,9 @@ This is where `redis3` can help. The internal data structure is:
``` ```
The directory list is stored as a skip list, and the child names are spread into the list items. The directory list is stored as a skip list, and the child names are spread into the list items.
This prevents each Redis entry from being too large and slower to access. This prevents each directory entry from being too large and slower to access.
Compared to `redis2`, there are extra cost to maintain this list:
* Adding or deleting needs one additional lock operation.
* Updating an entry needs to takes O(log(N)) times to access the skip list item first.
So no need to jump to `redis3` in a hurry.