From 9ce0c8c72a615944dbd70dc054b1d8a4ec678a7b Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 7 Jun 2018 16:47:11 -0700 Subject: [PATCH] Updated Directories and Files (markdown) --- Directories-and-Files.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/Directories-and-Files.md b/Directories-and-Files.md index 953d842..254f473 100644 --- a/Directories-and-Files.md +++ b/Directories-and-Files.md @@ -45,6 +45,17 @@ We can take the following steps to map a full file path to the actual data block 1. (file_parent_directory, fileName) => meta data of attributes and a list of file_id. Implemented by "weed filer" server. 2. file_id => data_block. Implemented by default SeaweedFS master and volume servers. +#### Architecture + +For reads: +1. Client Read File Metadata => Weed Filer => Weed Filer database (LevelDB, Cassandra, Redis, Mysql, Postgres, etc) +2. Client Read File Chunks => Weed Volume Servers + +For writes: +1. Client write to Filer +2. Filer uploads data to Weed Volume Servers, and break the large files into chunks. +3. Filer writes the metadata and chunk information into Filer database. + #### Complexity For one file retrieval, the (file_parent_directory, fileName)=>meta data lookup will be O(logN) for LSM tree or Btree implementations, where N is number of existing entries, or O(1) for Redis. @@ -84,9 +95,19 @@ weed mount -filer=localhost:8888 -dir=/some/existing/dir Now you can browse/delete directories and files, read and write file as in local file system. For efficiency, only no more than 1000 sub directories and files under the same directory will be listed. To unmount, just shut it down. +### Weed Mount Architecture + +For reads: +1. Client Reads File Metadata => Weed Filer => Weed Filer database (LevelDB, Cassandra, Redis, Mysql, Postgres, etc) +2. Client Reads File Chunks => Weed Volume Servers + +For writes: +1. Client uploads data to Weed Volume Servers, and breaks the large files into chunks. +2. Client writes the metadata and chunk information into Filer and then into Filer database. + ### Filer Scalability -Filer has two use case. +Filer has two use cases. When filer is used directly to upload and download files, in addition to file meta data, the filer also need to process the file content during read and write. So it's a good idea to add multiple filer servers. Having an nginx server in front of the filer servers to load balance the requests would be a good idea.