Updated Directories and Files (markdown)

Chris Lu 2018-06-07 16:47:11 -07:00
parent 77ff8fb1d9
commit 9ce0c8c72a

@ -45,6 +45,17 @@ We can take the following steps to map a full file path to the actual data block
1. (file_parent_directory, fileName) => meta data of attributes and a list of file_id. Implemented by "weed filer" server.
2. file_id => data_block. Implemented by default SeaweedFS master and volume servers.
#### Architecture
For reads:
1. Client Read File Metadata => Weed Filer => Weed Filer database (LevelDB, Cassandra, Redis, Mysql, Postgres, etc)
2. Client Read File Chunks => Weed Volume Servers
For writes:
1. Client write to Filer
2. Filer uploads data to Weed Volume Servers, and break the large files into chunks.
3. Filer writes the metadata and chunk information into Filer database.
#### Complexity
For one file retrieval, the (file_parent_directory, fileName)=>meta data lookup will be O(logN) for LSM tree or Btree implementations, where N is number of existing entries, or O(1) for Redis.
@ -84,9 +95,19 @@ weed mount -filer=localhost:8888 -dir=/some/existing/dir
Now you can browse/delete directories and files, read and write file as in local file system. For efficiency, only no more than 1000 sub directories and files under the same directory will be listed. To unmount, just shut it down.
### Weed Mount Architecture
For reads:
1. Client Reads File Metadata => Weed Filer => Weed Filer database (LevelDB, Cassandra, Redis, Mysql, Postgres, etc)
2. Client Reads File Chunks => Weed Volume Servers
For writes:
1. Client uploads data to Weed Volume Servers, and breaks the large files into chunks.
2. Client writes the metadata and chunk information into Filer and then into Filer database.
### Filer Scalability
Filer has two use case.
Filer has two use cases.
When filer is used directly to upload and download files, in addition to file meta data, the filer also need to process the file content during read and write. So it's a good idea to add multiple filer servers. Having an nginx server in front of the filer servers to load balance the requests would be a good idea.