gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Data Structure for Large Files (markdown)

Chris Lu 2020-07-20 22:52:46 -07:00
parent c6751e7cd4
commit 871cbbb15d
1 changed files with 5 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
Data-Structure-for-Large-Files.md

@ -6,7 +6,7 @@ What are considered small files and what are large files? It does not have a cle
In SeaweedFS, the small files are defined as any file that can be saved in one chunk of data. The chunk size limit is not a fixed number, but flexible according to the hardware or the requirements. It can be anywhere ranging from 1MB ~ 10MB or a bit higher.
We will assume it is 4MB for following discussion.
We will assume it is 8MB for following discussion.
# Medium Files
@ -14,16 +14,16 @@ For larger files, usually the files are split into chunks. Each chunk info is re
A reasonable key value store can store about 1000 ~ 10000 pieces of chunk info, which is about 40KB ~ 400KB.
Since each chunk is about 4MB, these 1000~10000 pieces of chunk info can address files up to 4GB ~ 40GB in size efficiently.
Since each chunk is about 8MB, these 1000~10000 pieces of chunk info can address files up to 8GB ~ 80GB in size efficiently.
# Super Large File
For even larger files, the meta data, which is the list of chunk info, can grow big. One 400GB file can have a 4MB meta data. This would not scale.
For even larger files, the meta data, which is the list of chunk info, can grow big. One 800GB file can have a 4MB meta data. This would not scale.
SeaweedFS adds a manifest chunk to hold 1000 pieces of chunk info. This manifest chunk is stored together with the raw data on volume servers, which greatly reduces the storage load on key value stores and also reduces the access time.
For example, one super large file with 1000 manifest chunks is still 400KB in meta data in the key value store, but it can address a file of 4MB * 1000 * 1000 = 4TB in size.
For example, one super large file with 1000 manifest chunks is still 400KB in meta data in the key value store, but it can address a file of 8MB * 1000 * 1000 = 8TB in size.
This manifest chunk can also be easily implemented recursively, which can leads to even larger file size limit. But given the size limit is already very high and is beyond most common use cases, we will add the recursion later when really necessary.
The 4TB for one file is not a real limit, but only used for this discussion. However, if you have the need for files more than 4TB, please prepare some consulting fee and we will help you. :) Or you probably should already split the large files into smaller files.
The 8TB for one file is not a real limit, but only used for this discussion. However, if you have the need for files more than 4TB, please prepare some consulting fee and we will help you. :) Or you probably should already split the large files into smaller files.