gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Optimization (markdown)

Chris Lu 2020-04-14 20:20:26 -07:00
parent 0c578b7057
commit 6888665d77
1 changed files with 3 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
Optimization.md

@ -61,15 +61,11 @@ The SeaweedFS usually only open a few actual disk files. But the network file re
## Memory consumption
For volume servers, the memory consumption is tightly related to the number of files. For example, one 32G volume can easily have 1.5 million files if each file is only 20KB. To store the 1.5 million entries of meta data in memory, currently SeaweedFS consumes 36MB memory, about 24bytes per entry in memory. So if you allocate 64 volumes(2TB), you would need about 3GB memory. However, if the average file size is larger, say 200KB, only about 300MB memory is needed.
For each volume server, there are 2 things impacts the memory requirements.
SeaweedFS also has leveldb mode support, which reduces memory consumption even more.
By default, the volume server uses in memory index to achieve O(1) disk read. Roughly about 20 bytes is needed for each file. If one 30GB volume has 1 million files of averaged 30KB, the volume can cost 20MB memory to hold the index. You can also sue leveldb index to reduce memory consumption and speed up startup time. To use it, "weed server -volume.index=[memory|leveldb|leveldbMedium|leveldbLarge]", or "weed volume -index=[memory|leveldb|leveldbMedium|leveldbLarge]".
To use it, "weed server -volume.index=[memory|leveldb|leveldbMedium|leveldbLarge]", or "weed volume -index=[memory|leveldb|leveldbMedium|leveldbLarge]". You can switch between the 4 modes any time, as often as possible. If the files for leveldb is outdated or missing, they will be re-generated as needed.
To test the memory consumption, the leveldb index are created. There are 7 volumes in benchmark collection, each with about 1553K files. The server is restarted, then I start the benchmark tool to read lots of files.
For leveldb, server memory starts at 142,884KB, and stays at 179,340KB.
For in-memory, server memory starts at 368,152KB, and stays at 448,032KB.
Another aspect is the total file size that are read at the same time. If serving 100KB-sized file for 1000 concurrent reads, 100MB memory is needed.
## Insert with your own keys