diff --git a/Optimization.md b/Optimization.md index 726518c..91f816b 100644 --- a/Optimization.md +++ b/Optimization.md @@ -61,15 +61,11 @@ The SeaweedFS usually only open a few actual disk files. But the network file re ## Memory consumption -For volume servers, the memory consumption is tightly related to the number of files. For example, one 32G volume can easily have 1.5 million files if each file is only 20KB. To store the 1.5 million entries of meta data in memory, currently SeaweedFS consumes 36MB memory, about 24bytes per entry in memory. So if you allocate 64 volumes(2TB), you would need about 3GB memory. However, if the average file size is larger, say 200KB, only about 300MB memory is needed. +For each volume server, there are 2 things impacts the memory requirements. -SeaweedFS also has leveldb mode support, which reduces memory consumption even more. +By default, the volume server uses in memory index to achieve O(1) disk read. Roughly about 20 bytes is needed for each file. If one 30GB volume has 1 million files of averaged 30KB, the volume can cost 20MB memory to hold the index. You can also sue leveldb index to reduce memory consumption and speed up startup time. To use it, "weed server -volume.index=[memory|leveldb|leveldbMedium|leveldbLarge]", or "weed volume -index=[memory|leveldb|leveldbMedium|leveldbLarge]". -To use it, "weed server -volume.index=[memory|leveldb|leveldbMedium|leveldbLarge]", or "weed volume -index=[memory|leveldb|leveldbMedium|leveldbLarge]". You can switch between the 4 modes any time, as often as possible. If the files for leveldb is outdated or missing, they will be re-generated as needed. - -To test the memory consumption, the leveldb index are created. There are 7 volumes in benchmark collection, each with about 1553K files. The server is restarted, then I start the benchmark tool to read lots of files. -For leveldb, server memory starts at 142,884KB, and stays at 179,340KB. -For in-memory, server memory starts at 368,152KB, and stays at 448,032KB. +Another aspect is the total file size that are read at the same time. If serving 100KB-sized file for 1000 concurrent reads, 100MB memory is needed. ## Insert with your own keys