gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

docs

Chris Lu 2017-05-26 23:02:11 -07:00
parent 37812844ea
commit d9de7b2280
2 changed files with 8 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
Change-List.md

@ -2,9 +2,11 @@
This file contains list of recent changes, important features, usage changes, data format changes, etc. Do read this if you upgrade.
## v0.70 beta
## v0.75
1. Support btree mode, in addition to in-memory/leveldb/boltdb modes, for less memory with customized id.
## v0.70
1. Add "weed backup" command to backup a volume incrementally.
2.
## v0.69

6
Optimization.md

@ -48,9 +48,9 @@ You can also manually gzip content before submission. If you do so, make sure th
For volume servers, the memory consumption is tightly related to the number of files. For example, one 32G volume can easily have 1.5 million files if each file is only 20KB. To store the 1.5 million entries of meta data in memory, currently SeaweedFS consumes 36MB memory, about 24bytes per entry in memory. So if you allocate 64 volumes(2TB), you would need 2~3GB memory. However, if the average file size is larger, say 200KB, only 200~300MB memory is needed.
SeaweedFS also has leveldb and boltdb support, which reduces memory consumption even more.
SeaweedFS also has leveldb, boltdb, and btree mode support, which reduces memory consumption even more.
To use it, "weed server -volume.index=[memory|leveldb|boltdb]", or "weed volume -index=[memory|leveldb|boltdb]". You can switch between the 3 modes any time, as often as possible. If the files for leveldb or boltdb is outdated or missing, they will be re-generated as needed.
To use it, "weed server -volume.index=[memory|leveldb|boltdb|btree]", or "weed volume -index=[memory|leveldb|boltdb|btree]". You can switch between the 4 modes any time, as often as possible. If the files for leveldb or boltdb is outdated or missing, they will be re-generated as needed.
boltdb is fairly slow to write, about 6 minutes for recreating index for 1553934 files. Boltdb loads 1,553,934 x 16 = 24,862,944bytes from disk, and generate the boltdb as large as 134,217,728 bytes in 6 minutes.
To compare, leveldb recreates index as large as 27,188,148 bytes in 8 seconds.
@ -65,6 +65,8 @@ For boltdb, the write is about 4.1MB/s, 4.1K files/s
For leveldb, the writes is about 10.4MB/s, 10.4K files/s
For in-memory, it is a tiny bit faster, not statistically different. But I am using SSD, and os buffer cache also affect the numbers. So your results may be different.
Btree mode is added in v0.75, to optimize memory for out-of-order customized file key. Btree mode can cost more memory for normal file key assigned by SeaweedFS master, but are usually more efficient than customized file key. Please test for your cases.
Note: BoltDB has a limit that the max db size is 256MB on 32bit systems.
## Insert with your own keys