gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Erasure coding for warm storage (markdown)

Chris Lu 2019-06-10 15:21:11 -07:00
parent 7fb8617100
commit 12ba22aa79
1 changed files with 6 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
Erasure-coding-for-warm-storage.md → Erasure-Coding-for-warm-storage.md

@ -1,9 +1,13 @@
Warm data are accessed less frequently. To store them more efficiently, you can enable erasure coding.
## Benefit
* SeaweedFS implemented RS(10,4), which allows loss of 4 shards of data with 1.4x data size. Compared to replicating data 5 times to achieve the same robustness, it saves 3.6x disk space.
* SeaweedFS uses continuous 1GB block layout with 1MB block sizes for edge cases, optimized for both small file reads and storage efficiency.
* Storage Efficiency: SeaweedFS implemented RS(10,4), which allows loss of 4 shards of data with 1.4x data size. Compared to replicating data 5 times to achieve the same robustness, it saves 3.6x disk space.
* Fast read speed: SeaweedFS uses continuous 1GB block layout with 1MB block sizes for edge cases, optimized for both small file reads and storage efficiency.
* High availability: If up to 4 shards are down, the data is still accessible with reasonable speed.
* Efficient memory usage and faster startup time. The volume server does not load index data into memory.
* Rack-Aware data placement to minimize impact of volume server and rack failures.
* No requirement for large amount of servers. SeaweedFS manage erasure coding data via volumes. If the number of servers is less than 4, this can protect against hard drive failures. If the number of servers is greater than 4, this can protect against server failures. If the number of racks is greater than 4, this can protect against rack failures.
* Optimized for small files: there are no file size requirement for EC to take effect.
## How to enable it?
Run `weed scaffold -conf=master` to generate a `master.toml` file, put it in current directory, `~/.seaweedfs/`, or `/etc/seaweedfs/`.