From 637a22c409a5c98a1c79e5c08486e9880fc174d9 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Wed, 3 Mar 2021 22:07:25 -0800 Subject: [PATCH] Created Async Filer Metadata Backup (markdown) --- Async-Filer-Metadata-Backup.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 Async-Filer-Metadata-Backup.md diff --git a/Async-Filer-Metadata-Backup.md b/Async-Filer-Metadata-Backup.md new file mode 100644 index 0000000..1df538d --- /dev/null +++ b/Async-Filer-Metadata-Backup.md @@ -0,0 +1,25 @@ +It is desirable to have a copy of the filer metadata. However, it is not always easy to setup replications for filer metadata store: +* The store does support stream replication. +* The store is embedded leveldb, RocksDB, etc. +* It is just complicated. + +There are two ways to ensure a copy of the meta data. + +# Separate Filer instances connected with "-peers" option + +With `-peers` option, if the filers are not sharing the same filer metadata store, the metadata changes are asynchronously propagated to all peers. + +See https://github.com/chrislusf/seaweedfs/wiki/Filer-Store-Replication#file-store-replication + +# Async filer metadata backup to a remote store + +We can also continuously backup filer metadata to a remote store, without running a filer instance. +``` + Metadata Changes + Filer --------------------> `weed filer.meta.backup` ----> Remote Store(LevelDB/RocksDB/ + Mysql/Postgres/ + Redis/Cassanra/...) +``` +Just need to configure the remote store the same way as `filer.toml`. + +`weed filer.meta.backup` can be stopped and resumed. The metadata backup progress is tracked in the remote store itself. So you can pause/resume any time, and even resume from a separate machine.