From 121aba2878f0002833aae617e6f2224cc125e912 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 14 Aug 2021 07:55:06 -0700 Subject: [PATCH] Updated Remote Storage Architecture (markdown) --- Remote-Storage-Architecture.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/Remote-Storage-Architecture.md b/Remote-Storage-Architecture.md index 765a42c..9df0280 100644 --- a/Remote-Storage-Architecture.md +++ b/Remote-Storage-Architecture.md @@ -41,14 +41,18 @@ There are no format changes to the files on the cloud. The mounted files will be ## Mount Remote Storage The remote storage, e.g., AWS S3, can be [[configured|Configure Remote Storage]] and [[mounted|Mount Remote Storage]] directly to an empty folder in SeaweedFS. +``` +# in "weed shell" +> remote.mount -dir=xxx -remote=cloud1/bucket +``` On mount, all the metadata will be pulled down and cached to the local filer store. The metadata will be used for all metadata operations, such as listing, directory traversal, read file size, compare file modification time, etc, which will be free and fast as usual, without any API calls to the cloud. -## Cache/Uncache File Content +If the cloud data has any changes, just run `remote.mount -dir=xxx -remote=cloud1/bucket` again in `weed shell`. The local cache will get updated (except deletions). It is a light weight process and you can run it regularly. -A file has metadata and its content. +## Cache/Uncache File Content By default, the file content is [[cached|Cache Remote Storage]] to local volume servers on the first read. @@ -56,15 +60,13 @@ Sometimes you may want to fetch all file content for a set of files. But trying Here you can run command `remote.cache -dir=xxx` in `weed shell`, which will cache all files under the specified directory. -Correspondingly, you can also run `remote.uncache -dir=xxx` in `weed shell`, to avoid local duplicated storage. +Correspondingly, you can also run `remote.uncache -dir=xxx` in `weed shell`, to purge local cache. ## Write Back Cache -The cache is write back by the `weed filer.remote.sync` process. +The cache is write back by the `weed filer.remote.sync` process. This asynchronous write back will not slow down any local operations. -If not starting `weed filer.remote.sync`, the cache will be just read only. Changes are not forbidden, but any data changes will not be propagated back to the cloud. - -The asynchronous write back will not slow down any local operations. +If not starting `weed filer.remote.sync`, the data changes will not be propagated back to the cloud. # Possible Use Cases