From e8a63f0429514cdbc867a9f503812d9b6973fc85 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sun, 5 Sep 2021 14:07:00 -0700 Subject: [PATCH] Updated Gateway to Remote Object Storage (markdown) --- Gateway-to-Remote-Object-Storage.md | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/Gateway-to-Remote-Object-Storage.md b/Gateway-to-Remote-Object-Storage.md index ea2c770..4bf91ac 100644 --- a/Gateway-to-Remote-Object-Storage.md +++ b/Gateway-to-Remote-Object-Storage.md @@ -10,28 +10,46 @@ However, you may have multiple buckets in one cloud storage. How to synchronize # Design -The `weed filer.remote.sync` added a few of features: -* Upload all changes under `/buckets` to remote storage. -* If any buckets are created or deleted, the remote storage will also create or delete the corresponding buckets. +The `weed filer.remote.sync -createBucketAt=cloud1` process has these features: * The new buckets will be automatically mounted. +* If any buckets are created or deleted, the remote storage will also create or delete the corresponding buckets. +* Upload all changes under `/buckets` to remote storage. To do so, the `weed filer.remote.sync` process needs to know the default remote storage to create the new bucket. # Usage + ## Mount all existing remote buckets as local buckets +If there are some existing buckets, run this to mount all of them as local buckets and synchronize their metadata to the local SeaweedFS cluster. + ``` # in "weed shell" > remote.mount.buckets -remote=cloud1 -apply ``` ## Upload local changes in `/buckets` -This will also create new buckets in the specified remote storage. + +Start a `weed filer.remote.sync` and let it run continuously. + +If new buckets are created locally, this will also automatically create new buckets in the specified remote storage. + +It is OK to pause it, and resume. +It is also OK to change the `-createBucketAt=xxx` value to a different one, since it only affects new bucket creation. + ``` $ weed filer.remote.sync -createBucketAt=cloud1 synchronize /buckets, default new bucket creation in cloud1 ... ``` +In some cloud storage vendor, the bucket names need to be unique. To address that, run it with `-createBucketWithRandomSuffix` option. +It will create bucket as `localBucketName-xxxx`, appending a random number as the suffix. +``` +$ weed filer.remote.sync -createBucketAt=cloud1 -createBucketWithRandomSuffix +``` + + + ## Cache or uncache The basic implementation mechanism is the same as other mounted folders. You may need to create a cronjob to run it periodically.