gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Gateway to Remote Object Storage (markdown)

Chris Lu 2021-09-05 14:07:00 -07:00
parent 17ede02aa4
commit e8a63f0429
1 changed files with 22 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
Gateway-to-Remote-Object-Storage.md

@ -10,28 +10,46 @@ However, you may have multiple buckets in one cloud storage. How to synchronize
# Design # Design
The `weed filer.remote.sync` added a few of features: The `weed filer.remote.sync -createBucketAt=cloud1` process has these features:
* Upload all changes under `/buckets` to remote storage.
* If any buckets are created or deleted, the remote storage will also create or delete the corresponding buckets.
* The new buckets will be automatically mounted. * The new buckets will be automatically mounted.
* If any buckets are created or deleted, the remote storage will also create or delete the corresponding buckets.
* Upload all changes under `/buckets` to remote storage.
To do so, the `weed filer.remote.sync` process needs to know the default remote storage to create the new bucket. To do so, the `weed filer.remote.sync` process needs to know the default remote storage to create the new bucket.
# Usage # Usage
## Mount all existing remote buckets as local buckets ## Mount all existing remote buckets as local buckets
If there are some existing buckets, run this to mount all of them as local buckets and synchronize their metadata to the local SeaweedFS cluster.
``` ```
# in "weed shell" # in "weed shell"
> remote.mount.buckets -remote=cloud1 -apply > remote.mount.buckets -remote=cloud1 -apply
``` ```
## Upload local changes in `/buckets` ## Upload local changes in `/buckets`
This will also create new buckets in the specified remote storage.
Start a `weed filer.remote.sync` and let it run continuously.
If new buckets are created locally, this will also automatically create new buckets in the specified remote storage.
It is OK to pause it, and resume.
It is also OK to change the `-createBucketAt=xxx` value to a different one, since it only affects new bucket creation.
``` ```
$ weed filer.remote.sync -createBucketAt=cloud1 $ weed filer.remote.sync -createBucketAt=cloud1
synchronize /buckets, default new bucket creation in cloud1 ... synchronize /buckets, default new bucket creation in cloud1 ...
``` ```
In some cloud storage vendor, the bucket names need to be unique. To address that, run it with `-createBucketWithRandomSuffix` option.
It will create bucket as `localBucketName-xxxx`, appending a random number as the suffix.
```
$ weed filer.remote.sync -createBucketAt=cloud1 -createBucketWithRandomSuffix
```
## Cache or uncache ## Cache or uncache
The basic implementation mechanism is the same as other mounted folders. The basic implementation mechanism is the same as other mounted folders.
You may need to create a cronjob to run it periodically. You may need to create a cronjob to run it periodically.