gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Gateway to Remote Object Storage (markdown)

Chris Lu 2021-09-13 03:26:05 -07:00
parent 10761f0f0f
commit 664f70758b
1 changed files with 20 additions and 18 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

38
Gateway-to-Remote-Object-Storage.md

@ -1,25 +1,23 @@
# Intro # Intro
The [[Mount Remote Storage]] feature can mount one remote storage folder to one local folder. The [[Mount Remote Storage]] feature can mount one remote storage folder in one single bucket.
```
# in "weed shell"
> remote.mount -dir=/xxx -remote=cloud1/bucket
```
However, you may have multiple buckets in one cloud storage. How to synchronize them automatically? However,
* You may have multiple buckets in one cloud storage.
* You may want to create new buckets or remove old buckets.
How to synchronize them automatically?
# Design # Design
The `weed filer.remote.sync -createBucketAt=cloud1` process has these features: The `weed filer.remote.sync -createBucketAt=cloud1` process does these jobs:
* The new buckets will be automatically mounted. * Local new buckets will be automatically mounted to the remote storage specified in `-createBucketAt`
* If any buckets are created or deleted, the remote storage will also create or delete the corresponding buckets. * Local deleted buckets will be automatically deleted in its remote storage.
* Upload all changes under `/buckets` to remote storage. * All local changes under `/buckets` are uploaded to its remote storage.
To do so, the `weed filer.remote.sync` process needs to know the default remote storage to create the new bucket. # Setup Steps
# Usage ## 1. Mount all existing remote buckets as local buckets
## Mount all existing remote buckets as local buckets
If there are some existing buckets, run this to mount all of them as local buckets and synchronize their metadata to the local SeaweedFS cluster. If there are some existing buckets, run this to mount all of them as local buckets and synchronize their metadata to the local SeaweedFS cluster.
@ -28,7 +26,7 @@ If there are some existing buckets, run this to mount all of them as local bucke
> remote.mount.buckets -remote=cloud1 -apply > remote.mount.buckets -remote=cloud1 -apply
``` ```
## Upload local changes in `/buckets` ## 2. Upload local changes in `/buckets`
Start a `weed filer.remote.sync` and let it run continuously. Start a `weed filer.remote.sync` and let it run continuously.
@ -42,17 +40,21 @@ $ weed filer.remote.sync -createBucketAt=cloud1
synchronize /buckets, default new bucket creation in cloud1 ... synchronize /buckets, default new bucket creation in cloud1 ...
``` ```
In some cloud storage vendor, the bucket names need to be unique. To address that, run it with `-createBucketWithRandomSuffix` option. In some cloud storage vendor, the bucket names need to be unique. To address this, run it with `-createBucketWithRandomSuffix` option.
It will create bucket as `localBucketName-xxxx`, appending a random number as the suffix. It will create buckets with name as `localBucketName-xxxx`, appending a random number as the suffix.
``` ```
$ weed filer.remote.sync -createBucketAt=cloud1 -createBucketWithRandomSuffix $ weed filer.remote.sync -createBucketAt=cloud1 -createBucketWithRandomSuffix
``` ```
## Cache or uncache ## 3. Cache or uncache
Run these commands as needed to speed up access or to release disk spaces.
The basic implementation mechanism is the same as other mounted folders. The basic implementation mechanism is the same as other mounted folders.
You may need to create a cronjob to run it periodically. You may need to create a cronjob to run it periodically.
``` ```
# in "weed shell" # in "weed shell"