gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Gateway to Remote Object Storage (markdown)

Chris Lu 2021-09-13 03:26:05 -07:00
parent 10761f0f0f
commit 664f70758b
1 changed files with 20 additions and 18 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

38
Gateway-to-Remote-Object-Storage.md

@ -1,25 +1,23 @@
# Intro
The [[Mount Remote Storage]] feature can mount one remote storage folder to one local folder.
```
# in "weed shell"
> remote.mount -dir=/xxx -remote=cloud1/bucket
```
The [[Mount Remote Storage]] feature can mount one remote storage folder in one single bucket.
However, you may have multiple buckets in one cloud storage. How to synchronize them automatically?
However,
* You may have multiple buckets in one cloud storage.
* You may want to create new buckets or remove old buckets.
How to synchronize them automatically?
# Design
The `weed filer.remote.sync -createBucketAt=cloud1` process has these features:
* The new buckets will be automatically mounted.
* If any buckets are created or deleted, the remote storage will also create or delete the corresponding buckets.
* Upload all changes under `/buckets` to remote storage.
The `weed filer.remote.sync -createBucketAt=cloud1` process does these jobs:
* Local new buckets will be automatically mounted to the remote storage specified in `-createBucketAt`
* Local deleted buckets will be automatically deleted in its remote storage.
* All local changes under `/buckets` are uploaded to its remote storage.
To do so, the `weed filer.remote.sync` process needs to know the default remote storage to create the new bucket.
# Setup Steps
# Usage
## Mount all existing remote buckets as local buckets
## 1. Mount all existing remote buckets as local buckets
If there are some existing buckets, run this to mount all of them as local buckets and synchronize their metadata to the local SeaweedFS cluster.
@ -28,7 +26,7 @@ If there are some existing buckets, run this to mount all of them as local bucke
> remote.mount.buckets -remote=cloud1 -apply
```
## Upload local changes in `/buckets`
## 2. Upload local changes in `/buckets`
Start a `weed filer.remote.sync` and let it run continuously.
@ -42,17 +40,21 @@ $ weed filer.remote.sync -createBucketAt=cloud1
synchronize /buckets, default new bucket creation in cloud1 ...
```
In some cloud storage vendor, the bucket names need to be unique. To address that, run it with `-createBucketWithRandomSuffix` option.
It will create bucket as `localBucketName-xxxx`, appending a random number as the suffix.
In some cloud storage vendor, the bucket names need to be unique. To address this, run it with `-createBucketWithRandomSuffix` option.
It will create buckets with name as `localBucketName-xxxx`, appending a random number as the suffix.
```
$ weed filer.remote.sync -createBucketAt=cloud1 -createBucketWithRandomSuffix
```
## Cache or uncache
## 3. Cache or uncache
Run these commands as needed to speed up access or to release disk spaces.
The basic implementation mechanism is the same as other mounted folders.
You may need to create a cronjob to run it periodically.
```
# in "weed shell"