gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

rename to cloud drive

Chris Lu 2021-08-20 23:16:43 -07:00
parent 8234c91ace
commit 118cd68483
5 changed files with 25 additions and 25 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
Cloud-Cache-Benefits.md

@ -50,7 +50,7 @@ However, how to make SeaweedFS work with data already on cloud?
* Can transparently switch to different cloud storage vendors. * Can transparently switch to different cloud storage vendors.
* Can detach from the cloud storage if decided to move off cloud. * Can detach from the cloud storage if decided to move off cloud.
# What SeaweedFS Cloud Cache is not? # What SeaweedFS Cloud Drive is not?
* It is not a proxy. Proxy is not ideal for low-latency read or write operations. * It is not a proxy. Proxy is not ideal for low-latency read or write operations.
* Its cache size is not limited. With unlimited size, all file content can be localized. So the read can be fast even on the first attempt, without worrying cached items are evicted due to capacity. * Its cache size is not limited. With unlimited size, all file content can be localized. So the read can be fast even on the first attempt, without worrying cached items are evicted due to capacity.
@ -58,11 +58,11 @@ However, how to make SeaweedFS work with data already on cloud?
* It does not change file storage layout in the cloud storage. Other tools can access the files on the cloud storage as usual. * It does not change file storage layout in the cloud storage. Other tools can access the files on the cloud storage as usual.
* It does not need to link to the original cloud storage. After caching all the file content, you can choose to detach from the cloud storage. * It does not need to link to the original cloud storage. After caching all the file content, you can choose to detach from the cloud storage.
# Why SeaweedFS Cloud Cache? # Why SeaweedFS Cloud Drive?
There are other products also cache S3 data. What makes SeaweedFS Cloud Cache better? There are other products also cache S3 data. What makes SeaweedFS Cloud Drive better?
SeaweedFS Cloud Cache has these unique characteristics: SeaweedFS Cloud Drive has these unique characteristics:
* Scalable Capacity * Scalable Capacity
* Easy to add more capacity. * Easy to add more capacity.
* Disaggregated storage. * Disaggregated storage.
@ -81,8 +81,8 @@ SeaweedFS Cloud Cache has these unique characteristics:
* Problem * Problem
* Training jobs need to repeatedly visit a large set of files. * Training jobs need to repeatedly visit a large set of files.
* The randomized access pattern is hard for caching. * The randomized access pattern is hard for caching.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Drive
* Users can explicitly ask SeaweedFS Cloud Cache to cache one whole folder. * Users can explicitly ask SeaweedFS Cloud Drive to cache one whole folder.
* Increase training speed and reduce API cost and network cost. * Increase training speed and reduce API cost and network cost.
* Users can access data with FUSE mounted folders. * Users can access data with FUSE mounted folders.
## Data Hoarding ## Data Hoarding
@ -91,38 +91,38 @@ SeaweedFS Cloud Cache has these unique characteristics:
* With cloud capacity, reliability, and storage tiering, saving data files there may be a good idea. * With cloud capacity, reliability, and storage tiering, saving data files there may be a good idea.
* Accessing cloud data is slow and costly. * Accessing cloud data is slow and costly.
* Recently uploaded files very likely need to be accessed again. * Recently uploaded files very likely need to be accessed again.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Drive
* Users can explicitly ask SeaweedFS Cloud Cache to uncache by file age. * Users can explicitly ask SeaweedFS Cloud Drive to uncache by file age.
* Users can also choose to never uncache, basically treating cloud copy as a backup. * Users can also choose to never uncache, basically treating cloud copy as a backup.
## Big Data ## Big Data
* Problem * Problem
* Run MapReduce, Spark, and Flink jobs on cloud data is slow due to metadata operations. * Run MapReduce, Spark, and Flink jobs on cloud data is slow due to metadata operations.
* Repeated data access increases unnecessary cost. * Repeated data access increases unnecessary cost.
* May need to work with the cloud ecosystem. * May need to work with the cloud ecosystem.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Drive
* Avoiding slow cloud storage metadata access. * Avoiding slow cloud storage metadata access.
* Access data only once. * Access data only once.
* Write back data to work with cloud ecosystems. * Write back data to work with cloud ecosystems.
## Cloud Storage Vendor Agnostic ## Cloud Storage Vendor Agnostic
* Problem * Problem
* Different datasets may need to be on different vendors, based on access pattern, latency, cost, etc. * Different datasets may need to be on different vendors, based on access pattern, latency, cost, etc.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Drive
* Transparently switch to from one vendor to another. * Transparently switch to from one vendor to another.
## Switch S3 Compatible Storage Vendors ## Switch S3 Compatible Storage Vendors
* Problem * Problem
* Other S3 compatible vendors are known to have lesser flexibility, scalability and performance as SeaweedFS. * Other S3 compatible vendors are known to have lesser flexibility, scalability and performance as SeaweedFS.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Drive
* Compare SeaweedFS with the vendor over a long period of time. * Compare SeaweedFS with the vendor over a long period of time.
* Transparently switch away if SeaweedFS works better. * Transparently switch away if SeaweedFS works better.
## Move Off Cloud ## Move Off Cloud
* Problem * Problem
* Cloud storage is costly! * Cloud storage is costly!
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Drive
* Help to transition between on-cloud to off-cloud. * Help to transition between on-cloud to off-cloud.
* When you are happy with it, just stop the write back process (and cancel the monthly payment to the cloud vendor!). * When you are happy with it, just stop the write back process (and cancel the monthly payment to the cloud vendor!).
## Support multiple access methods. ## Support multiple access methods.
* Problem * Problem
* You may need to access cloud data by HDFS, or HTTP, or S3 API, or WebDav, or FUSE Mount. * You may need to access cloud data by HDFS, or HTTP, or S3 API, or WebDav, or FUSE Mount.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Drive
* Multiple ways to access remote storage. * Multiple ways to access remote storage.

2
Cloud-Cache-Quick-Setup.md

@ -1,5 +1,5 @@
To users not familiar with SeaweedFS, there seems many things to learn. To users not familiar with SeaweedFS, there seems many things to learn.
But for SeaweedFS Cloud Cache, the setup is easy. But for SeaweedFS Cloud Drive, the setup is easy.
# Setup a simple SeaweedFS cluster # Setup a simple SeaweedFS cluster

12
Cloud-Drive-Architecture.md

@ -1,8 +1,8 @@
# Introduction # Introduction
For normal computers, Google or Dropbox drive can be mounted as a network folder. The remote files are cached to local disks and automatically synchronized with the cloud. So it is fast to access. Similarly, SeaweedFS Cloud Cache can mount cloud storage as a special folder to the local cluster. The cloud files can be cached to the local cluster and automatically synchronized with the cloud. For normal computers, Google or Dropbox drive can be mounted as a network folder. The remote files are cached to local disks and automatically synchronized with the cloud. So it is fast to access. Similarly, SeaweedFS Cloud Drive can mount cloud storage as a special folder to the local cluster. The cloud files can be cached to the local cluster and automatically synchronized with the cloud.
| | SeaweedFS Cloud Cache | Google / Dropbox Drive | | | SeaweedFS Cloud Drive | Google / Dropbox Drive |
| -- | -- | -- | | -- | -- | -- |
| Cache To | Local Cluster | Local Disk | | Cache To | Local Cluster | Local Disk |
| Speed | Fast | Fast | | Speed | Fast | Fast |
@ -11,11 +11,11 @@ For normal computers, Google or Dropbox drive can be mounted as a network folder
| Management | Selectively Cache/Uncache | N/A | | Management | Selectively Cache/Uncache | N/A |
# SeaweedFS Cloud Cache # SeaweedFS Cloud Drive
With this feature, SeaweedFS can cache data that is on cloud. It can cache both metadata and file content. Given SeaweedFS unlimited scalability, the cache size is actually unlimited. Any local changes can be write back to the cloud asynchronously. With this feature, SeaweedFS can cache data that is on cloud. It can cache both metadata and file content. Given SeaweedFS unlimited scalability, the cache size is actually unlimited. Any local changes can be write back to the cloud asynchronously.
![SeaweedFS Cloud Cache](https://raw.githubusercontent.com/chrislusf/seaweedfs/master/note/SeaweedFS_RemoteMount.png) ![SeaweedFS Cloud Drive](https://raw.githubusercontent.com/chrislusf/seaweedfs/master/note/SeaweedFS_RemoteMount.png)
``` ```
[HDFS|Mount|HTTP|S3|WebDAV] <== Filer(metadata cache) <== Volume Servers (data cache) <== Cloud [HDFS|Mount|HTTP|S3|WebDAV] <== Filer(metadata cache) <== Volume Servers (data cache) <== Cloud
@ -63,8 +63,8 @@ If not starting `weed filer.remote.sync`, the data changes will not be propagate
# Continue to read # Continue to read
* [[Cloud Cache Benefits]] * [[Cloud Drive Benefits]]
* [[Cloud Cache Quick Setup]] * [[Cloud Drive Quick Setup]]
* [[Configure Remote Storage]] * [[Configure Remote Storage]]
* [[Mount Remote Storage]] * [[Mount Remote Storage]]
* [[Cache Remote Storage]] * [[Cache Remote Storage]]

2
Home.md

@ -6,7 +6,7 @@ Here is the white paper for [SeaweedFS Architecture.pdf](SeaweedFS_Architecture.
# Make Cloud Storage Cheaper and Faster! # Make Cloud Storage Cheaper and Faster!
To minimize cost and latency, you can mount your cloud data to a folder with [[SeaweedFS Cloud Cache|Cloud Cache Architecture]]. To minimize cost and latency, you can mount your cloud data to a folder with [[SeaweedFS Cloud Drive|Cloud Drive Architecture]].
* Read and write with local network speed. * Read and write with local network speed.
* Asynchronously propagate local updates to the cloud storage. * Asynchronously propagate local updates to the cloud storage.

8
_Sidebar.md

@ -34,13 +34,13 @@
* [[Filer Change Data Capture]] * [[Filer Change Data Capture]]
* [[FUSE Mount]] * [[FUSE Mount]]
* [[WebDAV]] * [[WebDAV]]
* Cloud Cache * Cloud Drive
* [[Cloud Cache Benefits]] * [[Cloud Drive Benefits]]
* [[Cloud Cache Architecture]] * [[Cloud Drive Architecture]]
* [[Configure Remote Storage]] * [[Configure Remote Storage]]
* [[Mount Remote Storage]] * [[Mount Remote Storage]]
* [[Cache Remote Storage]] * [[Cache Remote Storage]]
* [[Cloud Cache Quick Setup]] * [[Cloud Drive Quick Setup]]
* AWS S3 Gateway * AWS S3 Gateway
* [[Amazon S3 API]] * [[Amazon S3 API]]
* [[AWS CLI with SeaweedFS]] * [[AWS CLI with SeaweedFS]]