mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2024-01-19 02:48:24 +00:00
rename to cloud drive
parent
8234c91ace
commit
118cd68483
|
@ -50,7 +50,7 @@ However, how to make SeaweedFS work with data already on cloud?
|
|||
* Can transparently switch to different cloud storage vendors.
|
||||
* Can detach from the cloud storage if decided to move off cloud.
|
||||
|
||||
# What SeaweedFS Cloud Cache is not?
|
||||
# What SeaweedFS Cloud Drive is not?
|
||||
|
||||
* It is not a proxy. Proxy is not ideal for low-latency read or write operations.
|
||||
* Its cache size is not limited. With unlimited size, all file content can be localized. So the read can be fast even on the first attempt, without worrying cached items are evicted due to capacity.
|
||||
|
@ -58,11 +58,11 @@ However, how to make SeaweedFS work with data already on cloud?
|
|||
* It does not change file storage layout in the cloud storage. Other tools can access the files on the cloud storage as usual.
|
||||
* It does not need to link to the original cloud storage. After caching all the file content, you can choose to detach from the cloud storage.
|
||||
|
||||
# Why SeaweedFS Cloud Cache?
|
||||
# Why SeaweedFS Cloud Drive?
|
||||
|
||||
There are other products also cache S3 data. What makes SeaweedFS Cloud Cache better?
|
||||
There are other products also cache S3 data. What makes SeaweedFS Cloud Drive better?
|
||||
|
||||
SeaweedFS Cloud Cache has these unique characteristics:
|
||||
SeaweedFS Cloud Drive has these unique characteristics:
|
||||
* Scalable Capacity
|
||||
* Easy to add more capacity.
|
||||
* Disaggregated storage.
|
||||
|
@ -81,8 +81,8 @@ SeaweedFS Cloud Cache has these unique characteristics:
|
|||
* Problem
|
||||
* Training jobs need to repeatedly visit a large set of files.
|
||||
* The randomized access pattern is hard for caching.
|
||||
* With SeaweedFS Cloud Cache
|
||||
* Users can explicitly ask SeaweedFS Cloud Cache to cache one whole folder.
|
||||
* With SeaweedFS Cloud Drive
|
||||
* Users can explicitly ask SeaweedFS Cloud Drive to cache one whole folder.
|
||||
* Increase training speed and reduce API cost and network cost.
|
||||
* Users can access data with FUSE mounted folders.
|
||||
## Data Hoarding
|
||||
|
@ -91,38 +91,38 @@ SeaweedFS Cloud Cache has these unique characteristics:
|
|||
* With cloud capacity, reliability, and storage tiering, saving data files there may be a good idea.
|
||||
* Accessing cloud data is slow and costly.
|
||||
* Recently uploaded files very likely need to be accessed again.
|
||||
* With SeaweedFS Cloud Cache
|
||||
* Users can explicitly ask SeaweedFS Cloud Cache to uncache by file age.
|
||||
* With SeaweedFS Cloud Drive
|
||||
* Users can explicitly ask SeaweedFS Cloud Drive to uncache by file age.
|
||||
* Users can also choose to never uncache, basically treating cloud copy as a backup.
|
||||
## Big Data
|
||||
* Problem
|
||||
* Run MapReduce, Spark, and Flink jobs on cloud data is slow due to metadata operations.
|
||||
* Repeated data access increases unnecessary cost.
|
||||
* May need to work with the cloud ecosystem.
|
||||
* With SeaweedFS Cloud Cache
|
||||
* With SeaweedFS Cloud Drive
|
||||
* Avoiding slow cloud storage metadata access.
|
||||
* Access data only once.
|
||||
* Write back data to work with cloud ecosystems.
|
||||
## Cloud Storage Vendor Agnostic
|
||||
* Problem
|
||||
* Different datasets may need to be on different vendors, based on access pattern, latency, cost, etc.
|
||||
* With SeaweedFS Cloud Cache
|
||||
* With SeaweedFS Cloud Drive
|
||||
* Transparently switch to from one vendor to another.
|
||||
## Switch S3 Compatible Storage Vendors
|
||||
* Problem
|
||||
* Other S3 compatible vendors are known to have lesser flexibility, scalability and performance as SeaweedFS.
|
||||
* With SeaweedFS Cloud Cache
|
||||
* With SeaweedFS Cloud Drive
|
||||
* Compare SeaweedFS with the vendor over a long period of time.
|
||||
* Transparently switch away if SeaweedFS works better.
|
||||
## Move Off Cloud
|
||||
* Problem
|
||||
* Cloud storage is costly!
|
||||
* With SeaweedFS Cloud Cache
|
||||
* With SeaweedFS Cloud Drive
|
||||
* Help to transition between on-cloud to off-cloud.
|
||||
* When you are happy with it, just stop the write back process (and cancel the monthly payment to the cloud vendor!).
|
||||
## Support multiple access methods.
|
||||
* Problem
|
||||
* You may need to access cloud data by HDFS, or HTTP, or S3 API, or WebDav, or FUSE Mount.
|
||||
* With SeaweedFS Cloud Cache
|
||||
* With SeaweedFS Cloud Drive
|
||||
* Multiple ways to access remote storage.
|
||||
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
To users not familiar with SeaweedFS, there seems many things to learn.
|
||||
But for SeaweedFS Cloud Cache, the setup is easy.
|
||||
But for SeaweedFS Cloud Drive, the setup is easy.
|
||||
|
||||
# Setup a simple SeaweedFS cluster
|
||||
|
||||
|
|
|
@ -1,8 +1,8 @@
|
|||
# Introduction
|
||||
|
||||
For normal computers, Google or Dropbox drive can be mounted as a network folder. The remote files are cached to local disks and automatically synchronized with the cloud. So it is fast to access. Similarly, SeaweedFS Cloud Cache can mount cloud storage as a special folder to the local cluster. The cloud files can be cached to the local cluster and automatically synchronized with the cloud.
|
||||
For normal computers, Google or Dropbox drive can be mounted as a network folder. The remote files are cached to local disks and automatically synchronized with the cloud. So it is fast to access. Similarly, SeaweedFS Cloud Drive can mount cloud storage as a special folder to the local cluster. The cloud files can be cached to the local cluster and automatically synchronized with the cloud.
|
||||
|
||||
| | SeaweedFS Cloud Cache | Google / Dropbox Drive |
|
||||
| | SeaweedFS Cloud Drive | Google / Dropbox Drive |
|
||||
| -- | -- | -- |
|
||||
| Cache To | Local Cluster | Local Disk |
|
||||
| Speed | Fast | Fast |
|
||||
|
@ -11,11 +11,11 @@ For normal computers, Google or Dropbox drive can be mounted as a network folder
|
|||
| Management | Selectively Cache/Uncache | N/A |
|
||||
|
||||
|
||||
# SeaweedFS Cloud Cache
|
||||
# SeaweedFS Cloud Drive
|
||||
|
||||
With this feature, SeaweedFS can cache data that is on cloud. It can cache both metadata and file content. Given SeaweedFS unlimited scalability, the cache size is actually unlimited. Any local changes can be write back to the cloud asynchronously.
|
||||
|
||||
![SeaweedFS Cloud Cache](https://raw.githubusercontent.com/chrislusf/seaweedfs/master/note/SeaweedFS_RemoteMount.png)
|
||||
![SeaweedFS Cloud Drive](https://raw.githubusercontent.com/chrislusf/seaweedfs/master/note/SeaweedFS_RemoteMount.png)
|
||||
|
||||
```
|
||||
[HDFS|Mount|HTTP|S3|WebDAV] <== Filer(metadata cache) <== Volume Servers (data cache) <== Cloud
|
||||
|
@ -63,8 +63,8 @@ If not starting `weed filer.remote.sync`, the data changes will not be propagate
|
|||
|
||||
|
||||
# Continue to read
|
||||
* [[Cloud Cache Benefits]]
|
||||
* [[Cloud Cache Quick Setup]]
|
||||
* [[Cloud Drive Benefits]]
|
||||
* [[Cloud Drive Quick Setup]]
|
||||
* [[Configure Remote Storage]]
|
||||
* [[Mount Remote Storage]]
|
||||
* [[Cache Remote Storage]]
|
||||
|
|
2
Home.md
2
Home.md
|
@ -6,7 +6,7 @@ Here is the white paper for [SeaweedFS Architecture.pdf](SeaweedFS_Architecture.
|
|||
|
||||
# Make Cloud Storage Cheaper and Faster!
|
||||
|
||||
To minimize cost and latency, you can mount your cloud data to a folder with [[SeaweedFS Cloud Cache|Cloud Cache Architecture]].
|
||||
To minimize cost and latency, you can mount your cloud data to a folder with [[SeaweedFS Cloud Drive|Cloud Drive Architecture]].
|
||||
|
||||
* Read and write with local network speed.
|
||||
* Asynchronously propagate local updates to the cloud storage.
|
||||
|
|
|
@ -34,13 +34,13 @@
|
|||
* [[Filer Change Data Capture]]
|
||||
* [[FUSE Mount]]
|
||||
* [[WebDAV]]
|
||||
* Cloud Cache
|
||||
* [[Cloud Cache Benefits]]
|
||||
* [[Cloud Cache Architecture]]
|
||||
* Cloud Drive
|
||||
* [[Cloud Drive Benefits]]
|
||||
* [[Cloud Drive Architecture]]
|
||||
* [[Configure Remote Storage]]
|
||||
* [[Mount Remote Storage]]
|
||||
* [[Cache Remote Storage]]
|
||||
* [[Cloud Cache Quick Setup]]
|
||||
* [[Cloud Drive Quick Setup]]
|
||||
* AWS S3 Gateway
|
||||
* [[Amazon S3 API]]
|
||||
* [[AWS CLI with SeaweedFS]]
|
||||
|
|
Loading…
Reference in a new issue