gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Cloud Cache Benefits (markdown)

Chris Lu 2021-08-16 01:45:34 -07:00
parent 9a41f263c2
commit dfdcb90ae0
1 changed files with 7 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
Cloud-Cache-Benefits.md

@ -51,7 +51,7 @@ However, how to make SeaweedFS work with data already on cloud?
# Possible Use Cases # Possible Use Cases
* Machine learning ## Machine learning
* Problem * Problem
* Training jobs need to repeatedly visit a large set of files. * Training jobs need to repeatedly visit a large set of files.
* The randomized access pattern is hard for caching. * The randomized access pattern is hard for caching.
@ -59,14 +59,14 @@ However, how to make SeaweedFS work with data already on cloud?
* Users can explicitly ask SeaweedFS Cloud Cache to cache one whole folder. * Users can explicitly ask SeaweedFS Cloud Cache to cache one whole folder.
* Increase training speed and reduce API cost and network cost. * Increase training speed and reduce API cost and network cost.
* Users can access data with FUSE mounted folders. * Users can access data with FUSE mounted folders.
* Data Hoarding ## Data Hoarding
* Problem * Problem
* With cloud capacity and storage tiering, saving data files there may be a good idea. * With cloud capacity and storage tiering, saving data files there may be a good idea.
* Recently uploaded files very likely need to be accessed again. * Recently uploaded files very likely need to be accessed again.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Cache
* Users can explicitly ask SeaweedFS Cloud Cache to uncache by file age. * Users can explicitly ask SeaweedFS Cloud Cache to uncache by file age.
* Users can also choose to never uncache, basically treating cloud copy as a backup. * Users can also choose to never uncache, basically treating cloud copy as a backup.
* Big Data ## Big Data
* Problem * Problem
* Run MapReduce, Spark, and Flink jobs on cloud data is slow due to metadata operations. * Run MapReduce, Spark, and Flink jobs on cloud data is slow due to metadata operations.
* Repeated data access increases unnecessary cost. * Repeated data access increases unnecessary cost.
@ -75,24 +75,24 @@ However, how to make SeaweedFS work with data already on cloud?
* Avoiding slow cloud storage metadata access. * Avoiding slow cloud storage metadata access.
* Access data only once. * Access data only once.
* Write back data to work with cloud ecosystems. * Write back data to work with cloud ecosystems.
* Cloud Storage Vendor Agnostic ## Cloud Storage Vendor Agnostic
* Problem * Problem
* Different datasets may need to be on different vendors, based on access pattern, latency, cost, etc. * Different datasets may need to be on different vendors, based on access pattern, latency, cost, etc.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Cache
* Transparently switch to from one vendor to another. * Transparently switch to from one vendor to another.
* Switch S3 Compatible Storage Vendors ## Switch S3 Compatible Storage Vendors
* Problem * Problem
* Other S3 compatible vendors are known to have lesser flexibility, scalability and performance as SeaweedFS. * Other S3 compatible vendors are known to have lesser flexibility, scalability and performance as SeaweedFS.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Cache
* Compare SeaweedFS with the vendor over a long period of time. * Compare SeaweedFS with the vendor over a long period of time.
* Transparently switch away if SeaweedFS works better. * Transparently switch away if SeaweedFS works better.
* Move Off Cloud ## Move Off Cloud
* Problem * Problem
* Cloud storage is costly! * Cloud storage is costly!
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Cache
* Help to transition between on-cloud to off-cloud. * Help to transition between on-cloud to off-cloud.
* When you are happy with it, just stop the write back process (and cancel the monthly payment to the cloud vendor!). * When you are happy with it, just stop the write back process (and cancel the monthly payment to the cloud vendor!).
* Support multiple access methods. ## Support multiple access methods.
* Problem * Problem
* You may need to access cloud data by HDFS, or HTTP, or S3 API, or WebDav, or FUSE Mount. * You may need to access cloud data by HDFS, or HTTP, or S3 API, or WebDav, or FUSE Mount.
* With SeaweedFS Cloud Cache * With SeaweedFS Cloud Cache