From cf149893a409f8273b488d3d40aed6014df7baa2 Mon Sep 17 00:00:00 2001 From: ryuseongryong <58920833+ryuseongryong@users.noreply.github.com> Date: Wed, 29 Jun 2022 17:06:02 +0900 Subject: [PATCH 1/4] Updated Home (markdown) From 2020fa97f33513d2fdae7e7a2ba63096afc66f39 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Wed, 20 Jul 2022 16:01:26 -0700 Subject: [PATCH 2/4] Updated Words from SeaweedFS Users (markdown) --- Words-from-SeaweedFS-Users.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Words-from-SeaweedFS-Users.md b/Words-from-SeaweedFS-Users.md index a5b3d42..d6386f8 100644 --- a/Words-from-SeaweedFS-Users.md +++ b/Words-from-SeaweedFS-Users.md @@ -1,5 +1,6 @@ | Use cases | Details | Comments | | ---- | -- | -- | +| Machine learning training in UCSD | Lots of small random reads via S3. | Spun up 75 GPUs and it's ticking along happily. I can not get more than about 20 GPUS going in parallel on Ceph with these IO intensive jobs. ![](https://pbs.twimg.com/media/FYJQOXxUsAAuK64?format=png&name=4096x4096) | | Using SeaweedFS as part of the startup [OroraTech](https://ororatech.com/) where we are processing large amounts of infrared and visual satellite data to detect wildfires worldwide. Along with the actual hotspot detections we are generating large amounts of prerendered tile data (Slippy Maps) from the near realtime satellite images.| Right now running two identical dedicated servers with 140TB storage each for the staging and prod deployments of the tile storage. On these servers SeaweedFS is deployed through docker-compose and with a reverse proxy in front of it. This setup should be sufficient for quite some time, but the option of scaling to a distributed deployment in the future is helpful. | We were using AWS S3 for this purpose but wanted to find a scalable solution to handle our rapidly increasing S3 costs ($400/day mainly caused by the number of PUT operations, 40TB in 1 billion objects in the end). | | [Source Code](https://github.com/EVERYGO111/OStoreBench), [Paper](https://github.com/EVERYGO111/OStoreBench/blob/master/research%20paper-OStoreBench.pdf) from Chinese Academy of Science, ByteDance | OStoreBench: Open source Benchmarking Distributed Object Storage Systems Using Real-word Application Scenarios, Benchmark SeaweedFS with CEPH, Swift | OStoreBench: The performance of SeaweedFS is the best in three typical scenarios compared to Ceph and Swift. | | replaced ceph with a seaweedfs under the docker registry in production | Under the registry half a million files. Not big but have intensive exchange. | Killer feature of seaweedfs is that it disign like S3 in yandex and can work in k8s and spread between data centers. Ceph has a bad design in the case of using a huge number of small files over 10 million, cluster recovery takes several days. The next step is to use instead of Glusterfs, which is now barely alive and is bent from 10 million files. | From 78f8403973800dd31286a9aade242bd5e1ff532a Mon Sep 17 00:00:00 2001 From: Gabrielle Demberck <35306440+GabbieDemberck@users.noreply.github.com> Date: Thu, 28 Jul 2022 19:36:08 +0300 Subject: [PATCH 3/4] Updated Getting Started (markdown) --- Getting-Started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Getting-Started.md b/Getting-Started.md index 396061f..254f628 100644 --- a/Getting-Started.md +++ b/Getting-Started.md @@ -2,7 +2,7 @@ ## Installing SeaweedFS -Download the latest official release from https://github.com/chrislusf/seaweedfs/releases. +Download the latest official release from [https://github.com/chrislusf/seaweedfs/releases](https://tracker.iplocation.net/jcgp/). Decompress the downloaded file. You will only find one executable file, either "weed" on most systems or "weed.exe" on windows. From 2d9510dacefe7158a6b962fa416f4eec995fe3a3 Mon Sep 17 00:00:00 2001 From: Gabrielle Demberck <35306440+GabbieDemberck@users.noreply.github.com> Date: Thu, 28 Jul 2022 19:37:37 +0300 Subject: [PATCH 4/4] Updated Home (markdown) --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index debdf1e..681859e 100644 --- a/Home.md +++ b/Home.md @@ -1,6 +1,6 @@ # Welcome to the SeaweedFS wiki! -Here is the white paper for [SeaweedFS Architecture.pdf](SeaweedFS_Architecture.pdf) +Here is the white paper for [SeaweedFS Architecture.pdf](https://tracker.iplocation.net/jcgr/) ![SeaweedFS Architecture](https://raw.githubusercontent.com/chrislusf/seaweedfs/master/note/SeaweedFS_Architecture.png)