mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2024-01-19 02:48:24 +00:00
move to github.com/seaweedfs/seaweedfs
chrislu
parent
140455fa85
commit
0d8bd95030
|
|
@ -54,7 +54,7 @@ remove_bucket: newbucket3
|
|||
|
||||
# Presigned URL
|
||||
|
||||
If [authentication](https://github.com/chrislusf/seaweedfs/wiki/Amazon-S3-API#authentication) is enabled, the url is not accessible without proper credentials. But you can presign a url and access it.
|
||||
If [authentication](https://github.com/seaweedfs/seaweedfs/wiki/Amazon-S3-API#authentication) is enabled, the url is not accessible without proper credentials. But you can presign a url and access it.
|
||||
|
||||
```
|
||||
|
||||
|
|
@ -65,4 +65,4 @@ http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Cre
|
|||
# access the url
|
||||
$ curl "http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=some_access_key1%2F20200726%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200726T161749Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e0cc153209e414ca8168661f57827aa03ab84e7041ef9270ff639bcc519d24f5"
|
||||
|
||||
```
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Installation
|
||||
|
||||
See [AWS-CLI-with-SeaweedFS](https://github.com/chrislusf/seaweedfs/wiki/AWS-CLI-with-SeaweedFS#installation)
|
||||
See [AWS-CLI-with-SeaweedFS](https://github.com/seaweedfs/seaweedfs/wiki/AWS-CLI-with-SeaweedFS#installation)
|
||||
|
||||
# Execute commands
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ This will add 1 physical volume when existing volumes are full. If using replica
|
|||
fs.configure -locationPrefix=/buckets/ -replication=001 -volumeGrowthCount=2 -apply
|
||||
```
|
||||
|
||||
See https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Configuration
|
||||
See https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration
|
||||
|
||||
|
||||
# Supported APIs
|
||||
|
|
@ -251,4 +251,4 @@ uploading new files).** This is configured by setting
|
|||
|
||||
With both configurations (gRPC and JWT), it is possible to have Filer
|
||||
and S3 communicate in fully authenticated fashion; so Filer will reject
|
||||
any unauthenticated communication.
|
||||
any unauthenticated communication.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Deprecated
|
||||
|
||||
This feature is replace by https://github.com/chrislusf/seaweedfs/wiki/Filer-Active-Active-cross-cluster-continuous-synchronization
|
||||
This feature is replace by https://github.com/seaweedfs/seaweedfs/wiki/Filer-Active-Active-cross-cluster-continuous-synchronization
|
||||
|
||||
# Architecture
|
||||
|
||||
|
|
@ -96,4 +96,4 @@ Please read the source.filer and sink.filer configurations carefully and make su
|
|||
|
||||
The `weed filer.replicate` only replicates files that have been updated. If there are already a lot of files, very likely you would need to copy them too.
|
||||
|
||||
To do this, use `echo 'fs.meta.notify' | weed shell`, which will iterate all files and generate one "file creation" event for each file to the message queue. Thus, all files will be replicated.
|
||||
To do this, use `echo 'fs.meta.notify' | weed shell`, which will iterate all files and generate one "file creation" event for each file to the message queue. Thus, all files will be replicated.
|
||||
|
|
|
|||
|
|
@ -34,11 +34,11 @@ Name | Author | Language
|
|||
[Julia SeaweedFS Client](https://github.com/lawless-m/SeaweedFSClient.jl) | Lawless-m | Julia
|
||||
## GRPC APIs
|
||||
|
||||
SeaweedFS uses GRPC internally. You can use them too. Just check https://github.com/chrislusf/seaweedfs/tree/master/weed/pb for the proto files.
|
||||
SeaweedFS uses GRPC internally. You can use them too. Just check https://github.com/seaweedfs/seaweedfs/tree/master/weed/pb for the proto files.
|
||||
|
||||
When developing HDFS compatible file system, which allows replacing HDFS with SeaweedFS, a Java implementation of the GRPC client API is developed.
|
||||
|
||||
* Java GRPC client Source code: https://github.com/chrislusf/seaweedfs/tree/master/other/java/client
|
||||
* Java GRPC client Source code: https://github.com/seaweedfs/seaweedfs/tree/master/other/java/client
|
||||
* Java GRPC client Maven Repo: https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-client
|
||||
|
||||
## Projects using SeaweedFS
|
||||
|
|
|
|||
|
|
@ -8,9 +8,9 @@
|
|||
> file in SeaweedFS will also become a file in the cloud storage provider.
|
||||
> - This is useful in case you want to use the files inside the cloud provider's infrastructure.
|
||||
> - However, this does **not support file encryption** in any way (obviously), as the files are put to Cloud Storage as is.
|
||||
> 2) [Tiered Storage with Cloud Tier](https://github.com/chrislusf/seaweedfs/wiki/Cloud-Tier)
|
||||
> 2) [Tiered Storage with Cloud Tier](https://github.com/seaweedfs/seaweedfs/wiki/Cloud-Tier)
|
||||
> - In this mode, seaweedFS **moves full volume files to the cloud storage provider**, so files which are 1 GB (in our case) big.
|
||||
> - This mode supports [Filer Data Encryption](https://github.com/chrislusf/seaweedfs/wiki/Filer-Data-Encryption) transparently.
|
||||
> - This mode supports [Filer Data Encryption](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Data-Encryption) transparently.
|
||||
> - The chunk files uploaded to the cloud provider are not usable outside of SeaweedFS.
|
||||
|
||||
|
||||
|
|
@ -141,4 +141,3 @@ SeaweedFS Cloud Drive has these unique characteristics:
|
|||
* You may need to access cloud data by HDFS, or HTTP, or S3 API, or WebDav, or FUSE Mount.
|
||||
* With SeaweedFS Cloud Drive
|
||||
* Multiple ways to access remote storage.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
## Motivation
|
||||
|
||||
> NOTE: SeaweedFS provides **two mechanisms** to use cloud storage:
|
||||
> 1) [SeaweedFS Cloud Drive](https://github.com/chrislusf/seaweedfs/wiki/Cloud-Drive-Benefits)
|
||||
> 1) [SeaweedFS Cloud Drive](https://github.com/seaweedfs/seaweedfs/wiki/Cloud-Drive-Benefits)
|
||||
> - in this case, you can **mount** an S3 bucket to the Seaweedfs file system (in the filer), and access the remote files
|
||||
> through SeaweedFS. Effectively, SeaweedFS caches the files from the cloud storage.
|
||||
> - In this mode, the file structure in cloud store is exactly matching the SeaweedFS structure - so every
|
||||
|
|
@ -10,7 +10,7 @@
|
|||
> - However, this does **not support file encryption** in any way (obviously), as the files are put to Cloud Storage as is.
|
||||
> 2) **Tiered Storage with Cloud Tier** (**<== You are here**)
|
||||
> - In this mode, seaweedFS **moves full volume files to the cloud storage provider**, so files which are 1 GB (in our case) big.
|
||||
> - This mode supports [Filer Data Encryption](https://github.com/chrislusf/seaweedfs/wiki/Filer-Data-Encryption) transparently.
|
||||
> - This mode supports [Filer Data Encryption](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Data-Encryption) transparently.
|
||||
> - The chunk files uploaded to the cloud provider are not usable outside of SeaweedFS.
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ It is fairly easy if you need to store filer metadata with other data store.
|
|||
Let's use "yourstore" as the chosen name.
|
||||
|
||||
Here are the steps:
|
||||
1. Add a package under github.com/chrislusf/seaweedfs/weed/filer/yourstore
|
||||
1. Add a package under github.com/seaweedfs/seaweedfs/weed/filer/yourstore
|
||||
2. Implement the filer.FilerStore interface
|
||||
```
|
||||
package filer
|
||||
|
|
@ -45,21 +45,21 @@ func init() {
|
|||
filer2.Stores = append(filer2.Stores, &YourStore{})
|
||||
}
|
||||
```
|
||||
4. Load yourstore. Just import it in github.com/chrislusf/seaweedfs/weed/server/filer_server.go
|
||||
4. Load yourstore. Just import it in github.com/seaweedfs/seaweedfs/weed/server/filer_server.go
|
||||
```
|
||||
import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"github.com/chrislusf/seaweedfs/weed/filer"
|
||||
_ "github.com/chrislusf/seaweedfs/weed/filer/cassandra"
|
||||
_ "github.com/chrislusf/seaweedfs/weed/filer/leveldb"
|
||||
_ "github.com/chrislusf/seaweedfs/weed/filer/mysql"
|
||||
_ "github.com/chrislusf/seaweedfs/weed/filer/postgres"
|
||||
_ "github.com/chrislusf/seaweedfs/weed/filer/redis"
|
||||
_ "github.com/chrislusf/seaweedfs/weed/filer/yourstore"
|
||||
"github.com/seaweedfs/seaweedfs/weed/filer"
|
||||
_ "github.com/seaweedfs/seaweedfs/weed/filer/cassandra"
|
||||
_ "github.com/seaweedfs/seaweedfs/weed/filer/leveldb"
|
||||
_ "github.com/seaweedfs/seaweedfs/weed/filer/mysql"
|
||||
_ "github.com/seaweedfs/seaweedfs/weed/filer/postgres"
|
||||
_ "github.com/seaweedfs/seaweedfs/weed/filer/redis"
|
||||
_ "github.com/seaweedfs/seaweedfs/weed/filer/yourstore"
|
||||
// ^^ add here
|
||||
"github.com/chrislusf/seaweedfs/weed/security"
|
||||
"github.com/chrislusf/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/security"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
)
|
||||
```
|
||||
5. Send a pull request!
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ This is a really good guideline on how to setup minikube: https://thenewstack.io
|
|||
|
||||
# Steps
|
||||
|
||||
The default docker image is based on alpine Linux which causes issues with the DNS addon in Minikubes (see https://github.com/chrislusf/seaweedfs/issues/474). To get around this problem, I rebuilt the docker image from "scratch".
|
||||
The default docker image is based on alpine Linux which causes issues with the DNS addon in Minikubes (see https://github.com/seaweedfs/seaweedfs/issues/474). To get around this problem, I rebuilt the docker image from "scratch".
|
||||
|
||||
To do this I added (or modified) the following files to the local repository:
|
||||
|
||||
|
|
@ -40,8 +40,8 @@ The script uses the docker file to build my image. Note the followings:
|
|||
```
|
||||
#!/bin/sh
|
||||
|
||||
go get github.com/chrislusf/seaweedfs/weed/...
|
||||
CGO_ENABLED=0 GOOS=linux go build github.com/chrislusf/seaweedfs/weed
|
||||
go get github.com/seaweedfs/seaweedfs/weed/...
|
||||
CGO_ENABLED=0 GOOS=linux go build github.com/seaweedfs/seaweedfs/weed
|
||||
docker build -t weed:latest -f ./Dockerfile .
|
||||
docker tag weed:latest 192.168.42.23:80/weed:latest
|
||||
docker push 192.168.42.23:80/weed:latest
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ The downside:
|
|||
|
||||
Side Note:
|
||||
* The 10+4 can be easily adjusted via `DataShardsCount` and `ParityShardsCount` in
|
||||
https://github.com/chrislusf/seaweedfs/blob/master/weed/storage/erasure_coding/ec_encoder.go#L17
|
||||
https://github.com/seaweedfs/seaweedfs/blob/master/weed/storage/erasure_coding/ec_encoder.go#L17
|
||||
* If you are considering these enterprise-level customizations, please consider supporting SeaweedFS first.
|
||||
|
||||
## Architecture
|
||||
|
|
@ -189,4 +189,4 @@ Percentage of the requests served within a certain time (ms)
|
|||
98% 6.4 ms
|
||||
99% 7.7 ms
|
||||
100% 40.1 ms
|
||||
```
|
||||
```
|
||||
|
|
|
|||
4
FAQ.md
4
FAQ.md
|
|
@ -6,7 +6,7 @@ SeaweedFS has web dashboards for its different services:
|
|||
* Volume server dashboards can be accessed on `http://hostname:port/ui/index.html`.
|
||||
For example: `http://localhost:8080/ui/index.html`
|
||||
|
||||
Also see [#275](https://github.com/chrislusf/seaweedfs/issues/275).
|
||||
Also see [#275](https://github.com/seaweedfs/seaweedfs/issues/275).
|
||||
|
||||
### Does it support xxx language?
|
||||
If using `weed filer`, just send one HTTP POST to write, or one HTTP GET to read.
|
||||
|
|
@ -54,7 +54,7 @@ It is also important to leave some disk space for a couple of volume size, so th
|
|||
|
||||
If one volume has large number of small files, the memory usage would be high in order to keep each entry in memory or in leveldb.
|
||||
|
||||
To reduce memory usage, one way is to convert the older volumes into [Erasure-Coded volumes](https://github.com/chrislusf/seaweedfs/wiki/Erasure-Coding-for-warm-storage), which are read only. The volume server can will sort the index and store it as a sorted index file (with extension `.sdx`). So looking up one entry costs a binary search within the sorted index file, instead of O(1) memory lookup.
|
||||
To reduce memory usage, one way is to convert the older volumes into [Erasure-Coded volumes](https://github.com/seaweedfs/seaweedfs/wiki/Erasure-Coding-for-warm-storage), which are read only. The volume server can will sort the index and store it as a sorted index file (with extension `.sdx`). So looking up one entry costs a binary search within the sorted index file, instead of O(1) memory lookup.
|
||||
|
||||
### How to configure volumes larger than 30GB?
|
||||
|
||||
|
|
|
|||
|
|
@ -87,7 +87,7 @@ filer ---- mount1
|
|||
|
||||
#### Mount directory on host from docker-compose
|
||||
|
||||
If docker compose is being used to manage the server (eg. https://github.com/chrislusf/seaweedfs/wiki/Getting-Started#with-compose)
|
||||
If docker compose is being used to manage the server (eg. https://github.com/seaweedfs/seaweedfs/wiki/Getting-Started#with-compose)
|
||||
it's possible to mount a directory on the host with docker privileged mode like so:
|
||||
```
|
||||
mount_1:
|
||||
|
|
@ -280,7 +280,7 @@ From https://github.com/osxfuse/osxfuse/issues/358
|
|||
> FUSE needs to register a virtual device for exchanging messages between the kernel and the actual file system implementation running in user space. The number of available device slots is limited by macOS. So if you are using other software like VMware, VirtualBox, TunTap, Intel HAXM, ..., that eat up all free device slots, FUSE will not be able to register its virtual device.
|
||||
|
||||
### Samba share mounted folder ###
|
||||
From https://github.com/chrislusf/seaweedfs/issues/936
|
||||
From https://github.com/seaweedfs/seaweedfs/issues/936
|
||||
The issue is with samba.conf. If you see NT_STATUS_ACCESS_DENIED error, try to add `force user` and `force group` to your samba.conf file.
|
||||
```
|
||||
[profiles]
|
||||
|
|
|
|||
|
|
@ -81,7 +81,7 @@ The gRPC API is also open to public and can support many other languages.
|
|||
|
||||
Here is an example, in Java:
|
||||
|
||||
https://github.com/chrislusf/seaweedfs/blob/master/other/java/examples/src/main/java/com/seaweedfs/examples/WatchFiles.java
|
||||
https://github.com/seaweedfs/seaweedfs/blob/master/other/java/examples/src/main/java/com/seaweedfs/examples/WatchFiles.java
|
||||
|
||||
To subscribe the meta data changes:
|
||||
| Parameter | Meaning |
|
||||
|
|
@ -102,7 +102,7 @@ Basically there are four types of events to handle:
|
|||
|
||||
This is based on Filer gRPC API. You should be able to easily implement it in your own language.
|
||||
|
||||
https://github.com/chrislusf/seaweedfs/blob/master/weed/pb/filer.proto#L52
|
||||
https://github.com/seaweedfs/seaweedfs/blob/master/weed/pb/filer.proto#L52
|
||||
|
||||
# Possible Use Cases
|
||||
|
||||
|
|
|
|||
|
|
@ -21,5 +21,4 @@ Another side is, with GDPR, companies are required to "forget" customer data aft
|
|||
### Encryption Algorithm
|
||||
The encryption is through AES256-GCM https://en.wikipedia.org/wiki/Galois/Counter_Mode
|
||||
|
||||
There is one randomly generated cipher key of 256 bits for each file chunk. The cipher code is here https://github.com/chrislusf/seaweedfs/blob/master/weed/util/cipher.go
|
||||
|
||||
There is one randomly generated cipher key of 256 bits for each file chunk. The cipher code is here https://github.com/seaweedfs/seaweedfs/blob/master/weed/util/cipher.go
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ On filer, there is a `/topics/.system/log` folder, it stores all filer metadata
|
|||
## Metadata Event Format
|
||||
The events are stored in files organized by timestamp, `yyyy-MM-dd/hh-mm.segment`.
|
||||
|
||||
The events are encoded by protobuf, defined in https://github.com/chrislusf/seaweedfs/blob/master/weed/pb/filer.proto . The related sections are:
|
||||
The events are encoded by protobuf, defined in https://github.com/seaweedfs/seaweedfs/blob/master/weed/pb/filer.proto . The related sections are:
|
||||
```
|
||||
service SeaweedFiler {
|
||||
rpc SubscribeMetadata (SubscribeMetadataRequest) returns (stream SubscribeMetadataResponse) {
|
||||
|
|
@ -41,7 +41,7 @@ The ondisk file is a repeated bytes of the following format:
|
|||
The `LogEntry.data` stores serialized `SubscribeMetadataResponse`
|
||||
|
||||
## Read Metadata Events
|
||||
The events can be read by any program as files. One example is here: https://github.com/chrislusf/seaweedfs/blob/master/unmaintained/see_log_entry/see_log_entry.go
|
||||
The events can be read by any program as files. One example is here: https://github.com/seaweedfs/seaweedfs/blob/master/unmaintained/see_log_entry/see_log_entry.go
|
||||
|
||||
## Subscribe to Metadata
|
||||
|
||||
|
|
|
|||
|
|
@ -270,11 +270,11 @@ The patterns are case-sensitive and support wildcard characters '*' and '?'.
|
|||
// recursively delete everything, ignoring any recursive error
|
||||
> curl -X DELETE http://localhost:8888/path/to/dir?recursive=true&ignoreRecursiveError=true
|
||||
// For Experts Only: remove filer directories only, without removing data chunks.
|
||||
// see https://github.com/chrislusf/seaweedfs/pull/1153
|
||||
// see https://github.com/seaweedfs/seaweedfs/pull/1153
|
||||
> curl -X DELETE http://localhost:8888/path/to?recursive=true&skipChunkDeletion=true
|
||||
```
|
||||
| Parameter | Description | Default |
|
||||
| ---- | -- | -- |
|
||||
| recursive | if "recursive=true", recursively delete all files and folders | filer recursive_delete option from filer.toml |
|
||||
| ignoreRecursiveError | if "ignoreRecursiveError=true", ignore errors in recursive mode | false |
|
||||
| skipChunkDeletion | if "skipChunkDeletion=true", do not delete file chunks on volume servers | false |
|
||||
| skipChunkDeletion | if "skipChunkDeletion=true", do not delete file chunks on volume servers | false |
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
The Filer Store persists all file metadata and directory information.
|
||||
|
||||
| Filer Store Name | Lookup | number of entries in a folder | Scalability | Directory Renaming | TTL | [Fast Bucket Deletion](https://github.com/chrislusf/seaweedfs/wiki/S3-API-FAQ#how-to-speed-up-bucket-deletion) |Note |
|
||||
| Filer Store Name | Lookup | number of entries in a folder | Scalability | Directory Renaming | TTL | [Fast Bucket Deletion](https://github.com/seaweedfs/seaweedfs/wiki/S3-API-FAQ#how-to-speed-up-bucket-deletion) |Note |
|
||||
| ---------------- | -- | -- | -- | -- | -- | -- | -- |
|
||||
| memory | O(1) | limited by memory | Local, Fast | | Yes| | for testing only, no persistent storage |
|
||||
| leveldb | O(logN)| unlimited | Local, Very Fast | | Yes| | Default, fairly scalable |
|
||||
|
|
@ -13,8 +13,8 @@ The Filer Store persists all file metadata and directory information.
|
|||
| Mongodb | O(logN)| unlimited | Local or Distributed, Fast | | Yes| | Easy to manage |
|
||||
| Arangodb | O(logN)| unlimited | Local or Distributed, Fast | | Native| Yes | Easy to manage; Scalable |
|
||||
| YDB | O(logN)| unlimited | Local or Distributed, Fast |Atomic| Native| Yes | Easy to manage; True elastic Scalability; High Availability. Need to manually build. |
|
||||
| [Redis2](https://github.com/chrislusf/seaweedfs/wiki/Filer-Redis-Setup)| O(1) | limited | Local or Distributed, Fastest ||Native| | one directory's children are stored in one key~value entry |
|
||||
| [Redis3](https://github.com/chrislusf/seaweedfs/wiki/Filer-Redis-Setup)| O(1) | unlimited | Local or Distributed, Fastest ||Native| | one directory's children are spread into multiple key~value entries |
|
||||
| [Redis2](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Redis-Setup)| O(1) | limited | Local or Distributed, Fastest ||Native| | one directory's children are stored in one key~value entry |
|
||||
| [Redis3](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Redis-Setup)| O(1) | unlimited | Local or Distributed, Fastest ||Native| | one directory's children are spread into multiple key~value entries |
|
||||
| Cassandra | O(logN)| unlimited | Local or Distributed, Very Fast||Native| | |
|
||||
| MySql | O(logN)| unlimited | Local or Distributed, Fast |Atomic| Yes| | Easy to manage |
|
||||
| MySql2 | O(logN)| unlimited | Local or Distributed, Fast |Atomic| Yes| Yes| Easy to manage |
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
## Installing SeaweedFS
|
||||
|
||||
Download the latest official release from https://github.com/chrislusf/seaweedfs/releases.
|
||||
Download the latest official release from https://github.com/seaweedfs/seaweedfs/releases.
|
||||
|
||||
Decompress the downloaded file. You will only find one executable file, either "weed" on most systems or "weed.exe" on windows.
|
||||
|
||||
|
|
@ -92,7 +92,7 @@ docker-compose -f docker/seaweedfs-compose.yml -p seaweedfs up
|
|||
|
||||
You can use image "chrislusf/seaweedfs" or build your own with [dockerfile][] in the root of repo.
|
||||
|
||||
[dockerfile]: https://github.com/chrislusf/seaweedfs/blob/master/docker/Dockerfile
|
||||
[dockerfile]: https://github.com/seaweedfs/seaweedfs/blob/master/docker/Dockerfile
|
||||
|
||||
|
||||
### Using pre-built Docker image
|
||||
|
|
@ -118,7 +118,7 @@ curl "http://$IP:9333/cluster/status?pretty=y"
|
|||
Make a local copy of seaweedfs from github
|
||||
|
||||
```bash
|
||||
git clone https://github.com/chrislusf/seaweedfs.git
|
||||
git clone https://github.com/seaweedfs/seaweedfs.git
|
||||
```
|
||||
|
||||
Minimal Image (~19.6 MB)
|
||||
|
|
|
|||
|
|
@ -4,16 +4,16 @@ SeaweedFS excels on small files and has no issue to store large files. Now it is
|
|||
|
||||
# Build SeaweedFS Hadoop Client Jar
|
||||
```
|
||||
$cd $GOPATH/src/github.com/chrislusf/seaweedfs/other/java/client
|
||||
$cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/client
|
||||
$ mvn install
|
||||
|
||||
# build for hadoop2
|
||||
$cd $GOPATH/src/github.com/chrislusf/seaweedfs/other/java/hdfs2
|
||||
$cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2
|
||||
$ mvn package
|
||||
$ ls -al target/seaweedfs-hadoop2-client-3.13.jar
|
||||
|
||||
# build for hadoop3
|
||||
$cd $GOPATH/src/github.com/chrislusf/seaweedfs/other/java/hdfs3
|
||||
$cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3
|
||||
$ mvn package
|
||||
$ ls -al target/seaweedfs-hadoop3-client-3.13.jar
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ To support large files, SeaweedFS supports these two kinds of files:
|
|||
|
||||
This piece of code shows the json file structure:
|
||||
|
||||
https://github.com/chrislusf/seaweedfs/blob/master/weed/operation/chunked_file.go#L24
|
||||
https://github.com/seaweedfs/seaweedfs/blob/master/weed/operation/chunked_file.go#L24
|
||||
|
||||
```
|
||||
type ChunkInfo struct {
|
||||
|
|
@ -158,4 +158,4 @@ There are no particular limit in terms of chunk file size. Each chunk size does
|
|||
## weed filer and weed mount
|
||||
The filer server and the FUSE implementation that uses filer server are automatically chunking large files into smaller chunks.
|
||||
|
||||
The list of chunks are stored in filer storage, and managed by filer or weed mount client.
|
||||
The list of chunks are stored in filer storage, and managed by filer or weed mount client.
|
||||
|
|
|
|||
|
|
@ -134,7 +134,7 @@ will also generate a "pictures" collection and a "documents" collection if they
|
|||
|
||||
Actually, the actual data files have the collection name as the prefix, e.g., "pictures_1.dat", "documents_3.dat".
|
||||
|
||||
If you need to delete them later see https://github.com/chrislusf/seaweedfs/wiki/Master-Server-API#delete-collection
|
||||
If you need to delete them later see https://github.com/seaweedfs/seaweedfs/wiki/Master-Server-API#delete-collection
|
||||
|
||||
## Logging
|
||||
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ We will use 2 servers. Server 1 will host master, 2x volumes (2 disks, one volum
|
|||
# todo: use 2 step build process, copy over weed binary to fresh container (do not need curl and tar at runtime)
|
||||
FROM alpine
|
||||
RUN apk update && apk add wget tar
|
||||
RUN wget https://github.com/chrislusf/seaweedfs/releases/download/3.13/linux_amd64_large_disk.tar.gz
|
||||
RUN wget https://github.com/seaweedfs/seaweedfs/releases/download/3.13/linux_amd64_large_disk.tar.gz
|
||||
RUN tar -xf linux_amd64_large_disk.tar.gz
|
||||
RUN chmod +x weed
|
||||
RUN mv weed /usr/bin/
|
||||
|
|
@ -52,7 +52,7 @@ services:
|
|||
```
|
||||
|
||||
5. `docker-compose up` on both servers and check that the master sees the volume
|
||||
6. Follow [security guide](https://github.com/chrislusf/seaweedfs/wiki/Security-Configuration) to add secrets and certs. Scaffold `security.toml` file and generate certs, in this example, all certs are in `certs/` folder. Update `docker-compose.yml` of master server:
|
||||
6. Follow [security guide](https://github.com/seaweedfs/seaweedfs/wiki/Security-Configuration) to add secrets and certs. Scaffold `security.toml` file and generate certs, in this example, all certs are in `certs/` folder. Update `docker-compose.yml` of master server:
|
||||
```yml
|
||||
version: '3.7'
|
||||
services:
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ Here are some tests on one single computer. However, if you need more performanc
|
|||
# Test with Warp
|
||||
https://github.com/minio/warp
|
||||
|
||||
Warp is a more complete test suite. It needs identity access management. So you would need to configure authentication info with `weed s3 -conf=...` or `weed filer -s3 -s3.conf=...`. See https://github.com/chrislusf/seaweedfs/wiki/Amazon-S3-API#authentication
|
||||
Warp is a more complete test suite. It needs identity access management. So you would need to configure authentication info with `weed s3 -conf=...` or `weed filer -s3 -s3.conf=...`. See https://github.com/seaweedfs/seaweedfs/wiki/Amazon-S3-API#authentication
|
||||
|
||||
Here is the results from my local laptop written to an external SSD via USB 3.1:
|
||||
|
||||
|
|
@ -172,4 +172,4 @@ benchmark$ hsbench -a accesstoken -s secret -z 4K -d 10 -t 10 -b 10 -u http://l
|
|||
2020/11/17 20:06:20 Running Loop 0 BUCKET DELETE TEST
|
||||
2020/11/17 20:06:21 Loop: 0, Int: TOTAL, Dur(s): 0.4, Mode: BDEL, Ops: 10, MB/s: 0.00, IO/s: 24, Lat(ms): [ min: 371.6, avg: 407.9, 99%: 421.6, max: 421.6 ], Slowdowns: 0
|
||||
benchmark$
|
||||
```
|
||||
```
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
## Can not upload due to "no free volumes left"
|
||||
|
||||
The symptom is similar to https://github.com/chrislusf/seaweedfs/issues/1631 where the logs show
|
||||
The symptom is similar to https://github.com/seaweedfs/seaweedfs/issues/1631 where the logs show
|
||||
```
|
||||
Nov 20 18:49:37 s2375.j weed[31818]: E1120 18:49:37 31818 filer_server_handlers_write.go:42] failing to assign a file id: rpc error: code = Unknown desc = No free volumes left!
|
||||
Nov 20 18:49:37 s2375.j weed[31818]: I1120 18:49:37 31818 common.go:53] response method:PUT URL:/buckets/dev-passport-video-recordings/02342a46-7435-b698-2437-c778db34ef59.mp4 with httpStatus:500 and JSON:{"error":"rpc error: code = Unknown desc = No free volumes left!"}
|
||||
|
|
@ -24,14 +24,14 @@ This will add 1 physical volume when existing volumes are full. If using replica
|
|||
fs.configure -locationPrefix=/buckets/ -replication=001 -volumeGrowthCount=2 -apply
|
||||
```
|
||||
|
||||
See https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Configuration
|
||||
See https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration
|
||||
|
||||
|
||||
## How to speed up bucket deletion?
|
||||
|
||||
One common unexpected problem is the deletion can be slow. To delete a file, we need to delete the file content on the volume servers and delete the file entry from the filer store. It is almost the same amount of work as adding a file. If there are millions of files, it can take a long time to delete.
|
||||
|
||||
When you need to create large buckets and delete them often, you may choose `leveldb3` as the filer store, or any other stores that supports **Fast Bucket Deletion** in https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores
|
||||
When you need to create large buckets and delete them often, you may choose `leveldb3` as the filer store, or any other stores that supports **Fast Bucket Deletion** in https://github.com/seaweedfs/seaweedfs/wiki/Filer-Stores
|
||||
|
||||
`leveldb3` can automatically create a separate LevelDB instance for each bucket.
|
||||
So bucket deletion is as simple as deleting the LevelDB instance files and the collection of volume files.
|
||||
|
|
@ -50,4 +50,4 @@ Example using rclone:
|
|||
rclone lsf seaweedfs:my-bucket/dir
|
||||
```
|
||||
|
||||
If the directory `dir` exists in `my-bucket`, the orphaned metadata will be cleaned up. Note that due to slight API usage differences, `rclone ls` does not trigger cleanup, but `rclone lsf` will.
|
||||
If the directory `dir` exists in `my-bucket`, the orphaned metadata will be cleaned up. Note that due to slight API usage differences, `rclone ls` does not trigger cleanup, but `rclone lsf` will.
|
||||
|
|
|
|||
|
|
@ -2,12 +2,12 @@ There are a few external Java libraries available. But actually SeaweedFS alread
|
|||
|
||||
Here is an SeaweedFS Java API implementation refactored out of the existing code.
|
||||
|
||||
https://github.com/chrislusf/seaweedfs/tree/master/other/java/examples/src/main/java/com/seaweedfs/examples
|
||||
https://github.com/seaweedfs/seaweedfs/tree/master/other/java/examples/src/main/java/com/seaweedfs/examples
|
||||
|
||||
# Build Java Client Jar
|
||||
|
||||
```
|
||||
$cd $GOPATH/src/github.com/chrislusf/seaweedfs/other/java/client
|
||||
$cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/client
|
||||
$ mvn install
|
||||
```
|
||||
|
||||
|
|
@ -131,4 +131,4 @@ Sometimes you may need to go deeper. For example, change modification time `mtim
|
|||
// save the new entry
|
||||
entryBuilder.setAttributes(attrBuilder);
|
||||
filerClient.updateEntry("/some/dir", entryBuilder.build());
|
||||
```
|
||||
```
|
||||
|
|
|
|||
|
|
@ -82,8 +82,8 @@ To enable JWT-based access control for the Filer,
|
|||
|
||||
If `jwt.filer_signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `jwt.filer_signing.key`.
|
||||
|
||||
The JwtToken can be generated by calling `security.GenJwtForFilerServer(signingKey SigningKey, expiresAfterSec int)` in `github.com/chrislusf/seaweedfs/weed/security` package.
|
||||
https://github.com/chrislusf/seaweedfs/blob/9b941773805400c520558d83aed633adc821988c/weed/security/jwt.go#L53
|
||||
The JwtToken can be generated by calling `security.GenJwtForFilerServer(signingKey SigningKey, expiresAfterSec int)` in `github.com/seaweedfs/seaweedfs/weed/security` package.
|
||||
https://github.com/seaweedfs/seaweedfs/blob/9b941773805400c520558d83aed633adc821988c/weed/security/jwt.go#L53
|
||||
|
||||
If `jwt.filer_signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `jwt.filer_signing.read.key`.
|
||||
|
||||
|
|
|
|||
|
|
@ -41,10 +41,10 @@ And then you can configure your Prometheus to crawl them periodically.
|
|||
|
||||
# Dashboard
|
||||
|
||||
The dashboard is shared at https://github.com/chrislusf/seaweedfs/blob/master/other/metrics/grafana_seaweedfs.json
|
||||
The dashboard is shared at https://github.com/seaweedfs/seaweedfs/blob/master/other/metrics/grafana_seaweedfs.json
|
||||
|
||||
If you modify the dashboard, please share your revisions.
|
||||
|
||||
## Example Dashboard
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ Example (within the weed shell):
|
|||
fs.configure -locationPrefix=/buckets/ssd_ -disk=ssd -apply
|
||||
```
|
||||
|
||||
https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Configuration
|
||||
https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration
|
||||
|
||||
# Custom Tags
|
||||
|
||||
|
|
|
|||
|
|
@ -39,7 +39,7 @@ curl -H "Content-Type:image/png" -F file=@myImage.png http://127.0.0.1:8080/5,27
|
|||
|
||||
The simple way is to front all master and volume servers with firewall.
|
||||
|
||||
The following white list option is deprecated. Please follow https://github.com/chrislusf/seaweedfs/wiki/Security-Overview
|
||||
The following white list option is deprecated. Please follow https://github.com/seaweedfs/seaweedfs/wiki/Security-Overview
|
||||
|
||||
A white list option can be used. Only traffic from the white list IP addresses have write permission.
|
||||
|
||||
|
|
@ -103,4 +103,4 @@ weed volume -dir=/tmp/vol1/ -mserver="localhost:9334" -ip="localhost" -port=8081
|
|||
weed volume -dir=/tmp/vol2/ -mserver="localhost:9334" -ip="localhost" -port=8082
|
||||
```
|
||||
|
||||
so we finished moving data of localhost:8083 to localhost:8081/localhost:8082
|
||||
so we finished moving data of localhost:8083 to localhost:8081/localhost:8082
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ curl -F file=@/home/chris/myphoto.jpg http://127.0.0.1:8080/3,01637037d6
|
|||
{"size": 43234}
|
||||
```
|
||||
|
||||
The size returned is the size stored on SeaweedFS, sometimes the file is automatically gzipped based on the file extension or mime type [(see when compression will be applied automatically)](https://github.com/chrislusf/seaweedfs/blob/c42b95c596f762dcca2bc9c7e7a918ab8ca8b206/weed/util/compression.go#L111).
|
||||
The size returned is the size stored on SeaweedFS, sometimes the file is automatically gzipped based on the file extension or mime type [(see when compression will be applied automatically)](https://github.com/seaweedfs/seaweedfs/blob/c42b95c596f762dcca2bc9c7e7a918ab8ca8b206/weed/util/compression.go#L111).
|
||||
|
||||
| URL Parameter | Description | Default |
|
||||
| ---- | -- | -- |
|
||||
|
|
@ -150,4 +150,3 @@ curl "http://localhost:8080/status?pretty=y"
|
|||
]
|
||||
}
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -37,8 +37,8 @@ And modify the configuration at runtime:
|
|||
1. change the spark-defaults.conf
|
||||
|
||||
```
|
||||
spark.driver.extraClassPath=/Users/chris/go/src/github.com/chrislusf/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.13.jar
|
||||
spark.executor.extraClassPath=/Users/chris/go/src/github.com/chrislusf/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.13.jar
|
||||
spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.13.jar
|
||||
spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.13.jar
|
||||
spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
|
||||
```
|
||||
|
||||
|
|
@ -81,8 +81,8 @@ spark.history.fs.cleaner.enabled=true
|
|||
spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark2-history/
|
||||
spark.eventLog.dir=seaweedfs://localhost:8888/spark2-history/
|
||||
|
||||
spark.driver.extraClassPath=/Users/chris/go/src/github.com/chrislusf/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.13.jar
|
||||
spark.executor.extraClassPath=/Users/chris/go/src/github.com/chrislusf/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.13.jar
|
||||
spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.13.jar
|
||||
spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.13.jar
|
||||
spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
|
||||
spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888
|
||||
```
|
||||
|
|
|
|||
|
|
@ -127,6 +127,6 @@ docker run \
|
|||
"fs.configure -locationPrefix=/buckets/foo -volumeGrowthCount=3 -replication=002 -apply"
|
||||
```
|
||||
|
||||
Here `shell` selects the [Docker image entrypoint](https://github.com/chrislusf/seaweedfs/blob/master/docker/entrypoint.sh#L60-L64).
|
||||
Here `shell` selects the [Docker image entrypoint](https://github.com/seaweedfs/seaweedfs/blob/master/docker/entrypoint.sh#L60-L64).
|
||||
|
||||
The arguments are `fs.configure -locationPrefix=/buckets/foo -volumeGrowthCount=3 -replication=002 -apply`
|
||||
The arguments are `fs.configure -locationPrefix=/buckets/foo -volumeGrowthCount=3 -replication=002 -apply`
|
||||
|
|
|
|||
Loading…
Reference in a new issue