gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
8565 commits 28 branches 278 tags 135 MiB
Commit graph

1350 commits

Author SHA1 Message Date
guol-fnst de6aa9cce8 avoid duplicated volume directory 2022-05-16 19:33:51 +08:00
Chris Lu a69bf53405
Merge pull request #2996 from kmlebedev/ydb 
ydb filer store
 2022-05-05 15:34:37 -07:00
chrislu b201edb9df fix wrong assignment 2022-05-03 07:12:33 -07:00
chrislu 9271866d1e fix segmentation violation 
fix https://github.com/chrislusf/seaweedfs/issues/3000
 2022-05-02 23:56:11 -07:00
Konstantin Lebedev d8925b4e83 Merge branch 'new_master' into ydb 
# Conflicts:
#	go.mod
#	go.sum
 2022-05-03 00:13:57 +05:00
chrislu 94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev 50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
chrislu 5d081d5c3c skip if already copied to remote 
fix https://github.com/chrislusf/seaweedfs/issues/2993
 2022-04-30 20:03:04 -07:00
Konstantin Lebedev 03aec9ce7e avoid panic if hashicorp raft not initialized 2022-04-30 13:56:35 +05:00
chrislu 94f824e1ce volume: sync to disk before copying volume files 
address https://github.com/chrislusf/seaweedfs/issues/2976
 2022-04-26 13:03:43 -07:00
guol-fnst 8b2552f944 fix http response error code 2022-04-24 11:07:27 +08:00
a 1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
Konstantin Lebedev 1e35b4929f shell vacuum volume by collection and volume id 2022-04-18 18:40:58 +05:00
leyou240 89eb87c1d1
Merge branch 'master' into slices.SortFunc 2022-04-18 10:39:29 +08:00
justin 3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu b4be56bb3b add timing info during ping operation 2022-04-16 12:45:49 -07:00
a 846858fb43 merge master 2022-04-12 14:42:29 -07:00
Chris Lu a87f57e47c
Merge pull request #2868 from kmlebedev/hashicorp_raft 
hashicorp raft
 2022-04-10 23:00:05 -07:00
Konstantin Lebedev d2fe0fae33 use the full ServerAddress as the identifier 2022-04-11 10:44:44 +05:00
Chris Lu 9f4ecd50ac
Merge pull request #2889 from blacktear23/filer-ui-bootstrap-style 
Change filer UI icon from picture to icon font.
 2022-04-08 00:50:32 -07:00
yulai.li ceb058ddd9 Change filer UI icon from picture to icon font. 2022-04-08 15:07:56 +08:00
chrislu 7f26676f57 adjust td width 2022-04-07 22:34:32 -07:00
yulai.li dff63e9024 Make operation buttons show when table row hovered 2022-04-08 13:10:12 +08:00
chrislu 4dba102137 use icon instead of button text 2022-04-07 11:15:01 -07:00
Chris Lu bbc95dac47
Merge pull request #2885 from binbinshi/master 
fix: master lose some volumes
 2022-04-07 10:16:19 -07:00
Konstantin Lebedev 104ea7b029 master ui add raft stats and Max Volume Id 2022-04-07 20:52:01 +05:00
a 41d396edc4 Merge branch 'master' into a 2022-04-07 08:22:59 -07:00
Konstantin Lebedev a2fdb3e277 fix master ui 2022-04-07 19:37:40 +05:00
Konstantin Lebedev 35bc67f030 fix conflicts 2022-04-07 18:52:13 +05:00
Konstantin Lebedev f5246b748d Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	weed/pb/master_pb/master.pb.go
 2022-04-07 18:50:27 +05:00
yulai.li 11a9f993a8 Make filer-ui bootstrap style 2022-04-07 20:17:00 +08:00
Konstantin Lebedev 7ff248d5cd refactor OnPeerUpdate 2022-04-07 16:23:22 +05:00
Konstantin Lebedev 85d80fd36d fix removing old raft server 2022-04-07 15:31:37 +05:00
chrislu abe3cc6df2 filer UI touch up 2022-04-07 01:25:55 -07:00
yulai.li 2347c21cdd Tune filer UI add rename feature 2022-04-07 15:19:41 +08:00
yulai.li 2454020a92 Add upload progress support 2022-04-07 15:19:37 +08:00
yulai.li 4f87ee7755 Add create directory and delete web UI features for filer 2022-04-07 15:19:33 +08:00
shibinbin c20e1edd99 fix: master lose some volumes 2022-04-07 15:18:28 +08:00
eddy-gfx ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
Konstantin Lebedev 14a2cc83bf raft update peers via OnPeerUpdate 2022-04-06 21:17:04 +05:00
Konstantin Lebedev 357aa818fe add raft shell cmds 2022-04-06 15:23:53 +05:00
chrislu 79b8e6a8c3 add a place holder for later merge consecutive chunks 2022-04-06 00:28:48 -07:00
Konstantin Lebedev c1aeded2dd force raft bootstrap 
avoid err bootstrap only works on new clusters
 2022-04-06 12:18:43 +05:00
chrislu bc888226fc erasure coding: tracking encoded/decoded volumes 
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
 2022-04-05 19:03:02 -07:00
Konstantin Lebedev 0e796a5582 rm set NoSnapshotRestoreOnStart 2022-04-05 18:28:42 +05:00
Konstantin Lebedev 68f11b9687 sleep bootstraping 2022-04-05 15:42:19 +05:00
Konstantin Lebedev 1ae7f509f5 LocalID from raw string 2022-04-05 13:50:39 +05:00
Konstantin Lebedev 89d32a0670 fix glog Info 2022-04-05 13:43:34 +05:00
Konstantin Lebedev 17c6e8e39f Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	go.mod
#	go.sum
 2022-04-05 13:29:46 +05:00
Konstantin Lebedev b7cdde14ae auto bootstraping and update peers 2022-04-05 13:17:53 +05:00
Konstantin Lebedev 622297f1a7 add stats raft handler 2022-04-04 19:16:06 +05:00
Konstantin Lebedev 1a1e5778c3 fix cluster status 2022-04-04 18:52:08 +05:00
Konstantin Lebedev 14dd971890 hashicorp raft with state machine 2022-04-04 17:51:51 +05:00
Konstantin Lebedev c514710b7b initial add hashicorp raft 2022-04-04 13:50:56 +05:00
chrislu 6d55716fc3 use constants 2022-04-01 16:55:26 -07:00
chrislu bbbbbd70a4 master supports grpc ping 2022-04-01 16:50:58 -07:00
chrislu 743ad690b6 filer supports grpc ping 2022-04-01 16:44:58 -07:00
chrislu 2305508b65 refactor: separate into two files 2022-04-01 16:40:49 -07:00
chrislu 800cbc004c volume server adds ping function 2022-04-01 16:37:06 -07:00
a 549e341c6f resolv 2022-03-30 06:31:19 -05:00
a bc603e534f diff 2022-03-28 16:50:28 +00:00
Konstantin Lebedev 84b7b83517 fix permission mkdir snapshot 
avoid open file operation not permitted
 2022-03-28 18:41:52 +05:00
Chris Lu 93615b2a49
Merge pull request #2828 from guo-sj/fix-tagging-error 
Fix delete all "Seaweed-" prefixed tagging error
 2022-03-28 00:35:22 -07:00
guosj dd2b9d93cc update code according to PR's comment 2022-03-28 15:23:31 +08:00
guosj 02076f01a6 fix tagging error 2022-03-28 14:50:48 +08:00
chrislu 21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
chrislu 4ba7127ab1 refactor 2022-03-26 13:13:19 -07:00
chrislu fba1cfc2d6 simplify a bit 2022-03-26 10:24:05 -07:00
chrislu a3411dd9da refactor 2022-03-26 10:21:26 -07:00
Chris Lu 1b27f96669
Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers 
Fix remove deleted peers
 2022-03-25 11:36:15 -07:00
Konstantin Lebedev ddd3945c26 fix remove deleted peers of raft server 
https://github.com/chrislusf/seaweedfs/issues/2804
 2022-03-25 15:09:38 +05:00
Konstantin Lebedev c1450bf9fe always clear previous log to avoid server is promotable 
https://github.com/chrislusf/seaweedfs/issues/2804
 2022-03-25 13:40:19 +05:00
Chris Lu 89d84e275b
Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request 
Need to exit waiting if request is was canceled
 2022-03-24 12:21:44 -07:00
Konstantin Lebedev 0b790d2bbf exclude the replication from the concurrentUploadLimitMB 2022-03-24 13:54:42 +05:00
chrislu 9b04f17555 remove dead code 2022-03-23 23:18:32 -07:00
chrislu 0b5faef612 fix 2022-03-23 23:16:54 -07:00
chrislu 0562fceb99 volume: fail fast if too many concurrent requests, to avoid dead lock due to replication. 
fix https://github.com/chrislusf/seaweedfs/issues/2755
 2022-03-23 22:53:58 -07:00
zzq09494 9f16df736e fix-filer: calculation error of the method skipCheckParentDirEntry 2022-03-19 09:15:42 +08:00
elee b25d03340b Merge branch 'a' of github.com:gfxlabs/seaweedfs into a 2022-03-17 04:51:59 -05:00
elee 921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
chrislu 4042fdf3bb rename to skipCheckParentDir 
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
 2022-03-16 23:55:31 -07:00
zzq09494 40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494 81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494 a6a8892255 Revert "filer: support uploading file without needEnsureParentDir" 
This reverts commit a93c4947ba.
 2022-03-17 10:27:17 +08:00
zzq09494 a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
Konstantin Lebedev f43c6daeda Need to exit waiting if request is was canceled 2022-03-15 19:55:22 +05:00
banjiaojuhao f28dbbe5c5 [bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user. 2022-03-10 11:40:39 +08:00
chrislu 6d3db4445b buffer for all range requests 2022-03-07 01:56:47 -08:00
chrislu f3bcbeb60a a little optimization 2022-03-07 00:24:59 -08:00
Chris Lu 1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest 
filer: support get metadata with resolved manifest chunk
 2022-03-07 00:05:46 -08:00
chrislu bb0b784544 minor 2022-03-07 00:04:59 -08:00
banjiaojuhao bfcc9ca808 filer: support metadata with resolved manifest chunk 2022-03-07 15:47:51 +08:00
banjiaojuhao 71f3046841 filer: add back isAppend function 2022-03-07 15:41:07 +08:00
Konstantin Lebedev cf444ebd07 Set default leveldb2 enabled 
avoid Filer store is enabled for both leveldb2 and mysql
 2022-03-06 18:27:25 +05:00
chrislu f1713c96ae avoid possible runtime error: index out of range [0] with length 0 2022-03-05 21:14:31 -08:00
chrislu a96d4254e9 filer, s3, volume server: a bit memory optimization 2022-03-02 20:15:28 -08:00
chrislu b2a148cb4c use file size as max range 2022-02-26 03:00:08 -08:00
chrislu 28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu e423548673 rename: pass along entry metadata 2022-02-25 02:53:37 -08:00
chrislu 320637dc7a use "mv.from" for moving files 2022-02-23 15:34:42 -08:00
banjiaojuhao 6ab09e9071 filer_http: support uploading file with offset 2022-02-22 00:15:00 +08:00
banjiaojuhao e6126cef62 filer_web: support moving entry 2022-02-20 23:56:23 +08:00
banjiaojuhao 4c30934cd9 filer: support get file entry 2022-02-18 22:52:26 +08:00
Konstantin Lebedev 9ea09cc41c healthz check to avoid drain pod with last replicas 2022-02-16 14:18:36 +05:00
garenchan bd032eabe7 [UPDATE] Make heartbeat interval and election timeout of masters configurable. 2022-02-14 21:09:07 +08:00
root 7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu 433fde4b18 move error to a separate file 
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
 2022-02-04 22:57:51 -08:00
Chris Lu a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics 
error metrics for filer and store
 2022-02-04 22:35:13 -08:00
chrislu affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
Konstantin Lebedev 9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev 3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Konstantin Lebedev c9952759c4 metrics master is leader 2022-01-24 20:13:07 +05:00
Konstantin Lebedev 28efe31524 new master metrics 2022-01-24 19:09:43 +05:00
chrislu 4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu e69c374956 minor 2022-01-13 02:01:53 -08:00
chrislu 826a7b307e master: remove hard coded filer settings in master.toml 
fix https://github.com/chrislusf/seaweedfs/issues/2529
 2022-01-12 01:11:25 -08:00
chrislu b8fbf19e9a mount: rename follow POSIX 2022-01-11 03:23:03 -08:00
Chris Lu 42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158 
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
 2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
 2021-12-30 14:45:27 +01:00
chrislu 5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
Sebastian Kurfuerst fcc09cef6f Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst d156d410ef rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
chrislu 67b723f74e Filer Server API support fsync 
fix https://github.com/chrislusf/seaweedfs/issues/2528
 2021-12-26 17:28:47 -08:00
chrislu 9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
banjiaojuhao 083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
Chris Lu 4e73705533
Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node 
filer server: allow upload file to specific dataNode
 2021-12-22 12:49:15 -08:00
banjiaojuhao 08336be92e filer server: allow upload file to specific dataNode 2021-12-22 21:57:26 +08:00
chrislu 4b8dcff448 reverting default admin scripts 
fix https://github.com/chrislusf/seaweedfs/issues/2525

this new default value was introduced in 2.80
this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.
 2021-12-20 13:34:57 -08:00
chrislu a152f17937 mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
chrislu 7210558c7b s3: pass through s3 presigned headers 
fix https://github.com/chrislusf/seaweedfs/discussions/2502
 2021-12-15 13:18:53 -08:00
chrislu 7bf14452f7 s3: handle Response-Content-Disposition 2021-12-12 23:05:40 -08:00
chrislu a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
chrislu 2d6fcdf83a add delay if need to resume from disk 2021-12-09 13:13:05 -08:00
liubaojiang c7abc34dbb rename operation returns the correct old parent path 2021-12-10 00:02:57 +08:00
chrislu 40ee9b4d7b sleep between the gap of each execution 
related to https://github.com/chrislusf/seaweedfs/issues/2476
 2021-12-07 11:44:08 -08:00
chrislu b70cb3e0b2 upgrade protoc to 3.17.3 
$brew install protobuf
$ protoc --version
libprotoc 3.17.3
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
 2021-12-05 00:42:25 -08:00
Chris Lu 3d7390302d add s3.clean.uploads -timeAgo=24h 2021-11-29 00:49:49 -08:00
Chris Lu 88ff8fc27b ensure uploaded chunks are deleted on error 2021-11-29 00:28:26 -08:00
Chris Lu bea26549f2 avoid empty extended attribute value 2021-11-28 23:50:52 -08:00
Chris Lu a0ef6e3611 prevent nil response 
fix https://github.com/chrislusf/seaweedfs/issues/2452
 2021-11-15 08:46:01 -08:00
Chris Lu 4729a57cc0 use constants 2021-11-08 17:47:56 -08:00
Chris Lu d9dd72ea56 rename pacakge 2021-11-08 17:47:56 -08:00
Chris Lu c4e22b5a9a filer: deprecate "-peers" option 2021-11-06 14:36:45 -07:00
Chris Lu e0fc2898e9 auto updated filer peer list 2021-11-06 14:23:35 -07:00
Chris Lu 84bb8e7365 send peers info to filers 2021-11-06 04:07:38 -07:00
Chris Lu 4b9c42996a refactor grpc API 2021-11-05 18:11:40 -07:00
Chris Lu 5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu 77f90ae288 add leader election in master 2021-11-04 00:54:38 -07:00
Chris Lu 3c245c69d3 handle possible race condition 2021-11-03 01:36:39 -07:00