Commit graph

1001 commits

Author SHA1 Message Date
chrislu 192983b464 s3 backend support customizing storage class 2022-04-30 17:36:40 -07:00
chrislu 3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
chrislu 6994e88192 fix typo 2022-04-21 00:02:18 -07:00
a 1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
chrislu a69f0b62c2 start filer local socket only on non windows
fix https://github.com/chrislusf/seaweedfs/issues/2930
2022-04-18 00:08:31 -07:00
leyou240 89eb87c1d1
Merge branch 'master' into slices.SortFunc 2022-04-18 10:39:29 +08:00
justin 3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu 1384529eb7 Fix filer.backup deletes files in backup folder in incremental mode
fix https://github.com/chrislusf/seaweedfs/issues/2919
2022-04-14 13:35:01 -07:00
a 846858fb43 merge master 2022-04-12 14:42:29 -07:00
a 41d396edc4 Merge branch 'master' into a 2022-04-07 08:22:59 -07:00
Konstantin Lebedev f5246b748d Merge branch 'new_master' into hashicorp_raft
# Conflicts:
#	weed/pb/master_pb/master.pb.go
2022-04-07 18:50:27 +05:00
chrislu 86ed27f602 mount: remove leftover socket file 2022-04-07 00:33:13 -07:00
eddy-gfx ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
Konstantin Lebedev 17c6e8e39f Merge branch 'new_master' into hashicorp_raft
# Conflicts:
#	go.mod
#	go.sum
2022-04-05 13:29:46 +05:00
Konstantin Lebedev 622297f1a7 add stats raft handler 2022-04-04 19:16:06 +05:00
Konstantin Lebedev 14dd971890 hashicorp raft with state machine 2022-04-04 17:51:51 +05:00
Konstantin Lebedev c514710b7b initial add hashicorp raft 2022-04-04 13:50:56 +05:00
chrislu 6a2bcd03aa configure mount quota 2022-04-02 21:34:26 -07:00
chrislu 958f880b70 mount: add grpc method to adjust quota 2022-04-02 15:14:37 -07:00
chrislu daca2d22a5 use original server address string as map key 2022-04-01 17:34:42 -07:00
a 7e92517571 change user and pass to username and password 2022-04-01 14:09:25 -05:00
Konstantin Lebedev b6a2b43efe allowDeleteBucketNotEmpty 2022-03-30 22:46:13 +05:00
a bc603e534f diff 2022-03-28 16:50:28 +00:00
chrislu 21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
chrislu 4ba7127ab1 refactor 2022-03-26 13:13:19 -07:00
Konstantin Lebedev 849a2c1f68 fix cmdFiler 2022-03-23 14:30:46 +05:00
Konstantin Lebedev 105bcfdbd4 set iam ip from filer 2022-03-23 13:45:51 +05:00
elee 423ce57cde prefix search, bucket implemented 2022-03-17 21:12:25 -05:00
Berck Nash 7ee38fa3a4 The fixes for https://github.com/chrislusf/seaweedfs/issues/1937 had a few problems:
(1) The help file says that in the absence of a ipBind being specified, that it will bind to the "ip" specified.  Instead, it bound to localhost which broke the default configuration.  This change implements the documented behavior instead.

(2) The new IAM filer ip address has no default. This instantiates it to the same as the filer IP.  I'm not sure if there should be a corresponding iam.ip or iam.ipBind option added to the filer command?
2022-03-17 15:30:23 -06:00
elee 921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
Berck Nash 9b14f0c81a Add mTLS support for both master and volume http server. 2022-03-16 09:52:17 -06:00
chrislu 3639cad69c master, filer, s3: also listen to "localhost" in addition to specific ip address
related to https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-15 22:28:18 -07:00
chrislu 3a6eb8ca5f default bind to one ip address
fix https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-11 14:02:39 -08:00
banjiaojuhao b9ff7723dd [bugfix] filer: nil pointer dereference 2022-03-07 23:26:25 +08:00
chrislu da3d330616 s3 and filer transport using unix domain socket instead of tcp 2022-03-07 02:00:14 -08:00
chrislu c7e8ac18f0 mount: quota for one mounted collection
related to https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48
2022-03-06 02:44:40 -08:00
chrislu f51e20028a mount: avoid comma in mount options
fix https://github.com/chrislusf/seaweedfs/issues/2719
2022-03-03 03:42:29 -08:00
chrislu 18543c6e8b minor 2022-02-27 23:11:09 -08:00
chrislu c3792c8352 remove dead code 2022-02-27 03:03:19 -08:00
chrislu aa9eef81e6 retire mount v1 2022-02-27 02:57:27 -08:00
chrislu 6e1ab97988 use debug option to see operations 2022-02-27 02:02:30 -08:00
chrislu 2112d99140 mount2: add back readonly mode 2022-02-27 01:13:32 -08:00
chrislu fc7a4957ea fix mount2 options 2022-02-25 21:22:44 -08:00
chrislu 202a29d014 refactoring 2022-02-25 01:17:26 -08:00
Tuan Vuong d2ec62656d initialize master address in iam options 2022-02-23 12:01:54 +07:00
banjiaojuhao b5ec346700 FilerStore: add redis_lua 2022-02-15 20:54:57 +08:00
chrislu 4e72863ba5 mount2 add debug mode 2022-02-15 00:26:30 -08:00
chrislu 4e181db21a mount: default disable cache
* Prevent cases as https://github.com/seaweedfs/seaweedfs-csi-driver/issues/43
* Improve read write benchmarks
* Improve AI training performance. Most of the files are just read once.
2022-02-14 20:42:33 -08:00
chrislu 377bf31445 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-14 13:48:51 -08:00
chrislu e8420aaed7 fix building for windows freebsd 2022-02-14 13:48:48 -08:00
Chris Lu d3ee621fce
Merge pull request #2661 from garenchan/ck-dev1 2022-02-14 10:08:37 -08:00
Konstantin Lebedev 526094d2da StopTimeout 30 sec 2022-02-14 21:42:27 +05:00
Konstantin Lebedev 275e9a4e86 reduce to default http server KillTimeout and StopTimeout 2022-02-14 21:38:24 +05:00
garenchan bd032eabe7 [UPDATE] Make heartbeat interval and election timeout of masters configurable. 2022-02-14 21:09:07 +08:00
chrislu 05724a68d4 skip other OS 2022-02-14 02:59:51 -08:00
chrislu b9c2bff931 clean up 2022-02-14 02:14:26 -08:00
chrislu dbeeda8123 listen for metadata updates 2022-02-14 01:09:31 -08:00
chrislu 180445f5a8 change to use fuse file system 2022-02-11 21:35:09 -08:00
chrislu f87da798a4 to be re-written following fuse virtual file system 2022-02-11 03:09:30 -08:00
chrislu b6143de52a mount with name 2022-02-10 22:43:55 -08:00
chrislu 7a0c35674c clean up previously mounted folder 2022-02-10 20:46:53 -08:00
chrislu c3f9d9fa2e initial setup 2022-02-10 20:32:13 -08:00
chrislu 85c1615b43 filer read empty file may cause OOM in some cases
fix https://github.com/chrislusf/seaweedfs/issues/2641
2022-02-07 23:08:54 -08:00
chrislu affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
chrislu 84c9bc4389 edge case: old entry was not replicated to remote storage 2022-01-30 20:23:24 -08:00
Chris Lu 7c66f3b5fb
Merge pull request #2602 from kmlebedev/master_metrics
master metricsHttpPort
2022-01-20 09:26:25 -08:00
chrislu b3e526ba95 url should be always using forward slash 2022-01-19 22:16:26 -08:00
Konstantin Lebedev 77c98b657e master metricsHttpPort 2022-01-19 21:43:22 +05:00
guol-fnst da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
chrislu fc0628c038 working 2022-01-17 01:53:56 -08:00
chrislu 2bfeb5d1c8 add filer to iam option 2022-01-15 03:37:52 -08:00
chrislu b17c426e99 weed server: optionally start IAM service
related to https://github.com/chrislusf/seaweedfs/issues/2560
2022-01-13 22:49:49 -08:00
chrislu 8907e6a40a add more help messages 2022-01-13 13:03:04 -08:00
chrislu 826a7b307e master: remove hard coded filer settings in master.toml
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
Kyle Sanderson 9e012001be
filer.copy: don't crash when volume creation fails
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
2022-01-11 22:22:39 -08:00
chrislu 1a7d5b5b5e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 12:24:56 -08:00
chrislu 41daecfdca Update mount_std.go 2022-01-11 12:23:12 -08:00
Chris Lu abe5da7d2c
Merge pull request #2575 from Radtoo/fix_paths2
Fix paths2
2022-01-11 12:04:30 -08:00
chrislu 6a12520a96 fix logging 2022-01-10 01:00:11 -08:00
Radtoo 389002f195 Using positional arguments rather than option flag to enable better shell usage 2022-01-08 16:52:12 +01:00
Radtoo fba1efb77a Now works with a single file too
Parsing removed from doFixOneVolume

Needle init removed from runFix
2022-01-08 16:31:53 +01:00
chrislu 110d5a5233 support fixing a collection of volumes, or volumes under one directory 2022-01-07 14:52:16 -08:00
Chris Lu 42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst 1cd3b6b4e1 BUGFIX: security.toml contained wrong keys 2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
chrislu 5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
chrislu fb434318e3 dynamically adjust connection timeout
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:44:39 -08:00
chrislu 5788bf2270 s3: increase timeout limit
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
chrislu 498661e3bb mount: remove limits on number of parallel requests 2021-12-28 17:41:01 -08:00
chrislu 0da2dfd640 fuse: change to direct io mode
before and after:

chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.86s system 84% cpu 4.561 total
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.22s system 83% cpu 3.857 total
2021-12-28 12:22:56 -08:00
chrislu 9f9ef1340c use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
Sebastian Kurfürst 6db49100d6 BUGFIX: add access.ui setting to scaffolded security.toml
... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)
2021-12-24 13:59:04 +01:00
chrislu 85c526c583 s3: bind to a specific IP
fix https://github.com/chrislusf/seaweedfs/issues/2516
2021-12-17 11:34:37 -08:00
chrislu bf4d7affc0 gateway to remote object store: skip replicating multipart upload part files
fix https://github.com/chrislusf/seaweedfs/issues/2509
2021-12-14 19:48:31 -08:00
chrislu 316f326464 add more help message 2021-12-13 13:14:36 -08:00
kmlebedev 4f98553ba9 audit log SignatureVersion 2021-12-10 19:40:32 +05:00
Konstantin Lebedev 98251fe16a non blocking audit log 2021-12-09 19:47:16 +05:00
Chris Lu 2ba08afed1
Merge pull request #2498 from kmlebedev/s3_audit_log 2021-12-07 09:35:48 -08:00
Konstantin Lebedev 10678cde81 audit log config 2021-12-07 18:20:52 +05:00