Commit graph

4460 commits

Author SHA1 Message Date
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
Sebastian Kurfuerst fcc09cef6f Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst d156d410ef rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst eda4c43a08 fix typo in error message 2021-12-29 12:38:14 +01:00
chrislu eb4ad2546f use proper chunk size limit option 2021-12-24 22:52:18 -08:00
chrislu 41bbf320bb use 2MB chunk size. cache size is the wrong option 2021-12-24 22:50:19 -08:00
chrislu 982ea85d81 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-24 22:40:07 -08:00
chrislu 083d8e9ece add stream writer
this should improve streaming write performance, which is common in many cases, e.g., copying large files.

This is additional to improved random read write operations: 3e69d19380...19084d8791
2021-12-24 22:38:22 -08:00
Sebastian Kurfürst 6db49100d6 BUGFIX: add access.ui setting to scaffolded security.toml
... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)
2021-12-24 13:59:04 +01:00
chrislu 255a1c7dcd refactor type names 2021-12-23 18:23:18 -08:00
chrislu f77ca41769 refactor 2021-12-23 17:48:34 -08:00
chrislu 1d36884845 rename files 2021-12-23 17:47:58 -08:00
chrislu 2d1a1f5e03 rename variables and functions 2021-12-23 17:35:57 -08:00
chrislu 7bf48ee135 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-23 17:23:26 -08:00
chrislu 6de331b014 clean up 2021-12-23 17:23:21 -08:00
chrislu 032df784ed chunked file works now 2021-12-23 17:17:32 -08:00
banjiaojuhao 083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
chrislu c2aad1c7ff detect non streaming mode on first write request 2021-12-22 17:20:44 -08:00
chrislu b541e39a2c fix tests 2021-12-22 16:17:30 -08:00
chrislu 2bc6fa90ff Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-22 16:05:43 -08:00
chrislu 0ec7bc6710 detect non streaming mode on the first read 2021-12-22 16:05:38 -08:00
chrislu 4c1368d621 fix test 2021-12-22 16:05:08 -08:00
Chris Lu 4e73705533
Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node
filer server: allow upload file to specific dataNode
2021-12-22 12:49:15 -08:00
banjiaojuhao 08336be92e filer server: allow upload file to specific dataNode 2021-12-22 21:57:26 +08:00
chrislu 7b78fc72b0 add page chunk interval list 2021-12-22 02:53:33 -08:00
chrislu b7cd52636b Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-21 22:24:43 -08:00
chrislu 3981d65b68 remove println 2021-12-21 22:24:38 -08:00
chrislu 9a73319b45 mount: different write strategy for streaming write and random write 2021-12-21 17:28:55 -08:00
Chris Lu b0665a15f4
Merge pull request #2527 from banjiaojuhao/master-assign-by-datanode 2021-12-21 08:56:51 -08:00
banjiaojuhao dda6b90d25 assign fileId according to DataNode with empty DataCenter and Rack 2021-12-21 17:28:33 +08:00
chrislu 4b8dcff448 reverting default admin scripts
fix https://github.com/chrislusf/seaweedfs/issues/2525

this new default value was introduced in 2.80
this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.
2021-12-20 13:34:57 -08:00
chrislu b21a67bbe6 add writer pattern object for later use 2021-12-20 11:53:48 -08:00
chrislu 4fd29dad86 remove writeOnly flag 2021-12-20 01:11:43 -08:00
chrislu bc96682760 refactor, change file locations 2021-12-20 01:02:23 -08:00
chrislu 866c2657f0 avoid FUSE cache only for the first 512 bytes 2021-12-19 23:13:36 -08:00
chrislu 0cb9036f66 mount: only cache the first chunk on stream read 2021-12-19 23:06:03 -08:00
chrislu a152f17937 mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
chrislu 85c526c583 s3: bind to a specific IP
fix https://github.com/chrislusf/seaweedfs/issues/2516
2021-12-17 11:34:37 -08:00
chrislu 5eacff9d4f log message adds server name
address https://github.com/chrislusf/seaweedfs/issues/2514#issuecomment-995925733
2021-12-16 10:46:26 -08:00
chrislu 50ddd8c8e2 remove debug messages
fix https://github.com/chrislusf/seaweedfs/issues/2514
2021-12-16 00:58:15 -08:00
chrislu 7210558c7b s3: pass through s3 presigned headers
fix https://github.com/chrislusf/seaweedfs/discussions/2502
2021-12-15 13:18:53 -08:00
chrislu bf4d7affc0 gateway to remote object store: skip replicating multipart upload part files
fix https://github.com/chrislusf/seaweedfs/issues/2509
2021-12-14 19:48:31 -08:00
chrislu 316f326464 add more help message 2021-12-13 13:14:36 -08:00
Konstantin Lebedev 969f513265 disable audit log for s3 statusHandler 2021-12-13 16:08:19 +05:00
Chris Lu ab0c6a84bf
Merge pull request #2503 from kmlebedev/audit_log_nonblocking
Audit log force async
2021-12-13 00:47:16 -08:00
Konstantin Lebedev 34779e8f38 force enable asynchronous I/O sending events to Fluentd 2021-12-13 13:39:39 +05:00
chrislu c3b73ec23b 2.82 2021-12-12 23:25:24 -08:00
chrislu 7bf14452f7 s3: handle Response-Content-Disposition 2021-12-12 23:05:40 -08:00
chrislu 94e5c0a454 skip purging from ec volumes 2021-12-12 22:55:27 -08:00
chrislu a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00