seaweedfs/weed/server
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
..
filer_ui remove QR code for mobile upload 2021-09-01 16:59:37 -07:00
master_ui extract embed html of master/volume/filer ui to separate files 2021-07-05 18:09:44 +08:00
volume_server_ui show RemoteVolumes/EcVolumes only if it is not empty 2021-07-06 15:20:18 +08:00
common.go s3: pass through s3 presigned headers 2021-12-15 13:18:53 -08:00
common_test.go jwt check the base file id 2019-03-03 10:17:44 -08:00
filer_grpc_server.go filer server: allow upload file to specific dataNode 2021-12-22 21:57:26 +08:00
filer_grpc_server_kv.go filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
filer_grpc_server_remote.go filer server: allow upload file to specific dataNode 2021-12-22 21:57:26 +08:00
filer_grpc_server_rename.go rename operation returns the correct old parent path 2021-12-10 00:02:57 +08:00
filer_grpc_server_sub_meta.go add delay if need to resume from disk 2021-12-09 13:13:05 -08:00
filer_server.go FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 2021-12-30 14:45:27 +01:00
filer_server_handlers.go FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 2021-12-30 14:45:27 +01:00
filer_server_handlers_proxy.go adjust http max idle connections per host 2021-02-12 03:47:15 -08:00
filer_server_handlers_read.go s3: pass through s3 presigned headers 2021-12-15 13:18:53 -08:00
filer_server_handlers_read_dir.go remove QR code for mobile upload 2021-09-01 16:59:37 -07:00
filer_server_handlers_tagging.go filer: delete specific tags 2021-05-02 21:53:43 -07:00
filer_server_handlers_write.go filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
filer_server_handlers_write_autochunk.go s3: handle Response-Content-Disposition 2021-12-12 23:05:40 -08:00
filer_server_handlers_write_cipher.go refactoring 2021-09-06 16:20:49 -07:00
filer_server_handlers_write_upload.go refactor: move from io/ioutil to io and os package 2021-10-14 12:27:58 +08:00
filer_server_rocksdb.go go fmt 2021-09-01 02:45:42 -07:00
master_grpc_server.go send peers info to filers 2021-11-06 04:07:38 -07:00
master_grpc_server_admin.go add lock messages 2021-12-10 13:24:38 -08:00
master_grpc_server_cluster.go auto updated filer peer list 2021-11-06 14:23:35 -07:00
master_grpc_server_collection.go change server address from string to a type 2021-09-12 22:47:52 -07:00
master_grpc_server_volume.go rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
master_server.go reverting default admin scripts 2021-12-20 13:34:57 -08:00
master_server_handlers.go rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
master_server_handlers_admin.go refactor 2021-10-05 00:40:04 -07:00
master_server_handlers_ui.go adjust data type 2021-08-12 17:54:34 -07:00
raft_server.go change server address from string to a type 2021-09-12 22:47:52 -07:00
raft_server_handlers.go change server address from string to a type 2021-09-12 22:47:52 -07:00
volume_grpc_admin.go volume: support concurrent download data size limit 2021-08-08 23:25:16 -07:00
volume_grpc_batch_delete.go volume: support concurrent download data size limit 2021-08-08 23:25:16 -07:00
volume_grpc_client_to_master.go change server address from string to a type 2021-09-12 22:47:52 -07:00
volume_grpc_copy.go prevent nil response 2021-11-15 08:46:01 -08:00
volume_grpc_copy_incremental.go fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
volume_grpc_erasure_coding.go volume copy: stream out copying progress and avoid grpc request timeout 2021-10-24 02:52:56 -07:00
volume_grpc_query.go volume: support concurrent download data size limit 2021-08-08 23:25:16 -07:00
volume_grpc_read_all.go volume stream read skips deleted content 2021-09-27 03:07:44 -07:00
volume_grpc_read_write.go remove imports 2021-08-07 16:20:17 -07:00
volume_grpc_remote.go go fmt 2021-09-14 10:37:06 -07:00
volume_grpc_tail.go change server address from string to a type 2021-09-12 22:47:52 -07:00
volume_grpc_tier_download.go volume: add "-dir.idx" option for separate index storage 2020-11-27 03:17:10 -08:00
volume_grpc_tier_upload.go cloud tier: remove tagging since not all s3 vendors support this 2021-10-29 12:39:19 -07:00
volume_grpc_vacuum.go go fmt 2021-11-02 23:39:16 -07:00
volume_server.go upgrade protoc to 3.17.3 2021-12-05 00:42:25 -08:00
volume_server_handlers.go Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
volume_server_handlers_admin.go support customizable disk type 2021-02-13 15:42:42 -08:00
volume_server_handlers_helper.go directory structure change to work with glide 2016-06-02 18:09:14 -07:00
volume_server_handlers_read.go mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
volume_server_handlers_ui.go change server address from string to a type 2021-09-12 22:47:52 -07:00
volume_server_handlers_write.go move some volume lookup operations to grpc 2021-08-12 20:33:00 -07:00
volume_server_tcp_handlers_write.go trim out trailing return character 2021-03-06 14:26:27 -08:00
webdav_server.go go fmt 2021-09-14 10:37:06 -07:00