seaweedfs/weed/command
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
..
scaffold FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 2021-12-30 14:45:27 +01:00
autocomplete.go minor changes 2021-08-02 11:49:40 -07:00
backup.go volume vacuum: avoid timeout with streaming progress report 2021-10-24 01:55:34 -07:00
benchmark.go Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 2021-11-05 17:52:15 -07:00
command.go add command filre.remote.gateway 2021-09-15 22:48:04 -07:00
compact.go volume vacuum: avoid timeout with streaming progress report 2021-10-24 01:55:34 -07:00
download.go refactor: move from io/ioutil to io and os package 2021-10-14 12:27:58 +08:00
export.go adjust logging 2021-04-14 10:04:26 -07:00
filer.go s3: bind to a specific IP 2021-12-17 11:34:37 -08:00
filer_backup.go change server address from string to a type 2021-09-12 22:47:52 -07:00
filer_cat.go change server address from string to a type 2021-09-12 22:47:52 -07:00
filer_copy.go refactor: move from io/ioutil to io and os package 2021-10-14 12:27:58 +08:00
filer_meta_backup.go change server address from string to a type 2021-09-12 22:47:52 -07:00
filer_meta_tail.go filer.meta.tail: add example to send metadata to elastic search 2021-10-31 18:01:33 -07:00
filer_remote_gateway.go go fmt 2021-09-19 12:06:15 -07:00
filer_remote_gateway_buckets.go gateway to remote object store: skip replicating multipart upload part files 2021-12-14 19:48:31 -08:00
filer_remote_sync.go add more help message 2021-12-13 13:14:36 -08:00
filer_remote_sync_dir.go adjust error log 2021-11-28 22:06:17 -08:00
filer_replication.go filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
filer_sync.go change server address from string to a type 2021-09-12 22:47:52 -07:00
filer_sync_std.go go fmt 2021-09-01 02:45:42 -07:00
filer_sync_windows.go filer.backup: escape colon from path on windows 2021-05-29 06:45:27 -07:00
fix.go do not idx file format 2020-09-12 12:42:36 -07:00
fuse.go Fix error handling 2021-07-21 22:48:04 +09:00
iam.go change server address from string to a type 2021-09-12 22:47:52 -07:00
imports.go removing tikv to resolve "go mod tidy" problem 2021-10-10 19:27:02 -07:00
master.go Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 2021-11-05 17:52:15 -07:00
master_follower.go Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 2021-11-05 17:52:15 -07:00
mount.go adjust help message 2021-05-21 01:38:57 -07:00
mount_darwin.go fuse: check mount point available before do runmount 2019-11-27 22:56:44 +08:00
mount_freebsd.go fuse: check mount point available before do runmount 2019-11-27 22:56:44 +08:00
mount_linux.go FUSE mount: make "nonempty" optional 2020-03-21 19:14:25 -07:00
mount_notsupported.go go fmt 2021-09-01 02:45:42 -07:00
mount_std.go change server address from string to a type 2021-09-12 22:47:52 -07:00
msg_broker.go change server address from string to a type 2021-09-12 22:47:52 -07:00
s3.go s3: bind to a specific IP 2021-12-17 11:34:37 -08:00
scaffold.go refactor: move from io/ioutil to io and os package 2021-10-14 12:27:58 +08:00
scaffold_test.go fix test 2019-11-28 18:44:27 -08:00
server.go s3: bind to a specific IP 2021-12-17 11:34:37 -08:00
shell.go better printout 2021-11-02 23:45:47 -07:00
upload.go change server address from string to a type 2021-09-12 22:47:52 -07:00
version.go inject git version into build 2020-06-02 00:10:38 -07:00
volume.go use default 10000 for grpc port 2021-09-20 14:05:59 -07:00
volume_test.go Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
webdav.go go fmt 2021-09-14 10:37:06 -07:00