seaweedfs/weed/util
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
..
bounded_tree a better fix 2021-05-09 01:42:19 -07:00
buffered_writer webdav: improve webdav upload speed 2020-12-01 15:32:27 -08:00
chunk_cache add warning error 2021-10-16 15:57:30 -07:00
fla9 adjust help message 2021-03-03 02:12:46 -08:00
grace go fmt 2021-09-01 02:45:42 -07:00
httpdown test passed 2019-11-17 11:40:36 +08:00
log_buffer continue to read from memory if there is no flush 2021-10-17 13:53:04 -07:00
mem allocate memory by slabs 2021-11-27 12:13:00 -08:00
skiplist adjust test 2021-10-13 22:38:47 -07:00
bytes.go minFreeSpace refactored 2021-04-27 10:37:24 +08:00
bytes_test.go minFreeSpace refactored 2021-04-27 10:37:24 +08:00
cipher.go clean up, add test 2020-03-13 23:53:15 -07:00
cipher_test.go clean up, add test 2020-03-13 23:53:15 -07:00
compression.go reduce gzip allocation 2021-08-20 18:38:18 +08:00
compression_stream.go reduce gzip allocation 2021-08-20 18:38:18 +08:00
concurrent_read_map.go directory structure change to work with glide 2016-06-02 18:09:14 -07:00
config.go filer.backup: backup to local directory optionally is incremental 2021-05-25 17:19:20 -07:00
constants.go 2.82 2021-12-12 23:25:24 -08:00
constants_4bytes.go go fmt 2021-09-01 02:45:42 -07:00
constants_5bytes.go go fmt 2021-09-01 02:45:42 -07:00
file_util.go avoid possible metadata subscription data loss 2021-09-25 01:18:44 -07:00
file_util_non_posix.go go fmt 2021-09-01 02:45:42 -07:00
file_util_posix.go go fmt 2021-09-01 02:45:42 -07:00
fullpath.go shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
http_util.go FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 2021-12-30 14:45:27 +01:00
inits.go try showing the first 100 volume ids and an extra ... 2020-05-29 16:15:33 +08:00
inits_test.go try showing the first 100 volume ids and an extra ... 2020-05-29 16:15:33 +08:00
limiter.go skip limiting if limit is zero 2021-04-04 18:38:33 -07:00
minfreespace.go minFreeSpace refactored 2021-04-27 10:37:24 +08:00
minfreespace_test.go minFreeSpace refactored 2021-04-27 10:37:24 +08:00
net_timeout.go filer: linearize timeout for large chunk of data 2020-10-13 14:04:46 -07:00
network.go go fmt 2021-09-14 10:37:06 -07:00
parse.go weed shell: simplify CLI option for filer 2020-04-08 23:57:15 -07:00
queue.go refactoring 2019-12-08 19:44:16 -08:00
queue_unbounded.go filer: fix where deletion can miss under high concurrency 2020-03-20 23:39:32 -07:00
queue_unbounded_test.go filer: avoid possible timeouts for updates and deletions 2020-02-20 15:44:17 -08:00
reflect.go show RemoteVolumes/EcVolumes only if it is not empty 2021-07-06 15:20:18 +08:00
retry.go reset wait time 2021-09-02 19:55:01 -07:00
throttler.go refactor: extract out the write throttler 2019-05-06 13:56:08 -07:00