seaweedfs/weed/s3api
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
..
http s3: pass through s3 presigned headers 2021-12-15 13:18:53 -08:00
policy refactoring 2020-09-20 16:00:01 -07:00
s3_constants go fmt 2020-12-22 02:34:08 -08:00
s3err force enable asynchronous I/O sending events to Fluentd 2021-12-13 13:39:39 +05:00
AmazonS3.xsd add list all my buckets 2018-07-18 02:37:09 -07:00
auth_credentials.go audit log SignatureVersion 2021-12-10 19:40:32 +05:00
auth_credentials_subscribe.go save updated lastTsNs 2021-10-17 02:01:20 -07:00
auth_credentials_test.go break import cycle 2020-12-07 00:29:17 -08:00
auth_signature_v2.go s3: add support for PostPolicy 2020-09-19 20:14:19 -07:00
auth_signature_v4.go refactor: move from io/ioutil to io and os package 2021-10-14 12:27:58 +08:00
auto_signature_v4_test.go refactor: move from io/ioutil to io and os package 2021-10-14 12:27:58 +08:00
chunked_reader_v4.go audit log 2021-12-07 12:15:48 +05:00
custom_types.go weed s3: generate time format acceptable by aws cli s3 2019-04-01 16:13:29 -07:00
filer_multipart.go s3: copy object API needs to escape special characters 2021-12-04 23:24:53 -08:00
filer_multipart_test.go refactor 2021-06-10 21:50:21 -07:00
filer_util.go go fmt 2021-07-21 14:38:12 -07:00
filer_util_tags.go adjust s3 header file. 2020-10-29 16:05:40 +08:00
README.txt add list all my buckets 2018-07-18 02:37:09 -07:00
s3api_auth.go go fmt 2020-02-26 16:52:57 -08:00
s3api_bucket_handlers.go audit log 2021-12-07 12:15:48 +05:00
s3api_bucket_handlers_test.go refactor 2021-06-10 21:50:21 -07:00
s3api_handlers.go audit log config 2021-12-07 18:20:52 +05:00
s3api_object_copy_handlers.go FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 2021-12-30 14:45:27 +01:00
s3api_object_handlers.go FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 2021-12-30 14:45:27 +01:00
s3api_object_handlers_postpolicy.go audit log config 2021-12-07 18:20:52 +05:00
s3api_object_multipart_handlers.go audit log 2021-12-07 12:15:48 +05:00
s3api_object_skip_handlers.go AclHandlers 2021-10-11 15:03:56 +05:00
s3api_object_tagging_handlers.go audit log config 2021-12-07 18:20:52 +05:00
s3api_objects_list_handlers.go audit log 2021-12-07 12:15:48 +05:00
s3api_objects_list_handlers_test.go refactor 2021-06-10 21:50:21 -07:00
s3api_policy.go GetBucketLifecycleConfigurationHandler 2021-10-14 01:35:33 +05:00
s3api_server.go FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 2021-12-30 14:45:27 +01:00
s3api_status_handlers.go disable audit log for s3 statusHandler 2021-12-13 16:08:19 +05:00
s3api_test.go refactor 2021-06-10 21:50:21 -07:00
s3api_xsd_generated.go AclHandlers 2021-10-11 15:03:56 +05:00
stats.go s3: avoid possible attacks by version 2021-11-08 17:47:56 -08:00
tags.go fix Tagging add xmlns 2021-10-20 19:40:22 +05:00
tags_test.go fix Tagging test 2021-10-20 19:12:00 +05:00

see https://blog.aqwari.net/xml-schema-go/

1. go get aqwari.net/xml/cmd/xsdgen
2. xsdgen -o s3api_xsd_generated.go -pkg s3api AmazonS3.xsd