Commit graph

57 commits

Author SHA1 Message Date
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 2ab0ad24a3 use memory pool 2022-02-26 02:59:19 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
chrislu 5788bf2270 s3: increase timeout limit
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
Tanmoy Majumdar ea09fb477a return ' shouldRetry=true' so that filer can retry the failed chunk 2021-12-03 11:54:20 +06:00
Eng Zer Jun a23bcbb7ec
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu 5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Chris Lu f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu af313dff58 add gateway for easier POST and DELETE blobs 2021-04-10 23:47:47 -07:00
Chris Lu 4b1ed227d1 revert fasthttp changes
related to https://github.com/chrislusf/seaweedfs/issues/1907
2021-03-16 00:33:14 -07:00
Chris Lu 7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Chris Lu 487e435679 adjust http max idle connections per host
related to https://github.com/chrislusf/seaweedfs/issues/1802
2021-02-12 03:47:15 -08:00
Chris Lu 141ce67c09 close http request body 2020-11-30 02:45:00 -08:00
Chris Lu 2bd6fd3bbe remove unused function 2020-11-15 20:15:07 -08:00
Chris Lu 73f934d5de s3: do not close reader too early
fix https://github.com/chrislusf/seaweedfs/issues/1609
2020-11-11 01:49:39 -08:00
Chris Lu 3f7d1d1bf1 Only wait on retryable requests 2020-10-13 00:29:46 -07:00
Chris Lu 4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
Chris Lu 5f55a87101 close http response 2020-08-24 00:32:44 -07:00
Chris Lu 1b3a80dd3d non-fatal error 2020-08-01 13:46:52 -07:00
Chris Lu bbbea8159c http request use gzip if possible 2020-08-01 13:20:52 -07:00
Chris Lu 2f03481cb2 in case when content is not compressed 2020-08-01 13:08:10 -07:00
Chris Lu 3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu e0f5996560 fix "call of Unmarshal passes non-pointer as second argument" 2020-04-17 10:04:41 -07:00
Chris Lu 057722bbf4 return part of the chunk if chunkview is not the full chunk 2020-03-29 00:54:39 -07:00
Chris Lu 2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu 13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu 96c1ae8471 refactoring the close http response 2020-02-14 09:46:36 -08:00
Chris Lu cf5064d702 properly close http response 2020-02-14 09:09:15 -08:00
Chris Lu 33b3bd467c Revert "HEAD operation changes to fasthttp"
This reverts commit 58f126fd27.
2020-02-10 13:43:53 -08:00
Chris Lu 58f126fd27 HEAD operation changes to fasthttp 2020-02-10 09:13:29 -08:00
Chris Lu a80ecbfe84 s3: add s3 copy
fix https://github.com/chrislusf/seaweedfs/issues/1190
2020-01-31 00:11:12 -08:00
Chris Lu 6a5c037099 fix http range requests 2020-01-22 22:59:12 -08:00
Chris Lu 89e16bd2e8 skip error when draining reader
fix https://github.com/chrislusf/seaweedfs/issues/1179
2020-01-08 18:07:07 -08:00
Chris Lu 1fd8926ac7 ignore draining error 2019-12-25 21:39:33 -08:00
divinerapier 4cbb6fa199
feat: drains http body if buffer is too small
Signed-off-by: divinerapier <poriter.coco@gmail.com>
2019-12-26 11:28:43 +08:00
divinerapier 84640d07b7
fix: handle errors for ReadUrl
Signed-off-by: divinerapier <poriter.coco@gmail.com>
2019-12-26 10:07:07 +08:00
Chris Lu 392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu 20d90dea5a filer: avoid hard-coded upload timeout 2019-09-05 01:26:59 -07:00
Chris Lu 98a03b38e5 avoid util package depends on security package 2019-02-16 12:23:35 -08:00
chenwanli 39c7455881 Set http timeout to 5s 2019-01-10 20:21:08 +08:00
Chris Lu 1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu a6cfaba018 able to sync the changes 2018-09-21 01:54:29 -07:00
Chris Lu 865a017936 fix when if buffer is not aligned 2018-09-17 00:27:40 -07:00
Chris Lu 0d98949199 tmp commit 2018-07-27 01:54:45 -07:00
Chris Lu 07e0d13d2d filer support reading multiple chunks, with range support 2018-05-28 05:39:12 -07:00
Chris Lu 8b0718ac92 go vet 2018-05-27 11:56:49 -07:00
Chris Lu c11d84f314 fix reading from a url 2018-05-24 23:20:56 -07:00