gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6803 commits 28 branches 278 tags 135 MiB
Commit graph

553 commits

Author SHA1 Message Date
chrislu 19555385f7 2.85 2022-01-09 19:30:23 -08:00
chrislu 5799a20f71 2.84 2022-01-02 17:05:19 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158 
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
 2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
 2021-12-30 14:45:27 +01:00
chrislu fb434318e3 dynamically adjust connection timeout 
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:44:39 -08:00
chrislu 5788bf2270 s3: increase timeout limit 
https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:21:02 -08:00
chrislu c935b9669e 2.83 2021-12-25 01:01:34 -08:00
chrislu c3b73ec23b 2.82 2021-12-12 23:25:24 -08:00
chrislu 5ea9715721 2.81 
also sync java client version to SeaweedFS version
 2021-12-05 18:05:24 -08:00
Tanmoy Majumdar ea09fb477a return ' shouldRetry=true' so that filer can retry the failed chunk 2021-12-03 11:54:20 +06:00
Chris Lu 7227cfddf5 2.80 2021-11-29 00:57:08 -08:00
Chris Lu 3a19eea97c allocate memory by slabs 2021-11-27 12:13:00 -08:00
Chris Lu f3c789d662 2.79 2021-11-21 18:40:24 -08:00
Chris Lu 100c654ec3 2.78 2021-11-14 23:29:59 -08:00
Chris Lu 5cf332357b 2.77 2021-11-07 13:52:45 -08:00
Chris Lu fc9e246592 2.76 2021-10-31 18:08:28 -07:00
Chris Lu c9d3fb4a30 2.75 2021-10-24 18:15:59 -07:00
Chris Lu 182f43ae5f 2.74 2021-10-18 14:23:54 -07:00
Chris Lu cd4fa7561b 2.73 2021-10-18 10:47:48 -07:00
Chris Lu 97c963bac9 2.72 2021-10-17 17:40:27 -07:00
Chris Lu 3833dac3f7 continue to read from memory if there is no flush 2021-10-17 13:53:04 -07:00
Chris Lu 8965a53c4d add warning error 2021-10-16 15:57:30 -07:00
Chris Lu 5fd4b05c5e
Merge pull request #2381 from Juneezee/deprecate-ioutil 
refactor: move from io/ioutil to io and os package
 2021-10-13 22:38:58 -07:00
Chris Lu 46a09c6074 adjust test 2021-10-13 22:38:47 -07:00
Eng Zer Jun a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu 4cbd390fbe test: add fail message 2021-10-13 20:42:20 -07:00
Chris Lu 3d586be552 2.71 2021-10-10 22:40:44 -07:00
Chris Lu e4830bd93d go fmt 2021-10-07 21:13:31 -07:00
Chris Lu f3d8232e14 reduce one redis lookup on hot path 2021-10-06 22:01:19 -07:00
Chris Lu 371fead8a5 redis3 using redis native sorted set 2021-10-06 18:18:24 -07:00
Chris Lu 893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu 4ed2994555 use tsMemory to determine whether read from disk or memory 
remove lastFlushTime
 2021-10-04 16:02:56 -07:00
Chris Lu 513fed323a SkipListElementReference can be an empty object 2021-10-04 02:30:44 -07:00
Chris Lu 280ab7f95c add test 2021-10-04 02:30:24 -07:00
Chris Lu 366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu ba7fbac07f rename 2021-10-03 19:23:34 -07:00
Chris Lu e6196cdc50 add name list 2021-10-03 17:54:25 -07:00
Chris Lu a481c4a45e return previous element if visited 2021-10-03 13:50:52 -07:00
Chris Lu 22d8684e88 refactor out listStore 2021-10-03 02:19:21 -07:00
Chris Lu d343b0db57 update value 2021-10-03 01:15:14 -07:00
Chris Lu 4f50f8c2ca insert key and value 2021-10-03 01:07:35 -07:00
Chris Lu 69b84bb771 TestFindGreaterOrEqual 2021-10-02 14:15:49 -07:00
Chris Lu 57e2fd3f9b remove bptree 2021-10-02 14:03:54 -07:00
Chris Lu 4c1741fdbb working skiplist 2021-10-02 14:02:56 -07:00
Chris Lu b6694279d7 Merge branch 'master' into bptree 2021-10-01 16:55:44 -07:00
Chris Lu 1e3fdf366f go fmt 2021-10-01 12:10:24 -07:00
Chris Lu cee4d20bc1 2.70 2021-09-26 17:37:46 -07:00
Chris Lu 603ea2db73 avoid looping forever if there are no more metadata updates 2021-09-26 11:55:27 -07:00
Chris Lu 9887610b54 log tsNs should be processing time 2021-09-26 11:54:13 -07:00