Commit graph

1176 commits

Author SHA1 Message Date
Chris Lu 89d84e275b
Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request
Need to exit waiting if request is was canceled
2022-03-24 12:21:44 -07:00
Konstantin Lebedev 0b790d2bbf exclude the replication from the concurrentUploadLimitMB 2022-03-24 13:54:42 +05:00
chrislu 9b04f17555 remove dead code 2022-03-23 23:18:32 -07:00
chrislu 0b5faef612 fix 2022-03-23 23:16:54 -07:00
chrislu 0562fceb99 volume: fail fast if too many concurrent requests, to avoid dead lock due to replication.
fix https://github.com/chrislusf/seaweedfs/issues/2755
2022-03-23 22:53:58 -07:00
zzq09494 9f16df736e fix-filer: calculation error of the method skipCheckParentDirEntry 2022-03-19 09:15:42 +08:00
chrislu 4042fdf3bb rename to skipCheckParentDir
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
2022-03-16 23:55:31 -07:00
zzq09494 40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494 81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494 a6a8892255 Revert "filer: support uploading file without needEnsureParentDir"
This reverts commit a93c4947ba.
2022-03-17 10:27:17 +08:00
zzq09494 a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
Konstantin Lebedev f43c6daeda Need to exit waiting if request is was canceled 2022-03-15 19:55:22 +05:00
banjiaojuhao f28dbbe5c5 [bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user. 2022-03-10 11:40:39 +08:00
chrislu 6d3db4445b buffer for all range requests 2022-03-07 01:56:47 -08:00
chrislu f3bcbeb60a a little optimization 2022-03-07 00:24:59 -08:00
Chris Lu 1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest
filer: support get metadata with resolved manifest chunk
2022-03-07 00:05:46 -08:00
chrislu bb0b784544 minor 2022-03-07 00:04:59 -08:00
banjiaojuhao bfcc9ca808 filer: support metadata with resolved manifest chunk 2022-03-07 15:47:51 +08:00
banjiaojuhao 71f3046841 filer: add back isAppend function 2022-03-07 15:41:07 +08:00
Konstantin Lebedev cf444ebd07 Set default leveldb2 enabled
avoid Filer store is enabled for both leveldb2 and mysql
2022-03-06 18:27:25 +05:00
chrislu f1713c96ae avoid possible runtime error: index out of range [0] with length 0 2022-03-05 21:14:31 -08:00
chrislu a96d4254e9 filer, s3, volume server: a bit memory optimization 2022-03-02 20:15:28 -08:00
chrislu b2a148cb4c use file size as max range 2022-02-26 03:00:08 -08:00
chrislu 28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu e423548673 rename: pass along entry metadata 2022-02-25 02:53:37 -08:00
chrislu 320637dc7a use "mv.from" for moving files 2022-02-23 15:34:42 -08:00
banjiaojuhao 6ab09e9071 filer_http: support uploading file with offset 2022-02-22 00:15:00 +08:00
banjiaojuhao e6126cef62 filer_web: support moving entry 2022-02-20 23:56:23 +08:00
banjiaojuhao 4c30934cd9 filer: support get file entry 2022-02-18 22:52:26 +08:00
Konstantin Lebedev 9ea09cc41c healthz check to avoid drain pod with last replicas 2022-02-16 14:18:36 +05:00
garenchan bd032eabe7 [UPDATE] Make heartbeat interval and election timeout of masters configurable. 2022-02-14 21:09:07 +08:00
root 7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu 433fde4b18 move error to a separate file
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Chris Lu a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics
error metrics for filer and store
2022-02-04 22:35:13 -08:00
chrislu affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
Konstantin Lebedev 9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev 3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Konstantin Lebedev c9952759c4 metrics master is leader 2022-01-24 20:13:07 +05:00
Konstantin Lebedev 28efe31524 new master metrics 2022-01-24 19:09:43 +05:00
chrislu 4a311c7f5e dedup local metadata subscribers
fix https://github.com/chrislusf/seaweedfs/discussions/2542
2022-01-23 16:14:22 -08:00
chrislu e69c374956 minor 2022-01-13 02:01:53 -08:00
chrislu 826a7b307e master: remove hard coded filer settings in master.toml
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
chrislu b8fbf19e9a mount: rename follow POSIX 2022-01-11 03:23:03 -08:00
Chris Lu 42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
chrislu 5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
Sebastian Kurfuerst fcc09cef6f Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst d156d410ef rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
chrislu 67b723f74e Filer Server API support fsync
fix https://github.com/chrislusf/seaweedfs/issues/2528
2021-12-26 17:28:47 -08:00