Commit graph

110 commits

Author SHA1 Message Date
Konstantin Lebedev 49f058da85 fix s3 tests:
ranged_request_invalid_range
ranged_request_empty_object
2022-06-08 13:25:13 +05:00
Konstantin Lebedev c07820178f fix s3 tests
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
2022-06-07 14:43:10 +05:00
chrislu 27732ecfa4 move s3 related constants from package http to s3_constants 2022-05-30 22:57:41 -07:00
chrislu 596c3860ca use final destination to resolve fs configuration
related to https://github.com/chrislusf/seaweedfs/issues/3075
2022-05-30 22:47:26 -07:00
Konstantin Lebedev af562e1a20 rm log info 2022-05-30 23:26:26 +05:00
Konstantin Lebedev 26f3646961 show fake dirs
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-30 15:03:27 +05:00
justin 3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
root b7e34016d4 fix error response in case of wrong expire time format 2022-04-02 11:01:33 +08:00
Konstantin Lebedev 9231971c88 fix test_s3.test_multi_object_delete_key_limit and
test_multi_objectv2_delete_key_limit
2022-03-29 16:55:09 +05:00
chrislu da3d330616 s3 and filer transport using unix domain socket instead of tcp 2022-03-07 02:00:14 -08:00
chrislu 0cb17b45b1 refactoring 2022-03-07 01:59:01 -08:00
chrislu a96d4254e9 filer, s3, volume server: a bit memory optimization 2022-03-02 20:15:28 -08:00
Lapshinn Vitaly 6bdc274d4d add s3api error for copy in file, not directory 2022-02-04 03:28:37 +03:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
chrislu 5788bf2270 s3: increase timeout limit
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
chrislu 9f9ef1340c use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
chrislu 7210558c7b s3: pass through s3 presigned headers
fix https://github.com/chrislusf/seaweedfs/discussions/2502
2021-12-15 13:18:53 -08:00
Konstantin Lebedev 34779e8f38 force enable asynchronous I/O sending events to Fluentd 2021-12-13 13:39:39 +05:00
Konstantin Lebedev 98251fe16a non blocking audit log 2021-12-09 19:47:16 +05:00
Konstantin Lebedev 10678cde81 audit log config 2021-12-07 18:20:52 +05:00
Konstantin Lebedev 4ec8715f20 audit log 2021-12-07 12:15:48 +05:00
Chris Lu f17fa400d5 refactoring 2021-10-31 18:05:34 -07:00
Chris Lu 006b4ec0f8 refactoring 2021-10-31 18:02:08 -07:00
Chris Lu a6bb509cb7 adjust mime detection 2021-10-18 10:47:39 -07:00
Konstantin Lebedev dd2cc1acaf fix https://github.com/chrislusf/seaweedfs/issues/2387 2021-10-18 16:27:57 +05:00
Konstantin Lebedev 8de7915950 s3 mime detect 2021-10-14 15:03:11 +05:00
Eng Zer Jun a23bcbb7ec
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu 59dd271734 more glog 2021-09-19 00:29:51 -07:00
Chris Lu 4cbba2b1c6 add more glog to s3 2021-09-19 00:28:22 -07:00
Chris Lu 71175461ef add glog for s3 handlers 2021-09-19 00:18:59 -07:00
Chris Lu e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu 8b382a8209 refactor 2021-06-10 21:50:21 -07:00
Chris Lu 33b87244ef refactoring 2021-06-10 16:54:36 -07:00
Chris Lu 2b60e2abb1 only disallow streaming signed when no auth enabled
fix https://github.com/chrislusf/seaweedfs/issues/2101
2021-05-31 01:03:04 -07:00
Konstantin Lebedev b612d5aebd s3 test get w/ If-Match: bogus ETag 2021-05-24 16:59:44 +05:00
Konstantin Lebedev 69c768870b - object write cache control
- object write expires
2021-05-24 15:43:55 +05:00
Konstantin Lebedev 84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Chris Lu 431684798b s3: add errors if requests are signed by no authentication is setup
fix https://github.com/chrislusf/seaweedfs/issues/2075
2021-05-21 14:08:47 -07:00
Konstantin Lebedev 03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader"
This reverts commit ac71117e
2021-05-20 11:45:21 +05:00
Chris Lu ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Merlin Gaillard 4d4acc715e s3api: handle 304 response code from filer 2021-04-09 12:13:19 +02:00
Konstantin Lebedev bf94149920 add line 2021-03-17 23:41:34 +05:00
Konstantin Lebedev c5705e7a35 dir + slash key does not exist
https://github.com/chrislusf/seaweedfs/issues/1917
2021-03-17 23:40:42 +05:00
Konstantin Lebedev 3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Chris Lu d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-11 11:01:44 -08:00
Chris Lu 79280e1513 escape special characters
fix https://github.com/chrislusf/seaweedfs/issues/1884
2021-03-11 09:49:40 -08:00
Chris Lu ca546f47e1 s3: escape object key if containing special characters
fix https://github.com/chrislusf/seaweedfs/issues/1884
2021-03-10 13:19:28 -08:00
Konstantin Lebedev bb3af2d70b no directoriesWithDeletion 2021-03-10 19:52:41 +05:00