gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9295 commits 28 branches 278 tags 135 MiB
Commit graph

463 commits

Author SHA1 Message Date
chrislu 67814a5c79 refactor and fix strings.Split 2022-08-07 01:34:32 -07:00
Konstantin Lebedev 4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
chrislu 90db4b5a44 minor 2022-08-03 00:10:47 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
Chris Lu abeb7f65f8
Merge pull request #3314 from shichanglin5/fix_ETag 
File upload should succeed should return `ETag` instead of `Etag`
 2022-07-14 15:35:41 -07:00
LHHDZ c08f6d0ff7 File upload should succeed should return ETag instead of Etag 
at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`
 2022-07-14 19:45:01 +08:00
guosj ab1b9697e6 supplement check duplicate accesskey 2022-07-13 17:28:20 +08:00
chrislu 69ef6459c3 S3 ListObjectsV2 fix pagination under bucket root 
fix https://github.com/chrislusf/seaweedfs/issues/3166
 2022-07-06 02:28:34 -07:00
Konstantin Lebedev 3d5f1cbc5e fix pagination 2022-07-01 14:17:49 +05:00
Konstantin Lebedev bcbdc4cb37 use const multipart uploads folder 
avoid error bucket NotEmpty if multipart uploads folder exist
 2022-06-29 16:21:16 +05:00
Chris Lu c6e6e303db
Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy 
validate tags on copy object and add regex for validating tags
 2022-06-29 01:37:14 -07:00
Lapshin Vitaliy 7c3d9d0535 remove println 2022-06-29 10:43:02 +03:00
Lapshin Vitaliy 61b2e3f9a2 fix return 2022-06-28 18:31:46 +03:00
Lapshin Vitaliy 606252472c fix return 2022-06-28 18:24:03 +03:00
Lapshin Vitaliy e969370913 fix parseTagging function and add test for validate tags 2022-06-28 14:48:55 +03:00
Lapshin Vitaliy 1a5981d583 fix test 2022-06-27 18:46:21 +03:00
Lapshin Vitaliy d7c3493d15 Merge branch 'master' into bufix/validate-tags-on-copy 2022-06-27 13:53:57 +03:00
chrislu b9f7b6fb9a adjust log message 2022-06-26 23:12:16 -07:00
石昌林 15bfc60a88 Remove the collection query param of s3api and let the collection be determined by the filer 2022-06-27 13:47:53 +08:00
石昌林 be5c901bea Fix global counter key is incorrect & Add read lock when reading counter map to avoid memory problems 2022-06-20 19:16:53 +08:00
Lapshin Vitaliy 5f5fd0bc48 validate tags on copy object and add regex for validating tags 2022-06-20 11:32:58 +03:00
Chris Lu 9e2d6e897e
Merge pull request #3198 from guol-fnst/fix_statuscode 
AbortMultipartUploadHandler should return 204 instead of 200
 2022-06-19 22:57:13 -07:00
LHHDZ 8db9f13bc6
Merge branch 'master' into circuit_breaker 2022-06-20 13:42:18 +08:00
guol-fnst 1d77deccd0 AbortMultipartUploadHandler should return 204 instead of 200 
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
 2022-06-20 13:26:22 +08:00
石昌林 9e036df356 remove go.uber.org/atomic 2022-06-20 12:35:29 +08:00
石昌林 3dd60529c5 some code optimizations 2022-06-17 19:07:39 +08:00
石昌林 37df209195 add some unit tests and some code optimizes 2022-06-17 17:11:18 +08:00
zzq09494 62f74f5e3c add bucket label to s3 prometheus metrics 2022-06-16 13:21:25 +08:00
石昌林 78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations 
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
 2022-06-15 21:07:55 +08:00
shichanglin5 f01dd27752 fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup 2022-06-13 13:23:26 +08:00
Konstantin Lebedev 49f058da85 fix s3 tests: 
ranged_request_invalid_range
ranged_request_empty_object
 2022-06-08 13:25:13 +05:00
Chris Lu f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud 2022-06-07 08:56:55 -07:00
Konstantin Lebedev c07820178f fix s3 tests 
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
 2022-06-07 14:43:10 +05:00
creeew 02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
Konstantin Lebedev b86628f85d fix show empty dir 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-06-01 20:10:52 +05:00
chrislu 27732ecfa4 move s3 related constants from package http to s3_constants 2022-05-30 22:57:41 -07:00
chrislu f4a6da6cb2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-30 22:47:29 -07:00
chrislu 596c3860ca use final destination to resolve fs configuration 
related to https://github.com/chrislusf/seaweedfs/issues/3075
 2022-05-30 22:47:26 -07:00
Chris Lu ca01ce0524
Merge pull request #3122 from chrislusf/filer-sync-with-peers 
Filer bootstrap from peers
 2022-05-30 21:38:31 -07:00
chrislu a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
Chris Lu 730c9cf601
Merge pull request #3114 from kmlebedev/nextcloud 
Create folders in s3 via nextcloud
 2022-05-30 13:10:10 -07:00
Konstantin Lebedev af562e1a20 rm log info 2022-05-30 23:26:26 +05:00
Konstantin Lebedev 62e5e3822c list self dir 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-05-30 18:09:08 +05:00
Konstantin Lebedev 26f3646961 show fake dirs 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-05-30 15:03:27 +05:00
chrislu b8f3db0d46 s3: keep auth enabled in case identities are set to empty 
fix https://github.com/chrislusf/seaweedfs/issues/3084
 2022-05-29 19:00:03 -07:00
Konstantin Lebedev 54356211c2 list empty folder 2022-05-27 13:17:18 +05:00
chrislu 866fed1eab avoid grpc name conflict 
fix https://github.com/chrislusf/seaweedfs/issues/3055
 2022-05-15 21:31:29 -07:00
chrislu 4076d74813 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-15 00:43:39 -07:00
chrislu 29198720f2 s3: add grpc server to accept configuration changes 2022-05-15 00:43:37 -07:00
shichanglin5 688d55488c test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go 2022-05-14 10:40:29 +08:00
shichanglin5 1166dead00 fix the problem of metadata and tagging loss when files are copied 
by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)
 2022-05-13 19:46:20 +08:00
chrislu f7366a9668 skip unix socket mode for windows 
fix https://github.com/chrislusf/seaweedfs/issues/3013
 2022-05-04 10:14:34 -07:00
chrislu f17cd0d5cd return false if not found 
fix https://github.com/chrislusf/seaweedfs/issues/3011
 2022-05-03 07:18:34 -07:00
chrislu 00c1dfec4f go fmt 2022-05-01 23:16:29 -07:00
Konstantin Lebedev 306cf70c4a avoid empty listMultipartUploads response 2022-04-27 19:27:44 +05:00
leyou240 89eb87c1d1
Merge branch 'master' into slices.SortFunc 2022-04-18 10:39:29 +08:00
justin 3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
guol-fnst 44d810d163 rename functions and remove uncessary check 2022-04-12 13:40:53 +08:00
guol-fnst 180aa88a92 check uploadid using object name hash string 2022-04-12 11:04:38 +08:00
guol-fnst 2232cfb5b7 Check object name and uploadID when processing multipart uploading 2022-04-11 19:53:44 +08:00
root f15a737a0a Add "Location:" in response when creating bucket 
according to "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html"
 2022-04-06 13:56:27 +08:00
root b7e34016d4 fix error response in case of wrong expire time format 2022-04-02 11:01:33 +08:00
Konstantin Lebedev 691ed50957 use constant for ErrBucketNotEmpty 2022-03-31 08:44:48 +05:00
Konstantin Lebedev b6a2b43efe allowDeleteBucketNotEmpty 2022-03-30 22:46:13 +05:00
Konstantin Lebedev a8b15c676c s3 test bucket delete nonempty 2022-03-30 21:06:26 +05:00
Chris Lu ce8ef60c59
Merge pull request #2840 from kmlebedev/fix_multi_object_delete_key_limit 
fix test_s3.test_multi_object_delete_key_limit and test_multi_objectv2_delete_key_limit
 2022-03-30 01:27:13 -07:00
Konstantin Lebedev 6317f6a8a9 fix Description 2022-03-30 13:04:15 +05:00
Konstantin Lebedev 993283bb1d revert sort.Search 2022-03-29 19:53:12 +05:00
Konstantin Lebedev 5b90a39954 fix s3 tests: 
multipart_upload_incorrect_etag
multipart_resend_first_finishes_last
 2022-03-29 19:21:09 +05:00
Konstantin Lebedev 9231971c88 fix test_s3.test_multi_object_delete_key_limit and 
test_multi_objectv2_delete_key_limit
 2022-03-29 16:55:09 +05:00
chrislu e48764be75 s3: multipart upload verifies uploaded parts 2022-03-23 01:05:14 -07:00
chrislu 18ae8943ef better clean up upload directory 2022-03-23 01:03:51 -07:00
chrislu da3d330616 s3 and filer transport using unix domain socket instead of tcp 2022-03-07 02:00:14 -08:00
chrislu 0cb17b45b1 refactoring 2022-03-07 01:59:01 -08:00
chrislu a96d4254e9 filer, s3, volume server: a bit memory optimization 2022-03-02 20:15:28 -08:00
Chris Lu e1d3dd5e18
Merge pull request #2712 from guo-sj/correct_comments 2022-02-26 09:16:12 -08:00
guosj 82cad5e330
correct comments 2022-02-26 22:44:26 +08:00
Chris Lu 9873bae115
Merge pull request #2706 from guo-sj/fix_log_info 2022-02-24 22:48:19 -08:00
guosj 121b31f750 fix incorrect log information 2022-02-25 13:41:20 +08:00
chrislu 61811dc2f1 comments 2022-02-18 22:14:40 -08:00
chrislu 9014d00fd0 Revert "s3: listObjectParts return ErrNoSuchUpload if does not exist" 
This reverts commit 6cf2e7d493.
 2022-02-18 20:54:54 -08:00
chrislu 81f86c381d s3: avoid nil response 
fix https://github.com/chrislusf/seaweedfs/issues/2636
 2022-02-07 11:37:59 -08:00
Chris Lu 247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors 
add s3api error for copy in file, not directory
 2022-02-04 22:34:34 -08:00
zerospiel f3364fec99 weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting 
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).

Added generic comment rule of traversing methods
 2022-02-04 15:14:48 +03:00
Lapshinn Vitaly 6bdc274d4d add s3api error for copy in file, not directory 2022-02-04 03:28:37 +03:00
chrislu 6cf2e7d493 s3: listObjectParts return ErrNoSuchUpload if does not exist 
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
    "Initiator": null,
    "Owner": null,
    "StorageClass": "STANDARD"
}

If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
 2022-02-03 12:34:16 -08:00
zerospiel b54a65ba5a weed/s3api: added new bucket handlers for more compatibility with AWS S3 
Protocol

Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.

Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment

Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
 2022-02-03 17:17:05 +03:00
chrislu 77362700e1 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix #2593
 2022-01-18 12:04:40 -08:00
chrislu c87b8f4c30 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix https://github.com/chrislusf/seaweedfs/issues/2593
 2022-01-17 23:09:37 -08:00
Konstantin Lebedev edb753ab4d https://github.com/chrislusf/seaweedfs/issues/2583 2022-01-12 16:04:59 +05:00
chrislu e76105e2ab fix auth permission checking 2022-01-03 21:05:20 -08:00
chrislu a7887166cf wildcard prefix to restrict access to directories in s3 bucket 
https://github.com/chrislusf/seaweedfs/discussions/2551
 2022-01-03 15:39:36 -08:00
Chris Lu 42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158 
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
 2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
 2021-12-30 14:45:27 +01:00
chrislu 5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
chrislu 5788bf2270 s3: increase timeout limit 
https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:21:02 -08:00
chrislu 9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00