gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6000 commits 28 branches 278 tags 135 MiB
Commit graph

260 commits

Author SHA1 Message Date
Chris Lu e9d8201925 s3: batch purge empty folders 2021-01-28 13:20:06 -08:00
Chris Lu 00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu 6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu 096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu ccbdb38c89 s3: fix for listing objects if more than 1000 in the folder 
many existing tools have max key set to 1000
 2021-01-24 00:27:52 -08:00
Chris Lu 389426bbb7 s3: listing may repeat on the edge 2021-01-17 22:59:31 -08:00
Chris Lu 8a7302ce37 s3: KeyCount is zero 
fix https://github.com/chrislusf/seaweedfs/issues/1741
 2021-01-11 03:50:45 -08:00
Chris Lu e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu 6d2c979999 s3: fix regression 
fix https://github.com/chrislusf/seaweedfs/issues/1707
 2020-12-27 21:09:45 -08:00
Chris Lu 561a2ff0bc s3: add option for "alllowEmptyFolder" 2020-12-27 10:50:27 -08:00
Chris Lu aa020ee7e7 s3: restore V2 auth type 
The added headers were included in calculating the signatures, failing the comparison
 2020-12-26 22:01:16 -08:00
Chris Lu 090f85be4b s3: support config action Admin:bucket 2020-12-25 00:38:56 -08:00
Chris Lu 8e48a235e2 s3: avoid looping if the directory is empty 
fix https://github.com/chrislusf/seaweedfs/issues/1701
 2020-12-24 11:34:52 -08:00
Chris Lu 1620de08ae added more logs 2020-12-24 01:52:06 -08:00
Chris Lu a09cd28986 add debug messages 2020-12-24 01:42:15 -08:00
Chris Lu 3be3635799 just log error when checking empty dir failed 2020-12-23 14:36:48 -08:00
Chris Lu 0ca9d89589 s3: break loop if error 
fix #1701
 2020-12-23 14:34:59 -08:00
Chris Lu 4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu 5c465293e9 correctly determine whether a folder is empty 
avoid edge cases that deleting the folder if previous 32 directories are all empty

early terminate if one file is found
 2020-12-12 16:19:29 -08:00
Chris Lu f930c713fc more efficient recursion 2020-12-12 13:25:19 -08:00
Chris Lu 03637d6f57 s3: move "delete-directory-if-empty" to read time 
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
 2020-12-12 03:38:34 -08:00
Chris Lu 26731694f8 s3: use static configuration by default 
So that users can still use the previous configuration files.

If leave it empty, s3 will try to use the version from filer
 2020-12-10 00:59:04 -08:00
Chris Lu 765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu 02a160c3fd s3: recursively iterate all sub folders 
fix https://github.com/chrislusf/seaweedfs/issues/1656
 2020-12-09 16:47:34 -08:00
Chris Lu fce8803087 break import cycle 2020-12-07 00:29:17 -08:00
Chris Lu eed87791b7 s3: subscribe to s3.configure changes 2020-12-07 00:10:29 -08:00
Chris Lu 01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Konstantin Lebedev a26f1b2040 new pkg s3iam 2020-11-26 01:30:11 +05:00
Konstantin Lebedev 6206737df2 s3 configure 2020-11-25 21:02:31 +05:00
Konstantin Lebedev 27e73de797 Merge branch 'upstream_master' into store_s3cred 
# Conflicts:
#	weed/s3api/filer_util.go
 2020-11-19 18:16:44 +05:00
Chris Lu 781585b195 remove unnecessary checking 2020-11-17 17:00:06 -08:00
Chris Lu 7c9e592c2d s3: avoid nil resp when having error 
fix https://github.com/chrislusf/seaweedfs/issues/1622
 2020-11-17 11:23:17 -08:00
Chris Lu 0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
ruitao.liu a9990a1dc6 adjust check bucket if exist or has access. 2020-11-13 17:13:20 +08:00
Chris Lu e6333da65a enable admin to access all buckets 2020-11-12 13:57:54 -08:00
Chris Lu 7094492428 fix ErrNotFound 2020-11-12 13:30:46 -08:00
Chris Lu c6a0704019 rename function 2020-11-12 13:30:08 -08:00
Chris Lu ed3b0f81c6
Merge pull request #1613 from taozix/master 
check permission for bucket delete/head.
 2020-11-12 13:25:28 -08:00
ruitao.liu c4f0fd6e1b skip if entry.Extended map is nil. 2020-11-12 17:59:31 +08:00
ruitao.liu ab966410d2 return NoSuchBucket instead of InternalError delete non-existed bucket. 2020-11-12 16:44:16 +08:00
ruitao.liu e06676f007 check permission for bucket delete/head. 2020-11-12 16:15:59 +08:00
Chris Lu 0fbc27b640
Merge pull request #1612 from taozix/master 
add owner attr for bucket.
 2020-11-11 22:19:47 -08:00
ruitao.liu d7cc0498e0 check if bucket already exists. 2020-11-12 14:11:03 +08:00
ruitao.liu dbba8cb57e fix if nil map in entry. 2020-11-12 11:50:19 +08:00
Chris Lu 86cdb2a3e5 adjust logs 2020-11-11 02:01:24 -08:00
ruitao.liu 5b636b3242 Add bucket owner attr. 2020-11-11 16:20:59 +08:00
Konstantin Lebedev 6cd6ff6962 avoid blank response for 404 2020-11-04 22:54:47 +05:00
Konstantin Lebedev e1190b3224 load S3 config from filer 
https://github.com/chrislusf/seaweedfs/issues/1500
 2020-11-03 21:45:56 +05:00
ruitao.liu 22a9ea0512 adjust s3 header file. 2020-10-29 16:05:40 +08:00
ruitao.liu b917be7955 S3 bucket list, response with uploaded storageclass. 2020-10-29 14:57:19 +08:00
Chris Lu 6da87720eb shorter tag prefix 2020-10-27 15:33:26 -07:00
ruitao.liu 8766ca1b95 rename s3 tag prefix. 2020-10-27 17:33:24 +08:00
ruitao.liu 19026ae55d return x-amz-tag-count header when GET object. 2020-10-27 16:49:31 +08:00
Chris Lu da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Konstantin Lebedev 05b5f12f2e multiplate DomainNames through comma 2020-10-21 20:48:51 +05:00
Chris Lu b3aa2fab9a s3: report error to s3 when updating an object but it is already a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1545
 2020-10-20 10:25:16 -07:00
Chris Lu ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Chris Lu 723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu 9e7a2772b1 s3: only admin can list all buckets 2020-10-08 10:12:09 -07:00
Chris Lu 62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu e91b9c85a2 s3: Added support for "List" action in weed s3 -config=... in the config file. 
fix https://github.com/chrislusf/seaweedfs/issues/1511
 2020-10-07 23:22:35 -07:00
Chris Lu f781cce500 s3: support object tagging 
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
 2020-10-02 22:21:51 -07:00
Konstantin Lebedev 68463e92c1 add status code in S3RequestCounter 2020-10-01 00:59:39 +05:00
limd 402aef8f30 s3: 
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory

reference git revsion number: b41b7ea4d0
 2020-09-25 14:37:02 +08:00
Chris Lu 043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
Chris Lu dbf5327b98 s3: handle response-content-encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1487
 2020-09-24 18:09:52 -07:00
limd d506080c36 rollback 2020-09-23 14:29:53 +08:00
limd 8f9f29b773 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:33:13 +08:00
limd 1892677b22 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:15:06 +08:00
limd a99f63cb4d 1.add S3 copy directory function 
2.fixed the problem of empty directory when S3 deleted the directory
 2020-09-23 11:26:01 +08:00
Chris Lu 9cdbfc1a49 refactor 2020-09-21 17:34:38 -07:00
Chris Lu b9c1f3e9de s3: fixes for list multipart upload 2020-09-21 10:51:24 -07:00
Chris Lu 9a3b564508 adjust printout 2020-09-20 16:01:56 -07:00
Chris Lu 62563a895a refactoring 2020-09-20 16:00:01 -07:00
Chris Lu d013d09a9b adjust logging 2020-09-20 15:38:59 -07:00
Chris Lu dc4d2145ff open up 2020-09-19 20:23:12 -07:00
Chris Lu 29abe980df s3: add support for PostPolicy 
fix https://github.com/chrislusf/seaweedfs/issues/1426
 2020-09-19 20:14:19 -07:00
Chris Lu 5b40a2690a refactoring 2020-09-19 14:09:58 -07:00
Chris Lu 2cbd1cf121 fix compilation 2020-09-18 00:15:54 -07:00
Chris Lu 23e9ede068 s3: collect metrics 2020-09-18 00:09:04 -07:00
Chris Lu 3984c3962f add comment 2020-09-11 15:07:19 -07:00
Chris Lu baa6bdf4d4 s3: listMultipartUploads fix output format 2020-09-11 15:04:01 -07:00
Chris Lu 3eda8d6dfc s3: ListParts output xml format 
fix https://github.com/chrislusf/seaweedfs/issues/1461
 2020-09-11 14:53:50 -07:00
Chris Lu daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu 4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
Chris Lu eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu 99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu 83012f543a s: avoid possible completeMultipartUpload re-tries that can overwrite the uploaded file 2020-08-26 08:50:24 -07:00
Chris Lu 81e5124faf s3: list bucket permission change from admin to read 
fix https://github.com/chrislusf/seaweedfs/issues/1430
 2020-08-24 11:22:45 -07:00
Chris Lu c45ba5d7d4 fix listObjectsV2 response format 
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
 2020-08-19 12:08:04 -07:00
Chris Lu c647deace1 file size support set file length 
use Attr.FileSize and TotalChunkSize to determine file size
 2020-08-15 09:32:47 -07:00
Chris Lu f86c7d911a remove println 2020-08-09 14:42:25 -07:00
Chris Lu 9ecc1170a3 istObjects和listObjectsV2不能查询子目录 #1418 
fix https://github.com/chrislusf/seaweedfs/issues/1418
 2020-08-09 14:35:53 -07:00
Chris Lu 3f4aff5dde s3: fix delimiter in list response 2020-08-09 09:09:35 -07:00
Chris Lu 828a5ae429 check signature only when auth is enabled 2020-08-08 09:11:40 -07:00
Chris Lu 2b74abf766 S3: configurable access for anonymous user 
fix https://github.com/chrislusf/seaweedfs/issues/1413
 2020-08-06 03:41:34 -07:00
limd b41b7ea4d0 fix spark read s3 bug (sc.binaryFiles) 2020-08-01 01:08:30 +08:00
Chris Lu a566bfc6e1 s3: use bucket in the domain 
fix https://github.com/chrislusf/seaweedfs/issues/1405
 2020-07-28 08:47:27 -07:00
limd 92c32f9d46 PutObject: fix create folder bug 2020-07-28 00:58:42 +08:00
Chris Lu 54cd582653 refactor 2020-07-26 12:58:58 -07:00