gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6354 commits 28 branches 278 tags 135 MiB
Commit graph

265 commits

Author SHA1 Message Date
ruitao.liu 5b636b3242 Add bucket owner attr. 2020-11-11 16:20:59 +08:00
Konstantin Lebedev 6cd6ff6962 avoid blank response for 404 2020-11-04 22:54:47 +05:00
Konstantin Lebedev e1190b3224 load S3 config from filer 
https://github.com/chrislusf/seaweedfs/issues/1500
 2020-11-03 21:45:56 +05:00
ruitao.liu 22a9ea0512 adjust s3 header file. 2020-10-29 16:05:40 +08:00
ruitao.liu b917be7955 S3 bucket list, response with uploaded storageclass. 2020-10-29 14:57:19 +08:00
Chris Lu 6da87720eb shorter tag prefix 2020-10-27 15:33:26 -07:00
ruitao.liu 8766ca1b95 rename s3 tag prefix. 2020-10-27 17:33:24 +08:00
ruitao.liu 19026ae55d return x-amz-tag-count header when GET object. 2020-10-27 16:49:31 +08:00
Chris Lu da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Konstantin Lebedev 05b5f12f2e multiplate DomainNames through comma 2020-10-21 20:48:51 +05:00
Chris Lu b3aa2fab9a s3: report error to s3 when updating an object but it is already a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1545
 2020-10-20 10:25:16 -07:00
Chris Lu ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Chris Lu 723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu 9e7a2772b1 s3: only admin can list all buckets 2020-10-08 10:12:09 -07:00
Chris Lu 62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu e91b9c85a2 s3: Added support for "List" action in weed s3 -config=... in the config file. 
fix https://github.com/chrislusf/seaweedfs/issues/1511
 2020-10-07 23:22:35 -07:00
Chris Lu f781cce500 s3: support object tagging 
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
 2020-10-02 22:21:51 -07:00
Konstantin Lebedev 68463e92c1 add status code in S3RequestCounter 2020-10-01 00:59:39 +05:00
limd 402aef8f30 s3: 
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory

reference git revsion number: b41b7ea4d0
 2020-09-25 14:37:02 +08:00
Chris Lu 043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
Chris Lu dbf5327b98 s3: handle response-content-encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1487
 2020-09-24 18:09:52 -07:00
limd d506080c36 rollback 2020-09-23 14:29:53 +08:00
limd 8f9f29b773 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:33:13 +08:00
limd 1892677b22 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:15:06 +08:00
limd a99f63cb4d 1.add S3 copy directory function 
2.fixed the problem of empty directory when S3 deleted the directory
 2020-09-23 11:26:01 +08:00
Chris Lu 9cdbfc1a49 refactor 2020-09-21 17:34:38 -07:00
Chris Lu b9c1f3e9de s3: fixes for list multipart upload 2020-09-21 10:51:24 -07:00
Chris Lu 9a3b564508 adjust printout 2020-09-20 16:01:56 -07:00
Chris Lu 62563a895a refactoring 2020-09-20 16:00:01 -07:00
Chris Lu d013d09a9b adjust logging 2020-09-20 15:38:59 -07:00
Chris Lu dc4d2145ff open up 2020-09-19 20:23:12 -07:00
Chris Lu 29abe980df s3: add support for PostPolicy 
fix https://github.com/chrislusf/seaweedfs/issues/1426
 2020-09-19 20:14:19 -07:00
Chris Lu 5b40a2690a refactoring 2020-09-19 14:09:58 -07:00
Chris Lu 2cbd1cf121 fix compilation 2020-09-18 00:15:54 -07:00
Chris Lu 23e9ede068 s3: collect metrics 2020-09-18 00:09:04 -07:00
Chris Lu 3984c3962f add comment 2020-09-11 15:07:19 -07:00
Chris Lu baa6bdf4d4 s3: listMultipartUploads fix output format 2020-09-11 15:04:01 -07:00
Chris Lu 3eda8d6dfc s3: ListParts output xml format 
fix https://github.com/chrislusf/seaweedfs/issues/1461
 2020-09-11 14:53:50 -07:00
Chris Lu daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu 4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
Chris Lu eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu 99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu 83012f543a s: avoid possible completeMultipartUpload re-tries that can overwrite the uploaded file 2020-08-26 08:50:24 -07:00
Chris Lu 81e5124faf s3: list bucket permission change from admin to read 
fix https://github.com/chrislusf/seaweedfs/issues/1430
 2020-08-24 11:22:45 -07:00
Chris Lu c45ba5d7d4 fix listObjectsV2 response format 
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
 2020-08-19 12:08:04 -07:00
Chris Lu c647deace1 file size support set file length 
use Attr.FileSize and TotalChunkSize to determine file size
 2020-08-15 09:32:47 -07:00
Chris Lu f86c7d911a remove println 2020-08-09 14:42:25 -07:00
Chris Lu 9ecc1170a3 istObjects和listObjectsV2不能查询子目录 #1418 
fix https://github.com/chrislusf/seaweedfs/issues/1418
 2020-08-09 14:35:53 -07:00
Chris Lu 3f4aff5dde s3: fix delimiter in list response 2020-08-09 09:09:35 -07:00
Chris Lu 828a5ae429 check signature only when auth is enabled 2020-08-08 09:11:40 -07:00
Chris Lu 2b74abf766 S3: configurable access for anonymous user 
fix https://github.com/chrislusf/seaweedfs/issues/1413
 2020-08-06 03:41:34 -07:00
limd b41b7ea4d0 fix spark read s3 bug (sc.binaryFiles) 2020-08-01 01:08:30 +08:00
Chris Lu a566bfc6e1 s3: use bucket in the domain 
fix https://github.com/chrislusf/seaweedfs/issues/1405
 2020-07-28 08:47:27 -07:00
limd 92c32f9d46 PutObject: fix create folder bug 2020-07-28 00:58:42 +08:00
Chris Lu 54cd582653 refactor 2020-07-26 12:58:58 -07:00
Chris Lu c50ec1599c auth for presigned put object part 2020-07-26 12:09:50 -07:00
Chris Lu ea0180de4e auth for presigned put operation 2020-07-26 12:06:45 -07:00
popstk a09fcfcf20 fix s3api auth bug 2020-07-25 17:14:49 +08:00
popstk feb52400be fix s3api copy object handler SerializationError 2020-07-25 15:52:31 +08:00
Chris Lu bfabb4159b refactoring 2020-07-11 09:11:15 -07:00
Chris Lu a4e3cffe0a S3: DeleteObject API also support deleting a folder recursively 2020-06-22 10:01:00 -07:00
Rinat Shigapov 320d70d305 return last modified in UTC 2020-06-19 10:26:40 +03:00
Chris Lu 00f96d86a1 fix typo 2020-06-11 10:53:25 -07:00
Rinat Shigapov 283b749ff1 keep 204 response 2020-06-11 19:00:34 +03:00
Rinat Shigapov fafc41a27f return xml encoded NotFound status code for s3 delete 2020-06-11 17:53:15 +03:00
Rinat Shigapov 2d2c5dfa39 proxy status code of filer delete response 2020-06-11 15:00:47 +03:00
Chris Lu a25a8d8822 s3: print out time in UTC format 
fix https://github.com/chrislusf/seaweedfs/issues/1297
 2020-04-29 21:35:24 -07:00
Chris Lu 9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
Chris Lu d57559d3f7 chunk rememober encryption or not 2020-04-29 02:41:47 -07:00
Chris Lu ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu 86398d1efe S3 API: set deletion to recursive 
fix https://github.com/chrislusf/seaweedfs/issues/1251
 2020-03-26 11:05:54 -07:00
Chris Lu dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu c07bcd5065 refactoring 2020-03-23 00:30:02 -07:00
Chris Lu 654a69ff52 refactoring 2020-03-23 00:06:24 -07:00
Chris Lu fbca6b29bd refactoring 2020-03-22 23:52:55 -07:00
Chris Lu c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu 9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu 8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu ed0acd1722 go fmt 2020-02-26 16:52:57 -08:00
Chris Lu 97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu 892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu 35dde56711 refactoring 2020-02-25 12:58:45 -08:00
Chris Lu 986d63cd5e s3: increase list objects or parts limit from 1000 to 10000 
10x of Amazon S3 limits
 2020-02-25 11:45:40 -08:00
Chris Lu 7d10fdf737 fix directory lookup nil 2020-02-25 11:13:06 -08:00
Chris Lu 6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu 4ed6b584e2 s3: access control limited by bucket 2020-02-22 21:34:18 -08:00
Chris Lu 48b7ad5fa8 s3: deny anonymous type 2020-02-22 14:01:04 -08:00
Chris Lu dbabdd418e add unhandled request auth type 
fix

2020-02-18 11:43:57.396699 I | http: panic serving 172.28.0.43:50658: runtime error: invalid memory address or nil pointer dereference
goroutine 595 [running]:
net/http.(*conn).serve.func1(0xc0001fe3c0)
    /usr/lib/go/src/net/http/server.go:1767 +0x13b
panic(0x55c4e35f3820, 0x55c4e48b3c40)
    /usr/lib/go/src/runtime/panic.go:679 +0x1b6
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).authRequest(0xc0004b84e0, 0xc000115900, 0xc0000bb650, 0x1, 0x1, 0x55c4e399d740)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:143 +0x11c
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:111 +0x5e
net/http.HandlerFunc.ServeHTTP(0xc0004b87e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /usr/lib/go/src/net/http/server.go:2007 +0x46
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0004ba000, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /root/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe4
net/http.serverHandler.ServeHTTP(0xc00011e0e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /usr/lib/go/src/net/http/server.go:2802 +0xa6
net/http.(*conn).serve(0xc0001fe3c0, 0x55c4e399d680, 0xc000894180)
    /usr/lib/go/src/net/http/server.go:1890 +0x877
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:2927 +0x390
 2020-02-18 09:16:04 -08:00
Chris Lu 0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl 
Add s3 ACL
 2020-02-17 12:53:30 -08:00
Chris Lu 4cdde5f569 configuration stores the identity list 2020-02-17 12:31:59 -08:00
Chris Lu 96c1ae8471 refactoring the close http response 2020-02-14 09:46:36 -08:00
Chris Lu cf5064d702 properly close http response 2020-02-14 09:09:15 -08:00
Chris Lu b4abe3c081 unused 2020-02-09 18:02:17 -08:00
Chris Lu f3ce3166ad add streaming v4 2020-02-09 17:42:17 -08:00
Chris Lu b90ad6f452 add v2 support 2020-02-09 16:02:05 -08:00
Chris Lu 9ed364f053 support acl 2020-02-09 14:30:02 -08:00
Chris Lu b2743afaee remove println 2020-01-31 00:21:18 -08:00
Chris Lu a80ecbfe84 s3: add s3 copy 
fix https://github.com/chrislusf/seaweedfs/issues/1190
 2020-01-31 00:11:12 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu bbb6ebc3c0 filer: DeleteFolderChildren for deleting large folders 2019-12-13 00:23:05 -08:00
Chris Lu 0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu ba73c053c3 change uuid package 
fix https://github.com/chrislusf/seaweedfs/issues/1126
 2019-11-18 19:24:37 -08:00
Lei Liu 2cb348a2c2 s3 api: fix listbucket common_prefixes issue 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-09-29 21:15:40 +08:00
Chris Lu 972e881d48 add more logs 
add more logs
 2019-09-16 09:48:30 -07:00
Chris Lu 20ae559437 add logs 
add logs for https://github.com/chrislusf/seaweedfs/issues/1064
 2019-09-16 08:19:10 -07:00
Chris Lu 862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu 1e2127e261 s3: adding Location for multipart upload 
fix https://github.com/chrislusf/seaweedfs/issues/1002
 2019-07-08 12:37:45 -07:00
Chris Lu 62843991f2 s3: adjust object key 2019-07-08 12:37:20 -07:00
Chris Lu 3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu c789b496d8 use cached grpc client 2019-04-05 20:31:58 -07:00
Chris Lu 20dcb44077 fix tests 2019-04-01 23:59:31 -07:00
Chris Lu 78b9db34d5 weed s3: generate time format acceptable by aws cli s3 
fix https://github.com/chrislusf/seaweedfs/issues/911
 2019-04-01 16:13:29 -07:00
Chris Lu c5a3ff1c35 weed s3: multipart upload goes to the right bucket 
fix https://github.com/chrislusf/seaweedfs/issues/908
 2019-03-31 11:10:19 -07:00
Chris Lu 55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu cb07d15254 add namespace for s3 2019-02-27 00:21:37 -08:00
Chris Lu 70ac2f6ea8 add namespace for ListAllMyBucketsResult 2019-02-26 23:23:04 -08:00
Chris Lu 77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu 3339325334 fix s3 ListAllMyBucketsResult to work with s3cmd 2019-01-02 11:36:29 -08:00
Chris Lu 43db7ac123 fix spelling 2019-01-02 02:38:32 -08:00
Chris Lu 39ea8484c1 adjust proto API 2018-12-11 07:17:10 -08:00
Chris Lu a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu fdc8a267de remove unused function 2018-09-22 22:11:13 -07:00
Chris Lu d9b32db8d7 adjust uploads to a hidden folder 2018-09-19 22:03:16 -07:00
Chris Lu f5471bcebf add bucket name in the redirection 2018-09-19 22:01:41 -07:00
Chris Lu bea4f6ca14 go fmt 2018-09-12 01:00:57 -07:00
Chris Lu f628953c33 cleanup tmp upload directories 2018-09-12 01:00:51 -07:00
Chris Lu f6d8525d1d working S3 multipart uploads 2018-09-12 00:46:12 -07:00
Chris Lu 439e88b5a6 fix format error 2018-09-11 13:01:51 -07:00
Chris Lu 296acc4a0a go fmt 2018-09-09 16:26:11 -07:00
Chris Lu 164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu dd37f0540f add extended attributes 2018-09-07 13:12:52 -07:00
Chris Lu d614716ff2 fix compilation error 2018-09-04 00:43:46 -07:00
Chris Lu ce6a2140a2 preparing to support S3 multipart uploads 2018-09-04 00:42:44 -07:00
Chris Lu 7ad5a90127 go fmt 2018-09-03 16:47:00 -07:00
Chris Lu 2ed6d8cca6 refactoring filer rm 2018-09-03 13:16:26 -07:00
Chris Lu 03f852c799 refactoring 2018-09-03 13:03:16 -07:00
Chris Lu 0b0ece9649 add place holders for multiplarts upload 2018-09-03 11:38:10 -07:00
Chris Lu 88f1d32cc4 add s3ChunkedReader 
fix https://github.com/chrislusf/seaweedfs/issues/718
 2018-09-02 14:20:47 -07:00
Chris Lu 77fadab560 go fmt 2018-07-24 01:38:08 -07:00
Chris Lu 6039b25bc9 s3API working with listing objects 2018-07-23 01:55:26 -07:00
Chris Lu 3c7ffa2a2c go fmt 2018-07-23 01:16:17 -07:00
Chris Lu ebad3a44ab s3api do not proxy directory requests 2018-07-23 01:15:59 -07:00
Chris Lu c133e72f62 print out unsupported methods 2018-07-22 19:55:53 -07:00
Chris Lu 19c537b687 releasing 0.91 2018-07-22 02:12:02 -07:00