gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6850 commits 28 branches 278 tags 135 MiB
Commit graph

4534 commits

Author SHA1 Message Date
chrislu 3981d65b68 remove println 2021-12-21 22:24:38 -08:00
chrislu 9a73319b45 mount: different write strategy for streaming write and random write 2021-12-21 17:28:55 -08:00
Chris Lu b0665a15f4
Merge pull request #2527 from banjiaojuhao/master-assign-by-datanode 2021-12-21 08:56:51 -08:00
banjiaojuhao dda6b90d25 assign fileId according to DataNode with empty DataCenter and Rack 2021-12-21 17:28:33 +08:00
chrislu 4b8dcff448 reverting default admin scripts 
fix https://github.com/chrislusf/seaweedfs/issues/2525

this new default value was introduced in 2.80
this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.
 2021-12-20 13:34:57 -08:00
chrislu b21a67bbe6 add writer pattern object for later use 2021-12-20 11:53:48 -08:00
chrislu 4fd29dad86 remove writeOnly flag 2021-12-20 01:11:43 -08:00
chrislu bc96682760 refactor, change file locations 2021-12-20 01:02:23 -08:00
chrislu 866c2657f0 avoid FUSE cache only for the first 512 bytes 2021-12-19 23:13:36 -08:00
chrislu 0cb9036f66 mount: only cache the first chunk on stream read 2021-12-19 23:06:03 -08:00
chrislu a152f17937 mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
chrislu 85c526c583 s3: bind to a specific IP 
fix https://github.com/chrislusf/seaweedfs/issues/2516
 2021-12-17 11:34:37 -08:00
chrislu 5eacff9d4f log message adds server name 
address https://github.com/chrislusf/seaweedfs/issues/2514#issuecomment-995925733
 2021-12-16 10:46:26 -08:00
chrislu 50ddd8c8e2 remove debug messages 
fix https://github.com/chrislusf/seaweedfs/issues/2514
 2021-12-16 00:58:15 -08:00
chrislu 7210558c7b s3: pass through s3 presigned headers 
fix https://github.com/chrislusf/seaweedfs/discussions/2502
 2021-12-15 13:18:53 -08:00
chrislu bf4d7affc0 gateway to remote object store: skip replicating multipart upload part files 
fix https://github.com/chrislusf/seaweedfs/issues/2509
 2021-12-14 19:48:31 -08:00
chrislu 316f326464 add more help message 2021-12-13 13:14:36 -08:00
Konstantin Lebedev 969f513265 disable audit log for s3 statusHandler 2021-12-13 16:08:19 +05:00
Chris Lu ab0c6a84bf
Merge pull request #2503 from kmlebedev/audit_log_nonblocking 
Audit log force async
 2021-12-13 00:47:16 -08:00
Konstantin Lebedev 34779e8f38 force enable asynchronous I/O sending events to Fluentd 2021-12-13 13:39:39 +05:00
chrislu c3b73ec23b 2.82 2021-12-12 23:25:24 -08:00
chrislu 7bf14452f7 s3: handle Response-Content-Disposition 2021-12-12 23:05:40 -08:00
chrislu 94e5c0a454 skip purging from ec volumes 2021-12-12 22:55:27 -08:00
chrislu a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
chrislu 991a3dca0d rename file 2021-12-10 13:11:00 -08:00
kmlebedev 4f98553ba9 audit log SignatureVersion 2021-12-10 19:40:32 +05:00
chrislu 2d6fcdf83a add delay if need to resume from disk 2021-12-09 13:13:05 -08:00
liubaojiang c7abc34dbb rename operation returns the correct old parent path 2021-12-10 00:02:57 +08:00
Konstantin Lebedev 98251fe16a non blocking audit log 2021-12-09 19:47:16 +05:00
chrislu 40ee9b4d7b sleep between the gap of each execution 
related to https://github.com/chrislusf/seaweedfs/issues/2476
 2021-12-07 11:44:08 -08:00
Chris Lu 2ba08afed1
Merge pull request #2498 from kmlebedev/s3_audit_log 2021-12-07 09:35:48 -08:00
Konstantin Lebedev 10678cde81 audit log config 2021-12-07 18:20:52 +05:00
Konstantin Lebedev 4ec8715f20 audit log 2021-12-07 12:15:48 +05:00
Konstantin Lebedev 3ac48cd540 audit log 2021-12-07 12:13:21 +05:00
chrislu 7f0a97c7b6 shell: volume.list supports different verbosity level 2021-12-05 21:54:40 -08:00
chrislu 5ea9715721 2.81 
also sync java client version to SeaweedFS version
 2021-12-05 18:05:24 -08:00
chrislu e6c026db65 volume.fix.replication: fix misplaced volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2416
 2021-12-05 16:56:25 -08:00
chrislu 53e2dee177 [volume.check.disk] was using the wrong source and target locations 
fix https://github.com/chrislusf/seaweedfs/issues/2268
 2021-12-05 14:32:04 -08:00
chrislu c146c76d10 avoid creating the same bucket with a different randomized name 
related to https://github.com/chrislusf/seaweedfs/issues/2492
 2021-12-05 13:06:41 -08:00
chrislu 42d97a3442 adjust randomized bucket name 2021-12-05 12:36:58 -08:00
chrislu 488afa5002 volume: load volume can optionally be skipped, if ec volume exists 
fix https://github.com/chrislusf/seaweedfs/issues/2489
 2021-12-05 02:28:52 -08:00
chrislu 59e58c4b23 volume: fix loading old volume format 
fix https://github.com/chrislusf/seaweedfs/issues/2487
 2021-12-05 01:06:01 -08:00
chrislu b70cb3e0b2 upgrade protoc to 3.17.3 
$brew install protobuf
$ protoc --version
libprotoc 3.17.3
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
 2021-12-05 00:42:25 -08:00
chrislu 5c3b783310 s3: copy object API needs to escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/2482
 2021-12-04 23:24:53 -08:00
Bl1tz23 83bffca7e6 fix MongodbStore.ListDirectoryEntries panic on find failure 2021-12-03 13:58:37 +03:00
Tanmoy Majumdar ea09fb477a return ' shouldRetry=true' so that filer can retry the failed chunk 2021-12-03 11:54:20 +06:00
Chris Lu 689f5513a9 redis3 supports sentinel 2021-11-29 01:09:51 -08:00
Chris Lu 7227cfddf5 2.80 2021-11-29 00:57:08 -08:00
Chris Lu 3d7390302d add s3.clean.uploads -timeAgo=24h 2021-11-29 00:49:49 -08:00
Chris Lu 2f72c24498 skip the rest logic 2021-11-29 00:32:21 -08:00
Chris Lu 88ff8fc27b ensure uploaded chunks are deleted on error 2021-11-29 00:28:26 -08:00
Chris Lu bea26549f2 avoid empty extended attribute value 2021-11-28 23:50:52 -08:00
Chris Lu ce2af0082e revert 2021-11-28 23:35:22 -08:00
Chris Lu 1c9f3c7ac0 read deleted chunks when replcating data 2021-11-28 23:34:34 -08:00
Chris Lu 96c66ca2aa read deleted chunks when replicating data 2021-11-28 23:33:03 -08:00
Chris Lu f3a334965d Update command_remote_uncache.go 
minor
 2021-11-28 23:10:32 -08:00
Chris Lu 9ccfc1cfee Update command_volume_fsck.go 
minor
 2021-11-28 22:39:24 -08:00
Chris Lu 71a94267c9 Update command_volume_fsck.go 
remove unused code
 2021-11-28 22:37:11 -08:00
Chris Lu ad16221a35 adjust error log 2021-11-28 22:06:17 -08:00
Chris Lu cf1586a34d add logs for writing to remote file 2021-11-27 22:09:23 -08:00
Chris Lu 3a19eea97c allocate memory by slabs 2021-11-27 12:13:00 -08:00
limd 8805c04128 fix redis2 sentinel config example 2021-11-25 19:20:02 +08:00
limd ec03f22cc3 Merge remote-tracking branch 'origin/master' 2021-11-25 16:07:14 +08:00
limd 220797bd71 support redis sentinel 2021-11-25 15:57:03 +08:00
Chris Lu 6c27845be0 add retries when writing to remote s3 
fix https://github.com/chrislusf/seaweedfs/issues/2465
 2021-11-22 21:48:04 -08:00
Chris Lu f3c789d662 2.79 2021-11-21 18:40:24 -08:00
Konstantin Lebedev 40abae7caa Avoid forbiddening for HeadBucketHandler if owners are specified 
https://github.com/chrislusf/seaweedfs/issues/2434
 2021-11-17 14:42:24 +05:00
Chris Lu 3d87aa767d fix same dc and other dc 2021-11-16 09:14:01 -08:00
Chris Lu a0ef6e3611 prevent nil response 
fix https://github.com/chrislusf/seaweedfs/issues/2452
 2021-11-15 08:46:01 -08:00
Chris Lu 100c654ec3 2.78 2021-11-14 23:29:59 -08:00
Chris Lu 7bf891c00a randomize same-dc servers and other-dc servers 2021-11-12 11:30:11 -08:00
Chris Lu 1f75f1f9dc filer: fix mysql2 SQL template 2021-11-11 22:28:28 -08:00
Chris Lu 3abbaccb70 filer: fix mysql command to upsert 2021-11-11 22:27:13 -08:00
user 9668b15f38 Filtering by volume id is prioritized. 2021-11-11 18:18:56 +09:00
user 563a74a9eb Volume filter by collection pattern added. 2021-11-11 18:01:47 +09:00
user dbb8003ce3 Volume filter function added. 2021-11-11 17:45:17 +09:00
user c387fe957b Unused parameter removed. 2021-11-11 16:08:41 +09:00
Konstantin Lebedev 42f4aa824b force overwrite s3-is-admin 
https://github.com/chrislusf/seaweedfs/issues/2433
 2021-11-11 11:34:01 +05:00
Chris Lu c77c0de418 make install 2021-11-09 12:34:57 -08:00
Chris Lu a7eee8b14b
Merge pull request #2427 from Bl1tz23/master 
Fix: potencial iam identities data race
 2021-11-09 08:53:48 -08:00
Chris Lu 1d4a61af5e add brokers 2021-11-09 08:50:55 -08:00
Bl1tz23 c683409e92 s3: add RWMutex to iam, use RLock for concurrent reading 2021-11-09 18:11:06 +03:00
Bl1tz23 5018b22f36 s3: fix potencial iam identities data race 2021-11-09 12:19:50 +03:00
Chris Lu 59d1435d65 s3: avoid possible attacks by version 2021-11-08 17:47:56 -08:00
Chris Lu 5c99b8acc3 rename variables 2021-11-08 17:47:56 -08:00
Chris Lu 00fdab1284 adjust client name 2021-11-08 17:47:56 -08:00
Chris Lu 4729a57cc0 use constants 2021-11-08 17:47:56 -08:00
Chris Lu d9dd72ea56 rename pacakge 2021-11-08 17:47:56 -08:00
Chris Lu 5cf332357b 2.77 2021-11-07 13:52:45 -08:00
Chris Lu 244841f080 adjust logs 2021-11-07 13:27:57 -08:00
Chris Lu 6e999f1176 s3: fix ListObject if more than 10000 objects 2021-11-07 12:39:36 -08:00
Chris Lu a8b0f8864d fix help message 2021-11-07 12:38:35 -08:00
Chris Lu 55973e8572 log unknown access key 2021-11-07 12:37:46 -08:00
Chris Lu c4e22b5a9a filer: deprecate "-peers" option 2021-11-06 14:36:45 -07:00
Chris Lu 751a7073e3 Merge branch 'flexible_filer_addition' 2021-11-06 14:26:26 -07:00
Chris Lu e0fc2898e9 auto updated filer peer list 2021-11-06 14:23:35 -07:00
Chris Lu 04663c3611 remote.mount: print out metadata sync errors 2021-11-06 11:29:50 -07:00
Chris Lu 330d1fde7f send peers info to filers 2021-11-06 11:29:50 -07:00
Chris Lu 84bb8e7365 send peers info to filers 2021-11-06 04:07:38 -07:00
Chris Lu 4b9c42996a refactor grpc API 2021-11-05 18:11:40 -07:00
Chris Lu 5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu 314c32514b add command to list filers 2021-11-05 17:39:41 -07:00
Chris Lu 7348d2ae63 randomize traffic if replicated 2021-11-05 00:03:13 -07:00
Chris Lu 00ae965d8d randomize a bit for ec shards distribution 2021-11-04 09:23:40 -07:00
Chris Lu 77f90ae288 add leader election in master 2021-11-04 00:54:38 -07:00
Chris Lu 35c37562bc check auth only when enabled 2021-11-03 12:11:36 -07:00
Chris Lu 3c245c69d3 handle possible race condition 2021-11-03 01:36:39 -07:00
Chris Lu af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu ab97b17e62 better printout 2021-11-02 23:45:47 -07:00
Chris Lu e66865a8c5 adjust master client log level 2021-11-02 23:45:28 -07:00
Chris Lu 0c8dea9de8 go fmt 2021-11-02 23:39:16 -07:00
Chris Lu 5160eb08f7 shell: optionally read filer address from master 2021-11-02 23:38:45 -07:00
Chris Lu 18bfbf62fc add one unit test 2021-11-02 23:36:43 -07:00
Chris Lu e1ab8b01d0 s3: adjust permission for HEAD bucket operation 
fix https://github.com/chrislusf/seaweedfs/issues/2417#issuecomment-958391856
 2021-11-02 16:28:55 -07:00
Chris Lu b25661c6df s3: skip permission checking for creating bucket if the bucket already exists 
fix https://github.com/chrislusf/seaweedfs/issues/2417

Rclone was trying to create the bucket even though the bucket already exists.
 2021-11-02 13:13:36 -07:00
Chris Lu 1b90d60765 filer: paginate large directories to delete file chunks 2021-11-02 01:04:50 -07:00
Chris Lu 794375ca0a adjust help message since both fullPercent and quietFor are needed. 2021-11-01 17:22:47 -07:00
Chris Lu fc9e246592 2.76 2021-10-31 18:08:28 -07:00
Chris Lu 9cf756b188 S3: support CORS 
related to https://github.com/chrislusf/seaweedfs/issues/1271
 2021-10-31 18:06:43 -07:00
Chris Lu f17fa400d5 refactoring 2021-10-31 18:05:34 -07:00
Chris Lu 006b4ec0f8 refactoring 2021-10-31 18:02:08 -07:00
Chris Lu edbf6d297b filer.meta.tail: add example to send metadata to elastic search 2021-10-31 18:01:33 -07:00
Chris Lu 24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu f5824f369c avoid reporting error when it is a user request error 2021-10-29 14:14:29 -07:00
Chris Lu c857cc7286 cloud tier: remove tagging since not all s3 vendors support this 2021-10-29 12:39:19 -07:00
Chris Lu d04cdcf40d s3 header add user agent 2021-10-29 12:28:24 -07:00
Konstantin Lebedev dc3ae5e8a6 DeleteBucketLifecycleConfiguration return 204 2021-10-28 19:34:37 +05:00
Konstantin Lebedev 2afb5a13af fix DeleteBucketLifecycleConfiguration 2021-10-28 18:30:33 +05:00
Chris Lu 2e76834e4d filer store: redis2 fix wrong pagination 2021-10-27 23:46:25 -07:00
Chris Lu 54b6e0f3fd adjust logs 2021-10-27 23:46:07 -07:00
Chris Lu 900e888695 filer: avoid possible prefixed pagination loop on unsupported filer stores 2021-10-27 23:45:48 -07:00
Chris Lu d774fa6c9a rename variable 2021-10-25 14:39:20 -07:00
Chris Lu 2539ba0b62 fix compilation 2021-10-25 14:38:48 -07:00
Chris Lu 5f2d7c1589 erasure coding: skip erasure coding if less than recommended 4 nodes 2021-10-25 14:38:11 -07:00
Chris Lu c9d3fb4a30 2.75 2021-10-24 18:15:59 -07:00
Chris Lu 5435027ff0 volume copy: stream out copying progress and avoid grpc request timeout 
fix https://github.com/chrislusf/seaweedfs/issues/2386
 2021-10-24 02:52:56 -07:00
Chris Lu 3be3c17f59 volume vacuum: avoid timeout with streaming progress report 
fix https://github.com/chrislusf/seaweedfs/issues/2396
 2021-10-24 01:55:34 -07:00
Konstantin Lebedev 235329a92a fix Tagging add xmlns 2021-10-20 19:40:22 +05:00
Konstantin Lebedev dc7e525cb9 fix Tagging test 2021-10-20 19:12:00 +05:00
Konstantin Lebedev 0b834600d5 fix PutObjectTaggingHandler Unmarshal 2021-10-20 17:58:06 +05:00
Konstantin Lebedev c78220a7f2 fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389 2021-10-20 16:01:06 +05:00
Chris Lu 182f43ae5f 2.74 2021-10-18 14:23:54 -07:00
Chris Lu 309f46e3af S3: fix upload limit if the size is more than 8GB 2021-10-18 14:14:18 -07:00
Chris Lu cd4fa7561b 2.73 2021-10-18 10:47:48 -07:00
Chris Lu a6bb509cb7 adjust mime detection 2021-10-18 10:47:39 -07:00
Konstantin Lebedev dd2cc1acaf fix https://github.com/chrislusf/seaweedfs/issues/2387 2021-10-18 16:27:57 +05:00
Chris Lu 97c963bac9 2.72 2021-10-17 17:40:27 -07:00
Chris Lu 3833dac3f7 continue to read from memory if there is no flush 2021-10-17 13:53:04 -07:00
Chris Lu 29fa1b9fdc add logs 2021-10-17 13:50:34 -07:00
Chris Lu cca62fdb30 mount: streaming renaming folders 2021-10-17 04:22:42 -07:00