gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
8714 commits 28 branches 278 tags 135 MiB
Commit graph

5724 commits

Author SHA1 Message Date
chrislu 7807f6641a minor 2022-08-21 22:54:56 -07:00
chrislu c4e862e908 3.23 2022-08-21 19:13:26 -07:00
Guo Lei 81e7e6ceb3
the statistical methord for fileCount should stay same (#3477) 
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc)
 2022-08-21 19:11:18 -07:00
chrislu 63fbf281c7 remove same file copying rage limitation 2022-08-21 17:16:21 -07:00
Patrick Schmidt 3a75d7f7aa
Implement copy_file_range syscall in FUSE (#3475) 
See the man page of copy_file_range:
https://man7.org/linux/man-pages/man2/copy_file_range.2.html
 2022-08-21 17:11:43 -07:00
chrislu fef9c6a520 filer: do not always save files in "/etc" folder to filer store 
fix https://github.com/seaweedfs/seaweedfs/issues/3476
 2022-08-21 17:02:16 -07:00
chrislu 2762154130 fix compilation 2022-08-21 16:09:23 -07:00
chrislu fb5808e0c3 EC: with multiple volume locations, the ec rebuilding may fail 2022-08-21 15:04:50 -07:00
chrislu 5790d01c6f EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations 
related to https://github.com/seaweedfs/seaweedfs/issues/3459
 2022-08-21 14:52:37 -07:00
chrislu 8b3429858d align memory for atomic read/write 2022-08-21 12:20:27 -07:00
Patrick Schmidt 2ef6ab998c
Avoid race conditions with current filer address (#3474) 
When multiple filer requests are in-flight and the current filer
disappears and a new one is selected by the first goroutine, then
there can be a lot of race conditions while retrieving the current
filer.
Therefore, load/save the current filer index atomically.
 2022-08-21 12:18:13 -07:00
Patrick Schmidt f49a9297c2
Fix hanging reads in chunk cacher (#3473) 
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
 2022-08-21 11:54:02 -07:00
chrislu 388f82f322 minor 2022-08-21 11:49:29 -07:00
chrislu 77e4b1376e refactoring 2022-08-21 11:35:54 -07:00
Patrick Schmidt 3f758820c1
Fix FUSE server buffer leaks in file gaps (#3472) 
* Fix FUSE server buffer leaks in file gaps

This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.

* Move zero logic into its own method
 2022-08-21 11:33:58 -07:00
chrislu c7892bc7c4 volume: file counter should be all files 
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
 2022-08-20 23:35:31 -07:00
chrislu f7e0a65e75 retry for all errors 2022-08-20 23:34:45 -07:00
chrislu 649555b223 adjust log 2022-08-20 23:34:07 -07:00
chrislu 973f6dd162 refactoring 2022-08-20 22:38:15 -07:00
chrislu 65ff7198fe adjust type 2022-08-20 22:37:24 -07:00
chrislu 28b862f45f use util.Retry to retry 2022-08-20 22:03:27 -07:00
chrislu 3bf8e772f8 webdav: retryable data chunk upload 2022-08-20 19:18:12 -07:00
chrislu f8fa430257 filer: retryable data chunk upload 2022-08-20 19:15:44 -07:00
chrislu 4081d50607 filer sink: retryable data chunk uploading 2022-08-20 19:09:15 -07:00
chrislu 409f39390d mount: retryable data chunk uploading 2022-08-20 19:04:36 -07:00
chrislu 6c8822f269 filer.copy: retryable file part upload 2022-08-20 18:59:57 -07:00
chrislu a3553da7f7 add fileId to UploadWithRetry return result 2022-08-20 18:54:59 -07:00
chrislu d49d0a9fc2 filer.copy: retryable upload 2022-08-20 18:50:57 -07:00
chrislu 689b4ecdcc remove unused collection and replication from upload result 2022-08-20 18:14:57 -07:00
chrislu e3f40d538d cleaner code 2022-08-20 17:51:30 -07:00
chrislu 9f50b310a0 remove unused code 2022-08-20 17:51:12 -07:00
chrislu 11f99836c3 filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0) 
fix https://github.com/seaweedfs/seaweedfs/issues/3468
 2022-08-19 23:00:56 -07:00
chrislu fdd8c5d5e0 Filer if-modified-since check doesn't fire on "exact" #3467 
fix https://github.com/seaweedfs/seaweedfs/issues/3467
 2022-08-19 22:21:51 -07:00
chrislu aac45f3e89 filer: retryable when error is not found 2022-08-19 03:36:15 -07:00
chrislu 58dd880322 minor 2022-08-18 23:51:08 -07:00
Eric Yang b6f99c4d83
fix naming typo (#3465) 
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-08-18 23:25:32 -07:00
chrislu 5702727db5 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-18 02:39:27 -07:00
chrislu 4d72a1ce6e s3: remove deleted collection during metrics reporting 
fix https://github.com/seaweedfs/seaweedfs/issues/3453
 2022-08-18 02:39:07 -07:00
Guo Lei 3172c33fa6
fix redundant type from array (#3462) 2022-08-18 02:25:25 -07:00
chrislu 42c6e52513 s3: fix regression on HEAD directory operation 2022-08-18 02:13:58 -07:00
chrislu 4573c99ae3 fix tests 2022-08-18 00:15:46 -07:00
chrislu 2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu e259052a70 volume: fix metrics for volume request time 2022-08-17 04:29:31 -07:00
qzh 7fcfaf7bc9
fix(filer.sync): offset may be set to 0 (#3451) 
* fix(filer.sync): initializing the offset is related to the path

* fix(filer.sync): the offset maybe to be set to 0.

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
 2022-08-15 23:43:52 -07:00
qzh 400f0c3e5d
fix(filer.sync): initializing the offset is related to the path (#3450) 
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
 2022-08-15 21:56:47 -07:00
chrislu fa4d0093e1 3.22 2022-08-15 16:48:23 -07:00
chrislu 93261f5199 Revert "Refactor for Sync method (#3426)" 
This reverts commit 670cb759f8.

with the pr
weed/storage () - (master) > go test -count=1 ./...
ok  	github.com/seaweedfs/seaweedfs/weed/storage	18.486s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map	0.025s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding	0.864s
?   	github.com/seaweedfs/seaweedfs/weed/storage/idx	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle	0.110s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle_map	24.414s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/super_block	0.203s
?   	github.com/seaweedfs/seaweedfs/weed/storage/types	[no test files]
?   	github.com/seaweedfs/seaweedfs/weed/storage/volume_info	[no test files]
weed/storage () - (master) >
weed/storage () - (master) >

without the pr
weed/storage () - (master) >
weed/storage () - (master) > go test -count=1 ./...
ok  	github.com/seaweedfs/seaweedfs/weed/storage	1.617s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map	0.026s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding	0.906s
?   	github.com/seaweedfs/seaweedfs/weed/storage/idx	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle	0.202s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle_map	24.533s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/super_block	0.280s
?   	github.com/seaweedfs/seaweedfs/weed/storage/types	[no test files]
?   	github.com/seaweedfs/seaweedfs/weed/storage/volume_info	[no test files]
 2022-08-15 15:35:31 -07:00
chrislu 9c588d4010 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-15 10:03:56 -07:00
chrislu 96caf21d09 less verbose log 2022-08-15 10:03:52 -07:00
Andrey Triumfov 31faa6d43d
Remove duplicate slashes in object path to prevent 500 errors (#3442) 2022-08-15 08:19:28 -07:00
chrislu cb476a53ff remove logs 2022-08-15 01:05:35 -07:00
chrislu 7c029b2183 3.21 2022-08-15 00:32:15 -07:00
Chris Lu e88392b50f
Fix s3 pagination (#3436) 
* Revert previous changes

* s3: use cursor to track tree traversal

fix https://github.com/seaweedfs/seaweedfs/issues/3166

* special cases for empty prefix and empty directory

* use constants

* address empty folder

* undo local changes

* fix IsTruncated

* adjust counting directories

* fix cases when prefix is a directory

* s3: handle directory object

works for

aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"
 2022-08-15 00:30:19 -07:00
chrislu 7457c746f0 s3: fix aws s3api head-object 2022-08-14 23:52:35 -07:00
chrislu 789d9dbfe1 webdav: fix nil 
fix https://github.com/seaweedfs/seaweedfs/issues/3440
 2022-08-14 23:33:30 -07:00
Konstantin Lebedev 6199d9b923
rand start garbageThreshold and reset metric ReplicaPlacementMismatch (#3396) 
* rand start garbageThreshold
reset metrics MasterReplicaPlacementMismatch

* vacuum through sleep
 2022-08-12 13:11:09 -07:00
Konstantin Lebedev 4d4cd0948d
avoid infinite loop WaitUntilConnected() (#3431) 
https://github.com/seaweedfs/seaweedfs/issues/3421
 2022-08-11 15:03:26 -07:00
Eric Yang 986daecbfe
add upload response (#3428) 
* add upload response

* fixing comments

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-08-10 22:43:09 -07:00
chrislu ae93c966d9 ensure memory is aligned 
fix https://github.com/seaweedfs/seaweedfs/issues/3427
 2022-08-10 22:27:13 -07:00
Rain Li 670cb759f8
Refactor for Sync method (#3426) 2022-08-10 08:21:57 -07:00
chrislu 662ec97602 mount: adjust on du stats reporting 
fix https://github.com/seaweedfs/seaweedfs/issues/3424
 2022-08-09 20:21:01 -07:00
Abirdcfly b0633716b7
delete minor unreachable code (#3423) 
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
 2022-08-09 01:24:43 -07:00
chrislu 0854171d22 3.20 2022-08-07 14:42:28 -07:00
chrislu d8e69df295 adjust comments 2022-08-07 11:35:33 -07:00
chrislu 0aeec04c31 quicker to adapt to pattern change 2022-08-07 10:14:01 -07:00
chrislu 928d29af9e fix wrong logic about reader isRandomMode() 2022-08-07 10:13:04 -07:00
chrislu d3d52b1818 shell: fs.meta.load add quieter mode 
fix https://github.com/seaweedfs/seaweedfs/issues/3414
 2022-08-07 01:42:15 -07:00
chrislu 358ccb5e0e Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-07 01:34:36 -07:00
chrislu 67814a5c79 refactor and fix strings.Split 2022-08-07 01:34:32 -07:00
John W Higgins 3afda0c89c
Allow postgresql to use standard environment variables for connection (#3413) 2022-08-07 00:58:53 -07:00
chrislu 1a4bf0dcb5 filer.sync: parallelize the filer.sync 2022-08-07 00:56:15 -07:00
chrislu 02b38f894b debug 2022-08-07 00:55:49 -07:00
chrislu 0e9478488d filer.sync: fix when excluded paths is empty 2022-08-07 00:55:34 -07:00
chrislu 8a880a139d mount: fix truncate operation 
fix https://github.com/seaweedfs/seaweedfs/issues/2609
 2022-08-06 13:58:45 -07:00
LHHDZ 84ec68e11a
Add download speed limit support (#3408) 2022-08-05 01:16:42 -07:00
chrislu b278bb24d3 mount: adjust df stats reporting when close to the limit 
fix https://github.com/seaweedfs/seaweedfs/issues/3407
 2022-08-05 00:31:41 -07:00
Konstantin Lebedev 4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Konstantin Lebedev 22181dd018
refactor FilerRequest metrics (#3402) 
* refactor FilerRequest metrics

* avoid double count proxy

* defer to
 2022-08-04 01:44:54 -07:00
chrislu bd13a7968f remove unused symlink resolving 2022-08-04 01:35:18 -07:00
chrislu 88945d9954 mount: do not follow soft link with xattr 2022-08-04 01:21:15 -07:00
chrislu 90db4b5a44 minor 2022-08-03 00:10:47 -07:00
Chris Lu fc8035d672
Merge pull request #3394 from kmlebedev/metricsReplicatedWrite 
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
 2022-08-01 13:31:01 -07:00
Konstantin Lebedev 3ffa4ba91d clean 2022-08-02 01:30:38 +05:00
Konstantin Lebedev 2a5629dd28 align 2022-08-02 01:29:44 +05:00
Konstantin Lebedev df9cc31a0a refactor 2022-08-02 01:18:26 +05:00
Konstantin Lebedev 0a5d3c3cf6 rename to localWrite and replicatedWrite 2022-08-01 23:09:09 +05:00
chrislu bd5c9904be Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-01 09:40:05 -07:00
chrislu e26d1cf4ef fix test 2022-08-01 09:40:01 -07:00
Konstantin Lebedev fc65122766 rename to LoadAvg_1M 2022-08-01 21:32:21 +05:00
Konstantin Lebedev ab50f21ad9 Detailed metrics VolumeServerRequestHistogram for writing to disk and replication 2022-08-01 21:16:57 +05:00
Konstantin Lebedev 2f85e31f5c Merge branch 'master' into gentle_vacuum 2022-08-01 20:51:58 +05:00
Konstantin Lebedev 5209ebbeef remove percent 2022-08-01 20:40:38 +05:00
Chris Lu b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083 
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
 2022-08-01 08:23:10 -07:00
Konstantin Lebedev 78cbd8002f revert Sleep 2022-08-01 20:21:23 +05:00
Konstantin Lebedev cd5c7ad052 move to github.com/seaweedfs/seaweedfs 2022-08-01 16:36:32 +05:00
Konstantin Lebedev 634bc97840 fix conflicts 2022-08-01 14:47:00 +05:00
Konstantin Lebedev 3c75479e2b Merge branch 'master' into gentle_vacuum 
# Conflicts:
#	weed/pb/messaging_pb/messaging.pb.go
#	weed/pb/messaging_pb/messaging_grpc.pb.go
#	weed/pb/s3_pb/s3.pb.go
#	weed/pb/volume_server_pb/volume_server.pb.go
#	weed/server/volume_grpc_vacuum.go
 2022-08-01 14:45:22 +05:00
Konstantin Lebedev c0d92f61a1 comment 2022-08-01 14:40:42 +05:00
Konstantin Lebedev 1d29f67c02 revert disk stats 2022-08-01 14:29:41 +05:00
Konstantin Lebedev a98f6d66a3 rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping 2022-08-01 12:51:41 +05:00
chrislu 036566629a filer.sync: fix synchronization logic in active-active mode 
fix https://github.com/seaweedfs/seaweedfs/issues/3328
 2022-08-01 00:06:18 -07:00
chrislu 303bd067b5 Revert "rename: delete source entry metadata only, skipping hard links" 
This reverts commit 03466f955e.

fix https://github.com/seaweedfs/seaweedfs/issues/3386
 2022-07-31 22:51:41 -07:00
chrislu 03c6f978f4 3.19 2022-07-31 17:16:51 -07:00
chrislu afef014832 scaffold 2022-07-31 13:23:44 -07:00
Chris Lu 0bb3ba17ae
Merge pull request #3385 from Woellchen/feature/cluster-status-return-healthy 
Add healthy indicator for raft status
 2022-07-30 11:55:30 -07:00
Patrick Schmidt 1a4a36d510
Add healthy indicator for raft status 2022-07-30 19:34:26 +02:00
chrislu cb096d1f53 minor 2022-07-30 09:52:52 -07:00
chrislu 0423d50106 wip note 2022-07-30 09:36:27 -07:00
chrislu 0f50fcb96a moving 2022-07-29 23:39:11 -07:00
chrislu 13b9a52f80 fix deprecated functions 2022-07-29 01:34:39 -07:00
chrislu be0ec7bc2f fix test 2022-07-29 01:15:26 -07:00
chrislu 1f161f53dc add back rack 2022-07-29 01:15:19 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev 2f0dda384d vacuum show LA 2022-07-29 11:59:33 +05:00
chrislu de3afd4199 fix rebasing 2022-07-28 23:25:46 -07:00
chrislu 06cd491abc add message batch 2022-07-28 23:24:38 -07:00
chrislu 7576c244c4 add flatbuffer serde for message 2022-07-28 23:24:38 -07:00
chrislu a3fae7cd16 add an empty placeholder for mq.topic.list 2022-07-28 23:24:38 -07:00
chrislu aa4a22ad47 segment serde 2022-07-28 23:24:38 -07:00
chrislu 74f60f246f dynamically connect to a filer 2022-07-28 23:24:38 -07:00
chrislu 9f479aab98 allocate brokers to serve segments 2022-07-28 23:24:38 -07:00
chrislu bb01b68fa0 refactor 2022-07-28 23:24:38 -07:00
chrislu 94b8c42b2c clean up 2022-07-28 23:24:38 -07:00
chrislu 8060fdcac5 remove old code 2022-07-28 23:24:38 -07:00
chrislu f25e273e32 display data center and rack in cluster.ps 2022-07-28 23:22:52 -07:00
chrislu 68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu 8d31e73ffd mq broker adds data center and rack 2022-07-28 23:22:07 -07:00
chrislu 87ccef72db cluster.ps can list brokers 2022-07-28 23:22:07 -07:00
chrislu ac66227798 renaming 2022-07-28 23:22:07 -07:00
chrislu 21b6b07dd8 renaming 2022-07-28 23:22:06 -07:00
chrislu 8ca7d1ef17 rename to SeaweedMQ 2022-07-28 23:22:06 -07:00
Chris Lu 3e7e922d16
Merge pull request #3378 from vanhtuan0409/change-put-policy 
PutUserAction should completely replace identity actions
 2022-07-28 19:43:25 -07:00
chrislu 89e276de64 fix loop 2022-07-28 18:06:49 -07:00
chrislu a458bee9f2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-28 16:32:02 -07:00
chrislu e0eda52c54 mount: ensure symlink parent directory is tracked 
fix https://github.com/chrislusf/seaweedfs/issues/3373
 2022-07-28 16:32:00 -07:00
Tuan Vuong 1d9a888be9 PutUserAction should completely replace identity actions 2022-07-28 17:02:56 +07:00
zzq09494 01b7aa79fa fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted 2022-07-28 12:04:37 +08:00
chrislu ca836568ac Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-27 12:12:43 -07:00
chrislu 3828b8ce87 "github.com/chrislusf/raft" => "github.com/seaweedfs/raft" 2022-07-27 12:12:40 -07:00
Chris Lu b7e4e26778
Merge pull request #3371 from askeipx/issues/3341 
delete disk_size metrics when collection deleted
 2022-07-27 11:25:42 -07:00
Konstantin Lebedev 7e09a548a6 exclude directories to sync on filer 2022-07-27 19:22:57 +05:00
Evgeny Kuzhelev ac5ce31278 leave notion to refactor after prometheus upgrade 2022-07-27 16:48:56 +05:00
Evgeny Kuzhelev 47c72e6f35 remove all (currently existing) collection volume metrics 2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev 709c83716c delete disk_size metrics when collection deleted 2022-07-27 15:02:41 +05:00
chrislu f401b996eb mount: fix symlink size reporting 2022-07-26 23:12:25 -07:00
Konstantin Lebedev c88ea31f62 fix RUnlock of unlocked RWMutex 2022-07-26 12:57:07 +05:00
Konstantin Lebedev da9d3e8f6c refactor 2022-07-26 11:56:45 +05:00
Konstantin Lebedev 046c3d5ad4 fix logic else brake 2022-07-26 11:47:11 +05:00
chrislu 475185fb72 3.18 2022-07-25 22:41:38 -07:00