gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
8790 commits 28 branches 278 tags 135 MiB
Commit graph

8790 commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
chrislu b4131a3a72 filer.remote.sync: parallelize uploading to remove object storage 2022-08-22 00:01:48 -07:00
chrislu f6b92b8561 refactor 2022-08-22 00:00:51 -07:00
chrislu 3c8e95eeb3 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-21 23:26:09 -07:00
chrislu 8bac903d29 mount: add locking to prevent nil pointer 
fix https://github.com/seaweedfs/seaweedfs/issues/3479
 2022-08-21 23:26:06 -07:00
qzh 74b53729e1
feat(weed.move): add a speed limit parameter of moving files (#3478) 
* feat(weed.move): add a speed limit parameter of moving files

* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
 2022-08-21 23:08:31 -07:00
chrislu 7807f6641a minor 2022-08-21 22:54:56 -07:00
chrislu c4e862e908 3.23 2022-08-21 19:13:26 -07:00
Guo Lei 81e7e6ceb3
the statistical methord for fileCount should stay same (#3477) 
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc)
 2022-08-21 19:11:18 -07:00
chrislu 63fbf281c7 remove same file copying rage limitation 2022-08-21 17:16:21 -07:00
Patrick Schmidt 3a75d7f7aa
Implement copy_file_range syscall in FUSE (#3475) 
See the man page of copy_file_range:
https://man7.org/linux/man-pages/man2/copy_file_range.2.html
 2022-08-21 17:11:43 -07:00
chrislu fef9c6a520 filer: do not always save files in "/etc" folder to filer store 
fix https://github.com/seaweedfs/seaweedfs/issues/3476
 2022-08-21 17:02:16 -07:00
chrislu 2762154130 fix compilation 2022-08-21 16:09:23 -07:00
chrislu fb5808e0c3 EC: with multiple volume locations, the ec rebuilding may fail 2022-08-21 15:04:50 -07:00
chrislu 5790d01c6f EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations 
related to https://github.com/seaweedfs/seaweedfs/issues/3459
 2022-08-21 14:52:37 -07:00
chrislu 8b3429858d align memory for atomic read/write 2022-08-21 12:20:27 -07:00
Patrick Schmidt 2ef6ab998c
Avoid race conditions with current filer address (#3474) 
When multiple filer requests are in-flight and the current filer
disappears and a new one is selected by the first goroutine, then
there can be a lot of race conditions while retrieving the current
filer.
Therefore, load/save the current filer index atomically.
 2022-08-21 12:18:13 -07:00
Patrick Schmidt f49a9297c2
Fix hanging reads in chunk cacher (#3473) 
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
 2022-08-21 11:54:02 -07:00
chrislu 388f82f322 minor 2022-08-21 11:49:29 -07:00
chrislu 77e4b1376e refactoring 2022-08-21 11:35:54 -07:00
Patrick Schmidt 3f758820c1
Fix FUSE server buffer leaks in file gaps (#3472) 
* Fix FUSE server buffer leaks in file gaps

This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.

* Move zero logic into its own method
 2022-08-21 11:33:58 -07:00
chrislu c7892bc7c4 volume: file counter should be all files 
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
 2022-08-20 23:35:31 -07:00
chrislu f7e0a65e75 retry for all errors 2022-08-20 23:34:45 -07:00
chrislu 649555b223 adjust log 2022-08-20 23:34:07 -07:00
chrislu 973f6dd162 refactoring 2022-08-20 22:38:15 -07:00
chrislu 65ff7198fe adjust type 2022-08-20 22:37:24 -07:00
chrislu 28b862f45f use util.Retry to retry 2022-08-20 22:03:27 -07:00
chrislu 3bf8e772f8 webdav: retryable data chunk upload 2022-08-20 19:18:12 -07:00
chrislu f8fa430257 filer: retryable data chunk upload 2022-08-20 19:15:44 -07:00
chrislu 4081d50607 filer sink: retryable data chunk uploading 2022-08-20 19:09:15 -07:00
chrislu 409f39390d mount: retryable data chunk uploading 2022-08-20 19:04:36 -07:00
chrislu 6c8822f269 filer.copy: retryable file part upload 2022-08-20 18:59:57 -07:00
chrislu a3553da7f7 add fileId to UploadWithRetry return result 2022-08-20 18:54:59 -07:00
chrislu d49d0a9fc2 filer.copy: retryable upload 2022-08-20 18:50:57 -07:00
chrislu 689b4ecdcc remove unused collection and replication from upload result 2022-08-20 18:14:57 -07:00
chrislu e3f40d538d cleaner code 2022-08-20 17:51:30 -07:00
chrislu 9f50b310a0 remove unused code 2022-08-20 17:51:12 -07:00
chrislu 11f99836c3 filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0) 
fix https://github.com/seaweedfs/seaweedfs/issues/3468
 2022-08-19 23:00:56 -07:00
chrislu fdd8c5d5e0 Filer if-modified-since check doesn't fire on "exact" #3467 
fix https://github.com/seaweedfs/seaweedfs/issues/3467
 2022-08-19 22:21:51 -07:00
chrislu aac45f3e89 filer: retryable when error is not found 2022-08-19 03:36:15 -07:00
chrislu 58dd880322 minor 2022-08-18 23:51:08 -07:00
Eric Yang b6f99c4d83
fix naming typo (#3465) 
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-08-18 23:25:32 -07:00
chrislu 5702727db5 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-18 02:39:27 -07:00
chrislu 4d72a1ce6e s3: remove deleted collection during metrics reporting 
fix https://github.com/seaweedfs/seaweedfs/issues/3453
 2022-08-18 02:39:07 -07:00
Guo Lei 3172c33fa6
fix redundant type from array (#3462) 2022-08-18 02:25:25 -07:00
chrislu 42c6e52513 s3: fix regression on HEAD directory operation 2022-08-18 02:13:58 -07:00
chrislu 4573c99ae3 fix tests 2022-08-18 00:15:46 -07:00
chrislu 4ffbda1c43 fix building for unmaintained 2022-08-17 14:45:38 -07:00
chrislu 2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu e259052a70 volume: fix metrics for volume request time 2022-08-17 04:29:31 -07:00