Commit graph

686 commits

Author SHA1 Message Date
chrislu 9926ae5d29 fix tests 2022-12-27 16:24:55 -08:00
chrislu 744a757809 group chunks into sections 2022-12-27 15:45:56 -08:00
chrislu 3e590d303d add SeparateGarbageChunks 2022-12-27 15:44:56 -08:00
chrislu ab975f6a6d fix test compilation 2022-12-27 04:08:53 -08:00
chrislu 6d8bd1eedc small optimization 2022-12-27 04:08:13 -08:00
chrislu 86b9aa3173 more efficient readResolvedChunks with linked list 2022-12-27 03:29:11 -08:00
chrislu d68b59ab4c compare chunks by timestamp 2022-12-22 20:39:24 -08:00
chrislu 3fc3e7083c metadata skip reading un-available logs
this means the volumes for metadata histories have been lost.
2022-12-19 11:30:50 -08:00
wusong 549354e324
Fix hardlink counting (#4042)
Signed-off-by: wusong <wangwusong@virtaitech.com>

Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-12-08 10:50:57 -08:00
aronneagu 77699855a7
Return ETag from remote when file doesn't exist on Filer (#4025) 2022-11-30 07:43:30 -08:00
chrislu 70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu d90aa31d5f filer store: skip disabled location specific filer store initialization
fix https://github.com/seaweedfs/seaweedfs/issues/3971
2022-11-13 23:30:23 -08:00
Konstantin Lebedev 4a48332248
refactor error contains already deleted (#3932) 2022-10-31 11:34:45 -07:00
chrislu ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
chrislu f5d4952d73 filer: redis store reduce from 2 redis operations to 1 for updates. 2022-10-12 23:50:09 -07:00
chrislu cea73ac008 serialize sqlite operations
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu dff85e9c71 fix error handling 2022-10-12 00:03:14 -07:00
LHHDZ e00a12b099
associate Account and Identity by accountId (#3754) 2022-09-28 13:25:59 -07:00
chrislu b2a171632b mount: fix 0 file size in linux 2022-09-18 21:42:13 -07:00
chrislu 9d230521a4 report write error 2022-09-18 16:49:48 -07:00
Ryan Russell e0064ae097
docs: update fileIdsToDelete var (#3692)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:09:06 -07:00
chrislu 21c0587900 go fmt 2022-09-14 23:06:44 -07:00
Ryan Russell f2dde2b1da
refactor(ItemList): Incluseive -> Inclusive (#3673)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:00:16 -07:00
Ryan Russell 72d8a9f9a8
refactor(exclusive_locker): Interval readability batch of updates (#3668)
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `SafeRenewInteval` -> `SafeRenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `InitLockInteval` -> `InitLockInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `RenewInteval` -> `RenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 10:29:55 -07:00
Ryan Russell d54eb9966f
refactor: Directory readability (#3665) 2022-09-14 10:11:31 -07:00
gfx 48db56ddad
arangodb s3 bucket name compatibility (#3588)
* Update arangodb_store.go

* update readme, properly escape queries, add name patching

* use underscore

* use underscore

* better comment

* fix readme

Co-authored-by: a <a@a.a>
2022-09-09 09:43:42 -07:00
Konstantin Lebedev 3c3682fcce
more log detail for upload err and deleting (#3577) 2022-09-06 08:19:13 -07:00
famosss 131d389fc4
adjust log level (#3589) 2022-09-04 22:37:32 -07:00
gfx bf844d8e46
Fix crash in arangodb filer when attempting to access a deleted bucket (#3587)
* Update helpers.go

* Update arangodb_store_bucket.go
2022-09-04 20:32:10 -07:00
gfx 3621987727
Correctly pass arangodb driver Transaction ID into the context (#3586)
Update arangodb_store.go
2022-09-04 19:29:52 -07:00
chrislu a112cbe44c filer: rocksdb store report not found correctly
fix https://github.com/seaweedfs/seaweedfs/discussions/3555
2022-09-01 15:42:59 -07:00
chrislu 10efdc7aab align memory 2022-09-01 09:02:44 -07:00
Konstantin Lebedev 803ca3c958
avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch (#3566)
https://github.com/seaweedfs/seaweedfs/issues/3565
2022-09-01 09:01:47 -07:00
chrislu d81db3c703 s3: fix configuring IAM for the same user
hi, how can I add bucket permission to a user now?
Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name.
Now I am getting error:
duplicate accessKey[ХХХХ], already configured in user[YYYY]

s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1
s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1
2022-08-30 09:37:52 -07:00
chrislu 57a46f46a0 nano level precision 2022-08-26 16:55:15 -07:00
Patrick Schmidt 5df105b1f9
Fix a few data races when reading files in mount (#3527) 2022-08-26 16:41:37 -07:00
Konstantin Lebedev e90ab4ac60
avoid race conditions for OnPeerUpdate (#3525)
https://github.com/seaweedfs/seaweedfs/issues/3524
2022-08-26 10:18:49 -07:00
Patrick Schmidt 2930263dfd
Fix race conditions during in-flight size checks (#3505) 2022-08-24 20:03:34 -07:00
chrislu fef9c6a520 filer: do not always save files in "/etc" folder to filer store
fix https://github.com/seaweedfs/seaweedfs/issues/3476
2022-08-21 17:02:16 -07:00
Patrick Schmidt f49a9297c2
Fix hanging reads in chunk cacher (#3473)
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
2022-08-21 11:54:02 -07:00
chrislu 388f82f322 minor 2022-08-21 11:49:29 -07:00
chrislu 77e4b1376e refactoring 2022-08-21 11:35:54 -07:00
Patrick Schmidt 3f758820c1
Fix FUSE server buffer leaks in file gaps (#3472)
* Fix FUSE server buffer leaks in file gaps

This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.

* Move zero logic into its own method
2022-08-21 11:33:58 -07:00
chrislu 689b4ecdcc remove unused collection and replication from upload result 2022-08-20 18:14:57 -07:00
Guo Lei 3172c33fa6
fix redundant type from array (#3462) 2022-08-18 02:25:25 -07:00
chrislu 4573c99ae3 fix tests 2022-08-18 00:15:46 -07:00
chrislu 2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu 96caf21d09 less verbose log 2022-08-15 10:03:52 -07:00
chrislu 0aeec04c31 quicker to adapt to pattern change 2022-08-07 10:14:01 -07:00