Commit graph

696 commits

Author SHA1 Message Date
chrislu 81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
chrislu bfe5d910c6 use one readerCache for the whole file 2023-01-16 22:43:02 -08:00
chrislu 12f8251d32 avoid index out of slice range 2023-01-14 13:31:22 -08:00
Chris Lu 1cd2e64aac
merge chunks during upload (#4130)
* merge chunks during upload

* fix test
2023-01-12 23:07:36 -08:00
chrislu 361d97941b mysql: Missed escaping column name
fix https://github.com/seaweedfs/seaweedfs/issues/4129
2023-01-11 08:46:56 -08:00
chrislu 85f1b70207 reduce group size from 256MB to 64MB 2023-01-10 01:35:44 -08:00
chrislu 9e0d526163 fix compilation 2023-01-10 00:52:05 -08:00
chrislu 7f49c59c14 cluster.ps add filer meta sync progress 2023-01-10 00:51:25 -08:00
chrislu 340e7c3a2e chunk group remove manifestChunks and reset sections in SetChunks() 2023-01-10 00:49:31 -08:00
chrislu e650c8397d ensure section.chunks is always garbage free 2023-01-10 00:48:01 -08:00
chrislu 75bdd4a0d1 refactor 2023-01-10 00:46:46 -08:00
chrislu 296fdc296c mount: faster add chunks 2023-01-06 01:03:29 -08:00
chrislu 2abf817580 fix for stream reader
fix https://github.com/seaweedfs/seaweedfs/issues/4112
2023-01-05 11:19:21 -08:00
chrislu e367444586 add notes for full version 2023-01-04 09:52:25 -08:00
Chris Lu d4566d4aaa
more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
CommanderRoot c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094) 2023-01-01 05:07:53 -08:00
CommanderRoot 265a56630b
filer.store.mysql: Escape table columns in SQL query (#4095) 2023-01-01 05:06:41 -08:00
chrislu 3fc3e7083c metadata skip reading un-available logs
this means the volumes for metadata histories have been lost.
2022-12-19 11:30:50 -08:00
wusong 549354e324
Fix hardlink counting (#4042)
Signed-off-by: wusong <wangwusong@virtaitech.com>

Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-12-08 10:50:57 -08:00
aronneagu 77699855a7
Return ETag from remote when file doesn't exist on Filer (#4025) 2022-11-30 07:43:30 -08:00
chrislu 70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu d90aa31d5f filer store: skip disabled location specific filer store initialization
fix https://github.com/seaweedfs/seaweedfs/issues/3971
2022-11-13 23:30:23 -08:00
Konstantin Lebedev 4a48332248
refactor error contains already deleted (#3932) 2022-10-31 11:34:45 -07:00
chrislu ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
chrislu f5d4952d73 filer: redis store reduce from 2 redis operations to 1 for updates. 2022-10-12 23:50:09 -07:00
chrislu cea73ac008 serialize sqlite operations
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu dff85e9c71 fix error handling 2022-10-12 00:03:14 -07:00
LHHDZ e00a12b099
associate Account and Identity by accountId (#3754) 2022-09-28 13:25:59 -07:00
chrislu b2a171632b mount: fix 0 file size in linux 2022-09-18 21:42:13 -07:00
chrislu 9d230521a4 report write error 2022-09-18 16:49:48 -07:00
Ryan Russell e0064ae097
docs: update fileIdsToDelete var (#3692)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:09:06 -07:00
chrislu 21c0587900 go fmt 2022-09-14 23:06:44 -07:00
Ryan Russell f2dde2b1da
refactor(ItemList): Incluseive -> Inclusive (#3673)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:00:16 -07:00
Ryan Russell 72d8a9f9a8
refactor(exclusive_locker): Interval readability batch of updates (#3668)
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `SafeRenewInteval` -> `SafeRenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `InitLockInteval` -> `InitLockInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `RenewInteval` -> `RenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 10:29:55 -07:00
Ryan Russell d54eb9966f
refactor: Directory readability (#3665) 2022-09-14 10:11:31 -07:00
gfx 48db56ddad
arangodb s3 bucket name compatibility (#3588)
* Update arangodb_store.go

* update readme, properly escape queries, add name patching

* use underscore

* use underscore

* better comment

* fix readme

Co-authored-by: a <a@a.a>
2022-09-09 09:43:42 -07:00
Konstantin Lebedev 3c3682fcce
more log detail for upload err and deleting (#3577) 2022-09-06 08:19:13 -07:00
famosss 131d389fc4
adjust log level (#3589) 2022-09-04 22:37:32 -07:00
gfx bf844d8e46
Fix crash in arangodb filer when attempting to access a deleted bucket (#3587)
* Update helpers.go

* Update arangodb_store_bucket.go
2022-09-04 20:32:10 -07:00
gfx 3621987727
Correctly pass arangodb driver Transaction ID into the context (#3586)
Update arangodb_store.go
2022-09-04 19:29:52 -07:00
chrislu a112cbe44c filer: rocksdb store report not found correctly
fix https://github.com/seaweedfs/seaweedfs/discussions/3555
2022-09-01 15:42:59 -07:00
chrislu 10efdc7aab align memory 2022-09-01 09:02:44 -07:00
Konstantin Lebedev 803ca3c958
avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch (#3566)
https://github.com/seaweedfs/seaweedfs/issues/3565
2022-09-01 09:01:47 -07:00
chrislu d81db3c703 s3: fix configuring IAM for the same user
hi, how can I add bucket permission to a user now?
Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name.
Now I am getting error:
duplicate accessKey[ХХХХ], already configured in user[YYYY]

s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1
s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1
2022-08-30 09:37:52 -07:00
chrislu 57a46f46a0 nano level precision 2022-08-26 16:55:15 -07:00
Patrick Schmidt 5df105b1f9
Fix a few data races when reading files in mount (#3527) 2022-08-26 16:41:37 -07:00
Konstantin Lebedev e90ab4ac60
avoid race conditions for OnPeerUpdate (#3525)
https://github.com/seaweedfs/seaweedfs/issues/3524
2022-08-26 10:18:49 -07:00
Patrick Schmidt 2930263dfd
Fix race conditions during in-flight size checks (#3505) 2022-08-24 20:03:34 -07:00
chrislu fef9c6a520 filer: do not always save files in "/etc" folder to filer store
fix https://github.com/seaweedfs/seaweedfs/issues/3476
2022-08-21 17:02:16 -07:00
Patrick Schmidt f49a9297c2
Fix hanging reads in chunk cacher (#3473)
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
2022-08-21 11:54:02 -07:00