Commit graph

214 commits

Author SHA1 Message Date
wang wusong 9e5b1eef9b avoid passe lock by value 2023-09-14 06:51:12 -07:00
chrislu f5c4b642bb re-ordering 2023-08-16 23:54:23 -07:00
chrislu 3852307e94 renaming 2023-08-16 23:47:43 -07:00
chrislu 6c7fa567d4 add separate cache directory for write buffers 2023-08-16 23:39:21 -07:00
wusong 33b9192536
Fix large file chunks deleted by mistake (#4678) 2023-07-17 07:52:14 -07:00
wusong 0cb9ddd8ec
Fix data loss: add lock for metacache (#4664)
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-07-11 22:23:32 -07:00
wusong c44a6552d2
Fix buff use (#4628) 2023-07-03 08:33:34 -07:00
zemul 68f1cc34a5
[mount]Fix copy file range (#4407)
* fix:mount deadlock

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-23 22:34:38 -07:00
Chengyu Liu 0f653d2153
update fuse client log which should output the fh of FileHandle. (#4404) 2023-04-15 22:15:16 -07:00
zemul 0122e022ea
Mount concurrent read (#4400)
* fix:mount deadlock

* feat: concurrent read

* fix

* Remove useless code

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-13 22:32:45 -07:00
chrislu 5db9fcccd4 refactoring 2023-03-21 23:01:49 -07:00
Patrick Schmidt 71b33faef0
Skip parent directory creation in mount (#4310)
A POSIX filesystem does not implicitly create parent directories when
they do not exist. Directories must be explicitly created and
permissions be set.

This also fixes a bug where asynchronous operations would create a
file in the filer before the parent directory was created. If the
file was a symlink or another special type of file the directory
would inherit that type and become unusable in the mounted FS.
2023-03-15 12:37:25 -07:00
wusong 79af6b41d4
[mount] Fix read file-data in entry.content (#4207) 2023-02-15 07:18:56 -08:00
wusong fe573f26a8
memchunk save content twice (#4174) 2023-01-31 19:36:59 -08:00
wusong 665917fdf1
[mount]: delete redundant code (#4173) 2023-01-31 18:54:46 -08:00
wusong de081c0d64
[mount] fix metacache update (#4161) 2023-01-29 07:55:27 -08:00
chrislu 81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
chrislu bfe5d910c6 use one readerCache for the whole file 2023-01-16 22:43:02 -08:00
chrislu c24678153c use memory chunks only for sequential writes 2023-01-15 21:10:08 -08:00
chrislu 313fd17ae4 allow random writes for mem chunk 2023-01-13 20:32:18 -08:00
Chris Lu 1cd2e64aac
merge chunks during upload (#4130)
* merge chunks during upload

* fix test
2023-01-12 23:07:36 -08:00
chrislu efcffe4f77 no need to set chunks during flush 2023-01-10 10:01:52 -08:00
chrislu 86657ed3a8 avoid repeatedly adding manifest chunks 2023-01-10 01:35:10 -08:00
chrislu 28fe578944 avoid possible nil entry 2023-01-10 01:07:34 -08:00
chrislu 5423790b2c add back previous chunk upload selection algo 2023-01-03 22:50:39 -08:00
Chris Lu d4566d4aaa
more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
chrislu 77b7c9f7cc lock and unlock chunks for read 2022-12-21 14:15:35 -08:00
chrislu d1797deccb remove duplicated logic of removing from writableChunks 2022-12-21 14:15:11 -08:00
chrislu f7beba8515 unlock before submitting the uploading jobs 2022-12-20 12:52:30 -08:00
chrislu 28d479e5c0 mount: adjust locking for upload pipeline 2022-12-19 15:07:22 -08:00
wusong 549354e324
Fix hardlink counting (#4042)
Signed-off-by: wusong <wangwusong@virtaitech.com>

Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-12-08 10:50:57 -08:00
chrislu 659f28b855 simple refactoring 2022-12-05 23:19:38 -08:00
chrislu ca49ef8ed8 duplicated logic 2022-12-05 23:19:22 -08:00
Chris Lu dac9c28d05
Revert "refactor: moved to locked entry" (#4035)
* Revert "refactor: moved to locked entry"

This reverts commit 94bc9afd9d.

* only add LockedEntry, no changes to entryLock

* fix compilation
2022-12-05 12:32:27 -08:00
chrislu 94bc9afd9d refactor: moved to locked entry 2022-12-04 23:33:05 -08:00
chrislu 70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
chrislu 5c2eb602c9 Revert "mount: remove mount directory from generating cache directory id"
This reverts commit 109dc7fdf6.
2022-10-14 22:24:14 -07:00
chrislu 109dc7fdf6 mount: remove mount directory from generating cache directory id
fix https://github.com/seaweedfs/seaweedfs/discussions/3811
2022-10-10 00:12:10 -07:00
chrislu 31922b2bf2 s3 to watch specific directories 2022-09-20 09:30:05 -07:00
Ryan Russell d65bdeef08
docs(meta_cache): readability fixes (#3691)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:04:57 -07:00
chrislu 21c0587900 go fmt 2022-09-14 23:06:44 -07:00
Ryan Russell 72d8a9f9a8
refactor(exclusive_locker): Interval readability batch of updates (#3668)
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `SafeRenewInteval` -> `SafeRenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `InitLockInteval` -> `InitLockInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `RenewInteval` -> `RenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 10:29:55 -07:00
chrislu 22064c3425 mount: ensure ordered file handle lock and unlock 2022-09-11 19:44:34 -07:00
VictorSFF 10d545060f
mount: fix new inode allocation (#3633) 2022-09-09 06:32:19 -07:00
Patrick Schmidt a73e177ecf
Add an End-to-End workflow for FUSE mount (#3562)
* Add an e2e workflow to test FUSE mount

* Fix deadlocks during concurrent r/w
2022-08-31 09:27:53 -07:00
Patrick Schmidt 00041f6e08
Fix corrupted mount reads after changes on fh (#3536) 2022-08-29 08:27:13 -07:00
chrislu 301b49b63f atomic operation 2022-08-26 17:04:11 -07:00
Patrick Schmidt 5df105b1f9
Fix a few data races when reading files in mount (#3527) 2022-08-26 16:41:37 -07:00
askeipx 2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398)
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
2022-08-23 23:18:21 -07:00