Commit graph

698 commits

Author SHA1 Message Date
chrislu 7f49c59c14 cluster.ps add filer meta sync progress 2023-01-10 00:51:25 -08:00
Chris Lu d4566d4aaa
more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
Guo Lei ed5f3f073b
add more help message, in case of misunderstanding (#4092) 2022-12-29 07:05:05 -08:00
Guo Lei d8cfa1552b
support enable/disable vacuum (#4087)
* stop vacuum

* suspend/resume vacuum

* remove unused code

* rename

* rename param
2022-12-28 01:36:44 -08:00
Konstantin Lebedev 569bc7b54d
fs.verify needles status of file (#4060)
* fs.verify needles status of file

* fix parse args

* refactor logging

* set modifyTimeAgoAtSec
2022-12-21 15:54:17 -08:00
Konstantin Lebedev f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn (#4024) 2022-11-30 06:30:55 -08:00
Eric Yang 26d9b662b1
ADHOC: fix fs.meta.save typo (#3993)
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-11-20 21:05:00 -08:00
chrislu 70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
Konstantin Lebedev 0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes (#3934)
collect all file ids from the file without cut off time for finding missing data from volumes
2022-10-31 11:38:12 -07:00
Konstantin Lebedev a322ba042e
[volume.fsck] param volumeId is comma separated the volume id (#3933)
volume.fsck param volumeId is comma separated the volume id

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-31 11:36:26 -07:00
Konstantin Lebedev c0deaa4948
[volume.fsck] check needles status from volume server (#3926)
check needles status from volume server
2022-10-31 11:33:04 -07:00
Konstantin Lebedev bf8a9d2db1
[volume.chek.disk] sync of deletions the fix (#3923)
* sync of deletions the fix

* avoid return if only partiallyDeletedNeedles

* refactor sync deletions
2022-10-30 20:32:46 -07:00
chrislu 0d9f2f9e7a Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-28 12:53:23 -07:00
chrislu ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
Konstantin Lebedev 764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922)
needles older than the cutoff time are not missing yet

https://github.com/seaweedfs/seaweedfs/issues/3919
2022-10-28 12:12:20 -07:00
Konstantin Lebedev c6f2899375
avoid fix missing of deleted needles (#3909) 2022-10-25 21:35:54 -07:00
Eric Yang 51d462f204
ADHOC: volume fsck using append at ns (#3906)
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
chrislu 377870f4a9 keep system log data 2022-10-24 16:50:39 -07:00
chrislu d5364218b2 adjust help message 2022-10-24 09:25:08 -07:00
chrislu 52e0a88a15 shell: volume.tier.upload clean up replicated copies
fix https://github.com/seaweedfs/seaweedfs/issues/3804
2022-10-16 17:02:56 -07:00
Konstantin Lebedev 7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 (#3861)
* fix apply purging and add verifyNeedle

* common readSourceNeedleBlob

* use consts
2022-10-15 20:38:46 -07:00
Konstantin Lebedev f19c9e3d9d
Volume fsck by volume (#3851)
* refactor

* refactor args verbose and writer

* refactor readFilerFileIdFile

* fix filter by collectMtime

* skip system log collection
2022-10-13 23:30:30 -07:00
chrislu dcd0743a35 remove unused ReadNeedleBlobRequest.needle_id
fix https://github.com/seaweedfs/seaweedfs/issues/3853
2022-10-13 23:10:46 -07:00
chrislu fc54afdba9 tests add parsing ec shard info 2022-10-09 22:47:55 -07:00
chrislu 0623bf582e include ec shard for capacityByFreeVolumeCount 2022-10-09 22:31:30 -07:00
chrislu 98dc1e5c15 move volume: find target volume server by exiting/max ratio 2022-10-09 22:30:59 -07:00
chrislu 049f040c3c refactor 2022-10-09 22:28:39 -07:00
chrislu 124b97f6d7 parsing ec volumes 2022-10-09 19:21:23 -07:00
chrislu 1e27d36576 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-09 18:58:14 -07:00
chrislu f9383aa726 refactor to change capacity data type 2022-10-09 18:58:10 -07:00
Konstantin Lebedev 36daa7709d
show raft leader via shell (#3796) 2022-10-06 07:10:41 -07:00
Eric Yang 56c94cc08e
ADHOC: filter deleted files from idx file binary search (#3763)
* ADHOC: filter deleted files from idx file binary search

* remove unwanted check

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-29 12:48:36 -07:00
Konstantin Lebedev faa6167b6b
fs.meta.load load any dirs with prefix "important" (#3747)
* fs.meta.load load any dirs with prefix "important"

* replace dirPattern to dirPrefix

* help dirPrefix
2022-09-27 09:58:46 -07:00
chrislu b6d7556dda skip truncation on error
fix https://github.com/seaweedfs/seaweedfs/issues/3746
2022-09-27 09:48:23 -07:00
Ryan Russell 824f7ad9e1
refactor(shell): readability improvements (#3704)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 02:43:49 -07:00
chrislu 21c0587900 go fmt 2022-09-14 23:06:44 -07:00
Ryan Russell c30f6abb11
refactor(command_ec_rebuild): rebultErr -> rebuildErr (#3679)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 13:10:27 -07:00
Ryan Russell bd2dc6d641
refactor(shell): Decending -> Descending (#3675)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:06:48 -07:00
Ryan Russell dfbd8efd26
refactor(command_ec_decode): exisitngEcIndexBits -> `existingEcInde… (#3674)
refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcIndexBits`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:02:33 -07:00
Ryan Russell c4363c4b5e
refactor(command_fs_rm): entiries -> entries (#3670)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:27 -07:00
Ryan Russell d54eb9966f
refactor: Directory readability (#3665) 2022-09-14 10:11:31 -07:00
Ryan Russell 19652c1b83
refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… (#3662) 2022-09-14 09:14:44 -07:00
chrislu 4957d8eec6 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-10 15:36:02 -07:00
chrislu b1501f61dd Revert "a new test case"
This reverts commit eb7cf3de81.
2022-09-10 15:35:42 -07:00
Eric Yang ddd6bee970
ADHOC: Volume fsck use a time cutoff param (#3626)
* ADHOC: cut off volumn fsck

* more

* fix typo

* add test

* modify name

* fix comment

* fix comments

* nit

* fix typo

* Update weed/shell/command_volume_fsck.go

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-10 15:29:17 -07:00
Brian 4e3e2b1b82
Add option in volume.fix.replication to only fix under-replication and not delete volumes (#3640) 2022-09-10 08:05:28 -07:00
chrislu fc4208d128 volume.balance: default to balance ALL_COLLECTIONS 2022-09-09 16:06:05 -07:00
chrislu eb7cf3de81 a new test case 2022-09-08 14:04:42 -07:00
chrislu 4260804613 volume.balance: avoid moving out volume with max=1 2022-09-08 14:04:21 -07:00
chrislu d653c5f811 unused 2022-09-08 12:27:53 -07:00