gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9834 commits 28 branches 278 tags 135 MiB
Commit graph

756 commits

Author SHA1 Message Date
chrislu 5de93fe442 refactoring 2023-02-07 14:45:20 -08:00
Eric Yang 5083429704
ADHOC: s3 bucket name verification (#4189) 
* ADHOC: s3 bucket name verification

* add test
 2023-02-07 14:37:29 -08:00
Muhammad Hallaj bin Subery 9bd422d2c9
adding support for B2 region (#4177) 
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
 2023-02-05 21:24:21 -08:00
chrislu ef70415c70 skipping manifest chunk volume id change 2023-01-29 22:55:06 -08:00
chrislu 6e8dbf8868 add fs.meta.changeVolumeId 2023-01-29 22:51:58 -08:00
Guo Lei b63d1e9c6b
fix index (#4158) 2023-01-28 06:55:27 -08:00
chrislu 81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
chrislu 2452f3b2f7 sort chunks by offset 2023-01-12 20:29:07 -08:00
chrislu 7f49c59c14 cluster.ps add filer meta sync progress 2023-01-10 00:51:25 -08:00
Chris Lu d4566d4aaa
more solid weed mount (#4089) 
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
 2023-01-02 23:20:45 -08:00
Guo Lei ed5f3f073b
add more help message, in case of misunderstanding (#4092) 2022-12-29 07:05:05 -08:00
Guo Lei d8cfa1552b
support enable/disable vacuum (#4087) 
* stop vacuum

* suspend/resume vacuum

* remove unused code

* rename

* rename param
 2022-12-28 01:36:44 -08:00
Konstantin Lebedev 569bc7b54d
fs.verify needles status of file (#4060) 
* fs.verify needles status of file

* fix parse args

* refactor logging

* set modifyTimeAgoAtSec
 2022-12-21 15:54:17 -08:00
Konstantin Lebedev f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn (#4024) 2022-11-30 06:30:55 -08:00
Eric Yang 26d9b662b1
ADHOC: fix fs.meta.save typo (#3993) 
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-11-20 21:05:00 -08:00
chrislu 70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks() 
for later locking on reading chunks
 2022-11-15 06:33:36 -08:00
Konstantin Lebedev 0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes (#3934) 
collect all file ids from the file without cut off time for finding missing data from volumes
 2022-10-31 11:38:12 -07:00
Konstantin Lebedev a322ba042e
[volume.fsck] param volumeId is comma separated the volume id (#3933) 
volume.fsck param volumeId is comma separated the volume id

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-31 11:36:26 -07:00
Konstantin Lebedev c0deaa4948
[volume.fsck] check needles status from volume server (#3926) 
check needles status from volume server
 2022-10-31 11:33:04 -07:00
Konstantin Lebedev bf8a9d2db1
[volume.chek.disk] sync of deletions the fix (#3923) 
* sync of deletions the fix

* avoid return if only partiallyDeletedNeedles

* refactor sync deletions
 2022-10-30 20:32:46 -07:00
chrislu 0d9f2f9e7a Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-28 12:53:23 -07:00
chrislu ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
Konstantin Lebedev 764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922) 
needles older than the cutoff time are not missing yet

https://github.com/seaweedfs/seaweedfs/issues/3919
 2022-10-28 12:12:20 -07:00
Konstantin Lebedev c6f2899375
avoid fix missing of deleted needles (#3909) 2022-10-25 21:35:54 -07:00
Eric Yang 51d462f204
ADHOC: volume fsck using append at ns (#3906) 
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-10-24 22:09:38 -07:00
chrislu 377870f4a9 keep system log data 2022-10-24 16:50:39 -07:00
chrislu d5364218b2 adjust help message 2022-10-24 09:25:08 -07:00
chrislu 52e0a88a15 shell: volume.tier.upload clean up replicated copies 
fix https://github.com/seaweedfs/seaweedfs/issues/3804
 2022-10-16 17:02:56 -07:00
Konstantin Lebedev 7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 (#3861) 
* fix apply purging and add verifyNeedle

* common readSourceNeedleBlob

* use consts
 2022-10-15 20:38:46 -07:00
Konstantin Lebedev f19c9e3d9d
Volume fsck by volume (#3851) 
* refactor

* refactor args verbose and writer

* refactor readFilerFileIdFile

* fix filter by collectMtime

* skip system log collection
 2022-10-13 23:30:30 -07:00
chrislu dcd0743a35 remove unused ReadNeedleBlobRequest.needle_id 
fix https://github.com/seaweedfs/seaweedfs/issues/3853
 2022-10-13 23:10:46 -07:00
chrislu fc54afdba9 tests add parsing ec shard info 2022-10-09 22:47:55 -07:00
chrislu 0623bf582e include ec shard for capacityByFreeVolumeCount 2022-10-09 22:31:30 -07:00
chrislu 98dc1e5c15 move volume: find target volume server by exiting/max ratio 2022-10-09 22:30:59 -07:00
chrislu 049f040c3c refactor 2022-10-09 22:28:39 -07:00
chrislu 124b97f6d7 parsing ec volumes 2022-10-09 19:21:23 -07:00
chrislu 1e27d36576 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-09 18:58:14 -07:00
chrislu f9383aa726 refactor to change capacity data type 2022-10-09 18:58:10 -07:00
Konstantin Lebedev 36daa7709d
show raft leader via shell (#3796) 2022-10-06 07:10:41 -07:00
Eric Yang 56c94cc08e
ADHOC: filter deleted files from idx file binary search (#3763) 
* ADHOC: filter deleted files from idx file binary search

* remove unwanted check

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-29 12:48:36 -07:00
Konstantin Lebedev faa6167b6b
fs.meta.load load any dirs with prefix "important" (#3747) 
* fs.meta.load load any dirs with prefix "important"

* replace dirPattern to dirPrefix

* help dirPrefix
 2022-09-27 09:58:46 -07:00
chrislu b6d7556dda skip truncation on error 
fix https://github.com/seaweedfs/seaweedfs/issues/3746
 2022-09-27 09:48:23 -07:00
Ryan Russell 824f7ad9e1
refactor(shell): readability improvements (#3704) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-16 02:43:49 -07:00
chrislu 21c0587900 go fmt 2022-09-14 23:06:44 -07:00
Ryan Russell c30f6abb11
refactor(command_ec_rebuild): rebultErr -> rebuildErr (#3679) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 13:10:27 -07:00
Ryan Russell bd2dc6d641
refactor(shell): Decending -> Descending (#3675) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:06:48 -07:00
Ryan Russell dfbd8efd26
refactor(command_ec_decode): exisitngEcIndexBits -> `existingEcInde… (#3674) 
refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcIndexBits`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:02:33 -07:00
Ryan Russell c4363c4b5e
refactor(command_fs_rm): entiries -> entries (#3670) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 11:24:27 -07:00
Ryan Russell d54eb9966f
refactor: Directory readability (#3665) 2022-09-14 10:11:31 -07:00
Ryan Russell 19652c1b83
refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… (#3662) 2022-09-14 09:14:44 -07:00
chrislu 4957d8eec6 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-10 15:36:02 -07:00
chrislu b1501f61dd Revert "a new test case" 
This reverts commit eb7cf3de81.
 2022-09-10 15:35:42 -07:00
Eric Yang ddd6bee970
ADHOC: Volume fsck use a time cutoff param (#3626) 
* ADHOC: cut off volumn fsck

* more

* fix typo

* add test

* modify name

* fix comment

* fix comments

* nit

* fix typo

* Update weed/shell/command_volume_fsck.go

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-10 15:29:17 -07:00
Brian 4e3e2b1b82
Add option in volume.fix.replication to only fix under-replication and not delete volumes (#3640) 2022-09-10 08:05:28 -07:00
chrislu fc4208d128 volume.balance: default to balance ALL_COLLECTIONS 2022-09-09 16:06:05 -07:00
chrislu eb7cf3de81 a new test case 2022-09-08 14:04:42 -07:00
chrislu 4260804613 volume.balance: avoid moving out volume with max=1 2022-09-08 14:04:21 -07:00
chrislu d653c5f811 unused 2022-09-08 12:27:53 -07:00
chrislu 03e22ddb43 rename 2022-09-08 12:06:03 -07:00
Konstantin Lebedev 916673ae24
do not print diskInfos with doVolumeCheckDisk in error log (#3598) 2022-09-05 07:49:26 -07:00
Brian a28b668647
Added ability to change replication settings upon volume.tier.move (#3583) 2022-09-04 16:47:21 -07:00
chrislu 7eb121d5ae print process before do the work 2022-08-30 13:37:05 -07:00
askeipx 2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398) 
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
 2022-08-23 23:18:21 -07:00
chrislu f0b4a7659a fix test 2022-08-23 01:52:29 -07:00
chrislu bf88006037 format 2022-08-23 01:20:45 -07:00
famosss 911475526c
fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil (#3497) 2022-08-22 23:54:51 -07:00
chrislu 676e27c589 shell: stop long running jobs if lock is lost 2022-08-22 14:12:23 -07:00
chrislu 57e7582c36 refactoring 2022-08-22 14:11:13 -07:00
qzh 74b53729e1
feat(weed.move): add a speed limit parameter of moving files (#3478) 
* feat(weed.move): add a speed limit parameter of moving files

* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
 2022-08-21 23:08:31 -07:00
chrislu 4573c99ae3 fix tests 2022-08-18 00:15:46 -07:00
chrislu 2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu d3d52b1818 shell: fs.meta.load add quieter mode 
fix https://github.com/seaweedfs/seaweedfs/issues/3414
 2022-08-07 01:42:15 -07:00
chrislu 67814a5c79 refactor and fix strings.Split 2022-08-07 01:34:32 -07:00
Konstantin Lebedev 4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu a3fae7cd16 add an empty placeholder for mq.topic.list 2022-07-28 23:24:38 -07:00
chrislu 9f479aab98 allocate brokers to serve segments 2022-07-28 23:24:38 -07:00
chrislu f25e273e32 display data center and rack in cluster.ps 2022-07-28 23:22:52 -07:00
chrislu 68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu 87ccef72db cluster.ps can list brokers 2022-07-28 23:22:07 -07:00
chrislu 7a6c559ab4 fix Change replication via volume.configure.replication by collection 
fix https://github.com/chrislusf/seaweedfs/issues/3346
 2022-07-21 22:01:05 -07:00
Chris Lu 4820b594a7
Merge pull request #3298 from kmlebedev/evacuateTargetServer 
volume server evacuate to target server
 2022-07-19 13:56:49 -07:00
Konstantin Lebedev d3f7c09c03 remove ticker 
update the topology before each file
 2022-07-20 00:54:23 +05:00
Konstantin Lebedev d422e7769c ticker.Stop 2022-07-18 16:38:19 +05:00
Konstantin Lebedev 73a0dea16b sync update topologyInfo 2022-07-18 16:27:02 +05:00
Konstantin Lebedev 2b4112e462 update otherNodes 2022-07-18 11:32:28 +05:00
Konstantin Lebedev 3c2774ec3d fix update topologyInfo 2022-07-18 01:46:31 +05:00
Konstantin Lebedev 4d5144e50d clouse background update 2022-07-15 13:51:08 +05:00
guosj ab1b9697e6 supplement check duplicate accesskey 2022-07-13 17:28:20 +08:00
石昌林 392da4e038 Make the prompt information clearer 2022-07-13 11:39:45 +08:00
石昌林 ad5e8f68ec Check whether there is a duplicate accessKey when modifying iam 2022-07-13 11:36:11 +08:00
Konstantin Lebedev 6622240df7 fix TestVolumeServerEvacuate 2022-07-12 14:56:34 +05:00
Konstantin Lebedev 8372721a62 update topologyInfo 2022-07-12 13:47:21 +05:00
Konstantin Lebedev ee95d23a22 help rack 2022-07-12 11:56:58 +05:00
Konstantin Lebedev 087fa1347f volume server evacuate from rack 2022-07-12 11:33:08 +05:00
Konstantin Lebedev 4236c36599 volume server evacuate to target server 2022-07-11 16:58:15 +05:00
chrislu 57975b93ff adjust used size reporting 2022-07-04 13:47:42 -07:00
chrislu a79f5bd0d5 adjust used size reporting 2022-07-04 13:46:32 -07:00
guol-fnst 5bc895673e add getting bucket quota 2022-07-04 15:36:44 +08:00