gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
8911 commits 28 branches 278 tags 135 MiB
Commit graph

768 commits

Author SHA1 Message Date
chrislu e55076c46f cloud tier: add retry when copying data file 
fix https://github.com/seaweedfs/seaweedfs/issues/3828
 2022-10-12 00:38:32 -07:00
Guo Lei 84c401e693
Optimiz leveldb metric (#3830) 
* optimiz updating mapmetric for leveldb

* import loading leveldb

* add comments
 2022-10-11 21:13:25 -07:00
Konstantin Lebedev 5b28c3f728
revert disabling FSync for non Mac (#3814) 2022-10-10 07:28:02 -07:00
binbinshi b7de4a967e
fix: compact_map get error mismatching cokie (#3748) 
* fix: compact_map get error

* fix: CompactSection delete lock and move test to compact_map

Co-authored-by: shibinbin <shibinbin@megvii.com>
 2022-10-09 16:56:40 -07:00
Viktor Kuzmin f8d3ff466d
Allow parallel volume loading from different dirs during startup. (#3802) 2022-10-07 11:27:15 -07:00
Eric Yang 56c94cc08e
ADHOC: filter deleted files from idx file binary search (#3763) 
* ADHOC: filter deleted files from idx file binary search

* remove unwanted check

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-29 12:48:36 -07:00
Konstantin Lebedev 301b678147
[volume] Add new volumes to HUP(reload) signal (#3755) 
Add new volumes to HUP(reload) signal
 2022-09-28 12:44:13 -07:00
chrislu 3fc261d27c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-16 23:54:39 -07:00
chrislu 06748f2771 remove unused code 2022-09-16 23:54:37 -07:00
Ryan Russell 277976bd76
refactor(storage): readability improvements (#3703) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-16 02:43:17 -07:00
famosss d949a238b8
volume: add "readBufSize" option to customize read optimization (#3702) 
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit
 2022-09-16 00:30:40 -07:00
chrislu 896a85d6e4 volume: add "hasSlowRead" option to customize read optimization 2022-09-15 03:11:32 -07:00
chrislu 21c0587900 go fmt 2022-09-14 23:06:44 -07:00
chrislu 0baf3d38c2 adjust logs 2022-09-13 10:33:28 -07:00
Eric Yang ddd6bee970
ADHOC: Volume fsck use a time cutoff param (#3626) 
* ADHOC: cut off volumn fsck

* more

* fix typo

* add test

* modify name

* fix comment

* fix comments

* nit

* fix typo

* Update weed/shell/command_volume_fsck.go

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-10 15:29:17 -07:00
famosss 41af5a6ee5
simplify a bit (#3631) 2022-09-08 21:01:21 -07:00
chrislu 64e75a286e simplify a bit 2022-09-08 19:07:57 -07:00
chrislu d14d029c73 reduce v.dataFileAccessLock lock scope 2022-09-08 19:05:31 -07:00
chrislu 406a80da4b adjust variable names 2022-09-08 18:54:16 -07:00
chrislu 2bfc8970d2 refactor: move ReadNeedleDataInto into volume_read.go 2022-09-08 18:54:02 -07:00
chrislu 9b084d4c88 purge tcp implementation 2022-09-08 18:03:43 -07:00
Eric Yang b324a6536c
ADHOC: add read needle meta grpc (#3581) 
* ADHOC: add read needle meta grpc

* add test

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-06 23:51:27 -07:00
chrislu 39340f7e42 cloud tier: s3 consume all read response body 
fix https://github.com/seaweedfs/seaweedfs/issues/3584
 2022-09-04 22:23:28 -07:00
chrislu 7c6324b114 adjust log level 2022-09-04 22:21:24 -07:00
chrislu 8c5759a06d minor optimization 2022-09-04 18:50:45 -07:00
Konstantin Lebedev ade94b0d0a
avoid race conditions access to SuperBlock.Version (#3539) 
* avoid race conditions access to  SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515

* superBlockAccessLock replace to sync.Mutex
 2022-08-30 00:08:00 -07:00
chrislu 3f3a1341d8 make CodeQL happy 2022-08-26 17:09:11 -07:00
chrislu e0f4366f4c more detailed error 
related to https://github.com/seaweedfs/seaweedfs/issues/3528
 2022-08-26 16:49:48 -07:00
Konstantin Lebedev 4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount (#3526) 2022-08-26 08:41:42 -07:00
Guo Lei c57c79a0ab
optimiz commitig compact (#3388) 
* optimiz vacuuming volume

* fix bugx

* rename parameters

* fix conflict

* change copyDataBasedOnIndexFile to an instance method

* close needlemap

* optimiz commiting Vacuum volume for  leveldb index

* fix bugs

* fix leveldb loading bugs

* refactor

* fix leveldb loading bug

* add leveldb recovery

* add test case for levelDB

* modify test case to cover all the new branches

* use one tmpNm instead of two instances

* refactor

* refactor

* move setWatermark to the end

* add test for watermark and updating leveldb

* fix error logic

* refactor, add test

* check nil before close needlemapeer
add test case
fix metric bug

* add tests, fix bugs

* adjust log level
remove wrong test case
refactor

* avoid duplicate  updating metric for leveldb index
 2022-08-23 23:53:35 -07:00
Guo Lei 762dc219a6
fix deletecount bug (#3499) 2022-08-23 08:53:24 -07:00
Guo Lei 81e7e6ceb3
the statistical methord for fileCount should stay same (#3477) 
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc)
 2022-08-21 19:11:18 -07:00
chrislu c7892bc7c4 volume: file counter should be all files 
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
 2022-08-20 23:35:31 -07:00
chrislu 4d72a1ce6e s3: remove deleted collection during metrics reporting 
fix https://github.com/seaweedfs/seaweedfs/issues/3453
 2022-08-18 02:39:07 -07:00
chrislu 2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu 93261f5199 Revert "Refactor for Sync method (#3426)" 
This reverts commit 670cb759f8.

with the pr
weed/storage () - (master) > go test -count=1 ./...
ok  	github.com/seaweedfs/seaweedfs/weed/storage	18.486s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map	0.025s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding	0.864s
?   	github.com/seaweedfs/seaweedfs/weed/storage/idx	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle	0.110s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle_map	24.414s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/super_block	0.203s
?   	github.com/seaweedfs/seaweedfs/weed/storage/types	[no test files]
?   	github.com/seaweedfs/seaweedfs/weed/storage/volume_info	[no test files]
weed/storage () - (master) >
weed/storage () - (master) >

without the pr
weed/storage () - (master) >
weed/storage () - (master) > go test -count=1 ./...
ok  	github.com/seaweedfs/seaweedfs/weed/storage	1.617s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map	0.026s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding	0.906s
?   	github.com/seaweedfs/seaweedfs/weed/storage/idx	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle	0.202s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle_map	24.533s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/super_block	0.280s
?   	github.com/seaweedfs/seaweedfs/weed/storage/types	[no test files]
?   	github.com/seaweedfs/seaweedfs/weed/storage/volume_info	[no test files]
 2022-08-15 15:35:31 -07:00
Rain Li 670cb759f8
Refactor for Sync method (#3426) 2022-08-10 08:21:57 -07:00
Abirdcfly b0633716b7
delete minor unreachable code (#3423) 
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
 2022-08-09 01:24:43 -07:00
Konstantin Lebedev 3c75479e2b Merge branch 'master' into gentle_vacuum 
# Conflicts:
#	weed/pb/messaging_pb/messaging.pb.go
#	weed/pb/messaging_pb/messaging_grpc.pb.go
#	weed/pb/s3_pb/s3.pb.go
#	weed/pb/volume_server_pb/volume_server.pb.go
#	weed/server/volume_grpc_vacuum.go
 2022-08-01 14:45:22 +05:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev 2f0dda384d vacuum show LA 2022-07-29 11:59:33 +05:00
Evgeny Kuzhelev 47c72e6f35 remove all (currently existing) collection volume metrics 2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev 709c83716c delete disk_size metrics when collection deleted 2022-07-27 15:02:41 +05:00
guosj cc7a4b0a6e correct comment 2022-07-25 11:46:41 +08:00
guol-fnst ac694f0c8f rename parameter and reuse functions 
rename milestone to  watermark
 2022-07-20 17:00:40 +08:00
guol-fnst 91285bb51d remove uncessary code 2022-07-19 20:00:44 +08:00
guol-fnst b9256e0b34 optimiz 2022-07-19 16:58:53 +08:00
guol-fnst bec9f79659 use snapshot 2022-07-19 11:14:41 +08:00
guol-fnst 300b383cdf use 10 or numCPU workers if env is not found 2022-07-14 14:06:46 +08:00