Commit graph

99 commits

Author SHA1 Message Date
Konstantin Lebedev 1f7e52c63e
vacuum metrics and force sync dst files (#3832) 2022-10-13 00:51:20 -07:00
chrislu 7c6324b114 adjust log level 2022-09-04 22:21:24 -07:00
Konstantin Lebedev ade94b0d0a
avoid race conditions access to SuperBlock.Version (#3539)
* avoid race conditions access to  SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515

* superBlockAccessLock replace to sync.Mutex
2022-08-30 00:08:00 -07:00
Guo Lei c57c79a0ab
optimiz commitig compact (#3388)
* optimiz vacuuming volume

* fix bugx

* rename parameters

* fix conflict

* change copyDataBasedOnIndexFile to an instance method

* close needlemap

* optimiz commiting Vacuum volume for  leveldb index

* fix bugs

* fix leveldb loading bugs

* refactor

* fix leveldb loading bug

* add leveldb recovery

* add test case for levelDB

* modify test case to cover all the new branches

* use one tmpNm instead of two instances

* refactor

* refactor

* move setWatermark to the end

* add test for watermark and updating leveldb

* fix error logic

* refactor, add test

* check nil before close needlemapeer
add test case
fix metric bug

* add tests, fix bugs

* adjust log level
remove wrong test case
refactor

* avoid duplicate  updating metric for leveldb index
2022-08-23 23:53:35 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 9f8b72a54d Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID"
This reverts commit 37da689319, reversing
changes made to 00d53c34c4.
2022-06-10 06:38:17 -07:00
shichanglin5 f5b0c04b14 perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
2022-06-09 20:41:16 +08:00
chrislu 70e5a1b632 volume close should wait for committing compaction 2022-04-26 23:34:05 -07:00
Chris Lu ffe028f8d0
Merge pull request #2974 from kmlebedev/wait_volume_closed_compression
waite volume being closed during compression idx
2022-04-26 23:29:22 -07:00
chrislu 37ab8909b0 use two flags: v.isCompacting and v.isCommitCompacting 2022-04-26 23:28:34 -07:00
chrislu 94f824e1ce volume: sync to disk before copying volume files
address https://github.com/chrislusf/seaweedfs/issues/2976
2022-04-26 13:03:43 -07:00
Konstantin Lebedev 7315d1d039 waite volume being closed during compression idx 2022-04-26 13:40:42 +05:00
Konstantin Lebedev 9438738693 avoid invalid memory address or nil pointer dereference 2022-04-18 12:10:22 +05:00
chrislu a129bda7d9 sync data first before stopping 2022-02-16 09:11:34 -08:00
Konstantin Lebedev 99ef280c7c avoid data loss after restarting a container with a volum server 2021-06-02 17:07:19 +05:00
Chris Lu 972327f966 prevent nil volume nm 2021-03-13 11:04:51 -08:00
Chris Lu f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu 821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
bingoohuang 94ea3bd3a5 renaming NeedleMapType to NeedleMapKind 2021-02-07 09:00:03 +08:00
Chris Lu 4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu 94525aa0fd allocate volume by disk type 2020-12-13 23:08:21 -08:00
Chris Lu 0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu ae655033ac adjust logging 2020-12-11 16:57:53 -08:00
Chris Lu 2c913dde04 volume: detect and drop volumes with disk IO error
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
2020-11-28 00:09:29 -08:00
Chris Lu 6d30b21b10 volume: add "-dir.idx" option for separate index storage
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-11-27 03:17:10 -08:00
Chris Lu 9104cfa744 reduce locks 2020-10-24 19:40:35 -07:00
James Hartig 3ccfa4c6ad Added VolumeMarkWritable and VolumeStatus grpc methods
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
2020-08-19 11:42:56 -04:00
Chris Lu faa5c2e89a refactoring 2020-07-03 16:34:31 -07:00
Evgenii Kozlov 0e0db70f55 Set volumes ReadOnly if low free disk space 2020-06-05 18:18:15 +03:00
Chris Lu 5568395edd Revert "Revert "Merge pull request #1299 from song-zhang/master""
This reverts commit afb6a1dbb4.
2020-05-06 15:37:17 -07:00
Chris Lu afb6a1dbb4 Revert "Merge pull request #1299 from song-zhang/master"
This reverts commit 9016fa19ba, reversing
changes made to 47234760f4.
2020-05-04 20:34:26 -07:00
zhangsong f9e8702bb4 use async write to persistent file to disk - part1 2020-05-04 17:39:44 +08:00
Chris Lu c3cb6fa1d7 volume: compaction can cause readonly volumes
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-17 09:43:57 -07:00
Chris Lu 89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu 151114ff59 volume: fix readonly status reporting 2020-03-01 17:57:37 -08:00
Chris Lu c06f7eb48a load volume info from .vif file, use superblock as a backup 2019-12-28 12:28:58 -08:00
Chris Lu f4a74e03d1 refactoring: separating .vif from tier file loading 2019-12-28 11:35:27 -08:00
Chris Lu 2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu 1346437d71
Merge pull request #1171 from stlpmo-jn/redundant_type_conversion
remove the redundant type conversion
2019-12-25 17:02:05 -08:00
Chris Lu 35393b4a02 cleanly destroy remote files 2019-12-25 16:17:58 -08:00
stlpmo f3de4b6c18 remove the redundant type conversion 2019-12-25 10:13:45 +08:00
Chris Lu 09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu f61de28c69 volume: add deletion capability for previously readonly volumes 2019-12-19 00:42:46 -08:00
Chris Lu 10bd3c6b4b refactoring 2019-12-08 19:44:16 -08:00
Chris Lu 6827cabfbb volume: dataFileAccessLock change to RW Lock 2019-12-06 06:59:57 -08:00
Chris Lu 1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu ec8de250e2 tiered storage: can copy to s3, read from s3
master not aware tiered volume yet, file assigning is not working yet
2019-12-02 15:08:31 -08:00
Chris Lu f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu b7156291a8 adjust variable names 2019-10-29 23:16:43 -07:00