gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9527 commits 28 branches 278 tags 135 MiB
Commit graph

806 commits

Author SHA1 Message Date
LHHDZ 3730772e05
refresh max volume count after adjust (#4662) 
Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2023-07-10 23:37:33 -07:00
chrislu c278bac263 avoid nil needle map 
fix https://github.com/seaweedfs/seaweedfs/issues/4640
 2023-07-07 22:16:58 -07:00
chrislu 14546f0ad8 minor 2023-07-06 09:12:28 -07:00
wusong 61553beba2
Fix DataBackend nil pointer (#4641) 2023-07-04 23:22:10 -07:00
chrislu 2aa59ab37c fix copying level db files during commi 
fix https://github.com/seaweedfs/seaweedfs/issues/4635
 2023-07-04 11:28:12 -07:00
Damiano Albani 0d9ba8c612
Support templating name of files stored via Rclone backend (#4606) 2023-06-24 07:08:24 -07:00
Bai Jie 3b88ab42aa
remove duplicate fileCount query (#4588) 2023-06-18 00:14:14 -07:00
Bai Jie 44b9d72ef0
doIsEmpty() return error if v.DataBackend is nil (#4587) 2023-06-18 00:13:40 -07:00
柏杰 0b0fb9b9e4
avoid data race read volume.IsEmpty (#4574) 
* avoid data race read volume.IsEmpty

-   avoid phantom read isEmpty for onlyEmpty
-   use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope

* add Destroy(onlyEmpty: true) test

* add Destroy(onlyEmpty: false) test

* remove unused `IsEmpty()`

* change literal `8` to `SuperBlockSize`
 2023-06-14 14:39:58 -07:00
Konstantin Lebedev 1e22d5caf2
fix get file stats for IsEmpty (#4576) 2023-06-14 01:43:30 -07:00
Konstantin Lebedev 4527ead295
fix from comment delete volume is empty (#4573) 
* fix from coments
https://github.com/seaweedfs/seaweedfs/pull/4561

* fix tests

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 22:22:46 -07:00
Konstantin Lebedev 25535e9c36
Delete volume is empty (#4561) 
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559

* fix IsEmpty

* fix test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 10:42:44 -07:00
wusong 26f15d0079
Fix no more writable volumes by delay judgment (#4548) 
* fix nomore writables volumes while disk free space is sufficient by time delay

* reset

---------

Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-05 10:17:21 -07:00
chrislu e23f3d6eca set df.File to nil after it is closed 
possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530
 2023-06-02 17:16:45 -07:00
chrislu ff7b6d779e avoid overwriting variables 
fix https://github.com/seaweedfs/seaweedfs/issues/4365
 2023-04-22 11:49:59 -07:00
Damiano Albani 9f55c7c90e
Rclone storage backend (#4402) 
* Add Rclone storage backend

* Support templating the name of files stored via Rclone

* Enable Rclone accounting

* Remove redundant type conversion

* Provide progress information for Rclone download/upload operations

* Log error when Rclone can't instantiate filesystem

* Remove filename templating functionality for Rclone storage

To (maybe) be later reintroduced as a generic functionality for all
storage backends.

* Remove S3 specific check

* Move Rclone config initialisation to init() method
 2023-04-18 14:12:12 -07:00
chrislu 0a22eea55d collect ec shard from multiple locations 
fix https://github.com/seaweedfs/seaweedfs/issues/4365
 2023-04-17 22:56:21 -07:00
chrislu f5854d13df fix test 2023-03-14 22:05:16 -07:00
chrislu 0454bb2d88 fix test 
when size is zero, the "LastModified" is not written
 2023-03-14 22:02:49 -07:00
chrislu c589e9837c fix tests 2023-03-08 01:48:28 -08:00
zemul bd6f2716d3
fix_read_tomb_needle_meta (#4263) 
Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-03-02 08:26:55 -08:00
James Hartig 81624de27b
Include name/mime in ReadAllNeedles (#4005) 2022-11-23 15:59:38 -08:00
James Hartig 4c85da7844
Include meta in ReadAllNeedles (#3991) 
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.

Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
 2022-11-20 20:19:41 -08:00
Guo Lei 5b905fb2b7
Lazy loading (#3958) 
* types packages is imported more than onece

* lazy-loading

* fix bugs

* fix bugs

* fix unit tests

* fix test error

* rename function

* unload ldb after initial startup

* Don't load ldb when starting volume server if ldbtimeout is set.

* remove uncessary unloadldb

* Update weed/command/server.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/command/volume.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-11-14 00:19:27 -08:00
Konstantin Lebedev bf8a9d2db1
[volume.chek.disk] sync of deletions the fix (#3923) 
* sync of deletions the fix

* avoid return if only partiallyDeletedNeedles

* refactor sync deletions
 2022-10-30 20:32:46 -07:00
Konstantin Lebedev 764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922) 
needles older than the cutoff time are not missing yet

https://github.com/seaweedfs/seaweedfs/issues/3919
 2022-10-28 12:12:20 -07:00
Eric Yang 51d462f204
ADHOC: volume fsck using append at ns (#3906) 
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-10-24 22:09:38 -07:00
chrislu d880fc2bb3 fix merge 2022-10-23 18:26:22 -07:00
Konstantin Lebedev 6253058d9d
ensure monotonic n.AppendAtNs in each place (#3880) 
https://github.com/seaweedfs/seaweedfs/issues/3852

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-23 18:24:52 -07:00
Konstantin Lebedev 452202f35b
ReadNeedleMeta recovery panic occurred (#3883) 
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:28 -07:00
chrislu e3ea544d40 skip reading needle if size is invalid 
related to https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:15 -07:00
chrislu f2d9049e6a fix size variable 2022-10-23 13:05:37 -07:00
chrislu 184fbb6c50 volume server: remote tier volumes only soft delete in local index 
fix https://github.com/seaweedfs/seaweedfs/issues/3889
 2022-10-23 13:04:38 -07:00
Konstantin Lebedev e20f0dbd2d
avoid data race of TraverseBfs (#3856) 
* avoid data race of TraverseBfs

* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]

goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
 2022-10-16 11:30:41 -07:00
Konstantin Lebedev 2f72103c83
avoid load volume file with BytesOffset mismatch (#3841) 
* avoid load volume file with BytesOffset mismatch

https://github.com/seaweedfs/seaweedfs/issues/2966

* set BytesOffset if has not VolumeInfoFile

* typos fail => failed

* exit if bytesOffset mismatch
 2022-10-14 00:18:09 -07:00
chrislu 9c8678ded9 ensure monotonic n.AppendAtNs 
fix https://github.com/seaweedfs/seaweedfs/issues/3852
 2022-10-13 23:15:00 -07:00
Konstantin Lebedev 1f7e52c63e
vacuum metrics and force sync dst files (#3832) 2022-10-13 00:51:20 -07:00
Guo Lei f95c25e113
types packages is imported more than onece (#3838) 2022-10-12 22:59:07 -07:00
chrislu e55076c46f cloud tier: add retry when copying data file 
fix https://github.com/seaweedfs/seaweedfs/issues/3828
 2022-10-12 00:38:32 -07:00
Guo Lei 84c401e693
Optimiz leveldb metric (#3830) 
* optimiz updating mapmetric for leveldb

* import loading leveldb

* add comments
 2022-10-11 21:13:25 -07:00
Konstantin Lebedev 5b28c3f728
revert disabling FSync for non Mac (#3814) 2022-10-10 07:28:02 -07:00
binbinshi b7de4a967e
fix: compact_map get error mismatching cokie (#3748) 
* fix: compact_map get error

* fix: CompactSection delete lock and move test to compact_map

Co-authored-by: shibinbin <shibinbin@megvii.com>
 2022-10-09 16:56:40 -07:00
Viktor Kuzmin f8d3ff466d
Allow parallel volume loading from different dirs during startup. (#3802) 2022-10-07 11:27:15 -07:00
Eric Yang 56c94cc08e
ADHOC: filter deleted files from idx file binary search (#3763) 
* ADHOC: filter deleted files from idx file binary search

* remove unwanted check

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-29 12:48:36 -07:00
Konstantin Lebedev 301b678147
[volume] Add new volumes to HUP(reload) signal (#3755) 
Add new volumes to HUP(reload) signal
 2022-09-28 12:44:13 -07:00
chrislu 3fc261d27c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-16 23:54:39 -07:00
chrislu 06748f2771 remove unused code 2022-09-16 23:54:37 -07:00
Ryan Russell 277976bd76
refactor(storage): readability improvements (#3703) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-16 02:43:17 -07:00
famosss d949a238b8
volume: add "readBufSize" option to customize read optimization (#3702) 
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit
 2022-09-16 00:30:40 -07:00
chrislu 896a85d6e4 volume: add "hasSlowRead" option to customize read optimization 2022-09-15 03:11:32 -07:00