chrislu
1e0d64c048
master: delete partially created replicated volumes during volume growth
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120
2022-10-26 12:32:24 -07:00
Konstantin Lebedev
c6f2899375
avoid fix missing of deleted needles ( #3909 )
2022-10-25 21:35:54 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
...
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
famosss
34132b2c9f
fix: isTruncated returns problematic ( #3907 )
2022-10-24 21:15:37 -07:00
famosss
874fd197b5
feat: simplify a bit ( #3905 )
2022-10-24 17:58:26 -07:00
famosss
25e012d30b
fix: set user metadata key to lowercase ( #3894 )
...
* fix: set user metadata key to lowercase
* feat: simplify a bit
2022-10-24 17:29:52 -07:00
chrislu
091f999e8c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-24 16:50:41 -07:00
chrislu
377870f4a9
keep system log data
2022-10-24 16:50:39 -07:00
Guo Lei
97edb40275
Fix errinfo ( #3893 )
...
* types packages is imported more than onece
* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
2022-10-24 09:25:48 -07:00
chrislu
d5364218b2
adjust help message
2022-10-24 09:25:08 -07:00
chrislu
d880fc2bb3
fix merge
2022-10-23 18:26:22 -07:00
Konstantin Lebedev
6253058d9d
ensure monotonic n.AppendAtNs in each place ( #3880 )
...
https://github.com/seaweedfs/seaweedfs/issues/3852
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-23 18:24:52 -07:00
Konstantin Lebedev
452202f35b
ReadNeedleMeta recovery panic occurred ( #3883 )
...
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:28 -07:00
chrislu
e3ea544d40
skip reading needle if size is invalid
...
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:15 -07:00
chrislu
5294b6dd4d
remove volume server tcp client
2022-10-23 13:15:03 -07:00
chrislu
f2d9049e6a
fix size variable
2022-10-23 13:05:37 -07:00
chrislu
184fbb6c50
volume server: remote tier volumes only soft delete in local index
...
fix https://github.com/seaweedfs/seaweedfs/issues/3889
2022-10-23 13:04:38 -07:00
liubaojiang
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite ( #3881 )
2022-10-19 01:34:14 -07:00
chrislu
758d70bdc0
3.32
2022-10-16 20:43:37 -07:00
chrislu
de286fe662
shell: volume.move handles volume moved to cloud tier
...
fix https://github.com/seaweedfs/seaweedfs/issues/3803
2022-10-16 17:52:22 -07:00
chrislu
52e0a88a15
shell: volume.tier.upload clean up replicated copies
...
fix https://github.com/seaweedfs/seaweedfs/issues/3804
2022-10-16 17:02:56 -07:00
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs ( #3856 )
...
* avoid data race of TraverseBfs
* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]
goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
2022-10-16 11:30:41 -07:00
Konstantin Lebedev
ee38ab8581
avoid data race of LogBuffer isStopping ( #3859 )
2022-10-16 11:28:56 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 ( #3861 )
...
* fix apply purging and add verifyNeedle
* common readSourceNeedleBlob
* use consts
2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f476cf3403
fix volume_server.proto conflict ( #3858 )
...
fix conflict
https://github.com/seaweedfs/seaweedfs/pull/3841
2022-10-15 20:01:07 -07:00
chrislu
5c2eb602c9
Revert "mount: remove mount directory from generating cache directory id"
...
This reverts commit 109dc7fdf6
.
2022-10-14 22:24:14 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch ( #3841 )
...
* avoid load volume file with BytesOffset mismatch
https://github.com/seaweedfs/seaweedfs/issues/2966
* set BytesOffset if has not VolumeInfoFile
* typos fail => failed
* exit if bytesOffset mismatch
2022-10-14 00:18:09 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume ( #3851 )
...
* refactor
* refactor args verbose and writer
* refactor readFilerFileIdFile
* fix filter by collectMtime
* skip system log collection
2022-10-13 23:30:30 -07:00
chrislu
9c8678ded9
ensure monotonic n.AppendAtNs
...
fix https://github.com/seaweedfs/seaweedfs/issues/3852
2022-10-13 23:15:00 -07:00
chrislu
e05637c42c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-13 23:10:49 -07:00
chrislu
dcd0743a35
remove unused ReadNeedleBlobRequest.needle_id
...
fix https://github.com/seaweedfs/seaweedfs/issues/3853
2022-10-13 23:10:46 -07:00
LHHDZ
a5b867af69
don't skip empty value at AssembleEntryWithAcp
( #3855 )
...
* add acl helper functionalities
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* add tests
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* remove 0 when create map
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* delete when empty at `AssembleEntryWithAcp`
`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp` shouldn't skip empty value
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-13 22:50:44 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files ( #3832 )
2022-10-13 00:51:20 -07:00
chrislu
f5d4952d73
filer: redis store reduce from 2 redis operations to 1 for updates.
2022-10-12 23:50:09 -07:00
Guo Lei
f95c25e113
types packages is imported more than onece ( #3838 )
2022-10-12 22:59:07 -07:00
chrislu
9a339a9cfb
default hasSlowRead to true
2022-10-12 21:15:10 -07:00
famosss
cacc3e883b
volume server:set the default value of "hasSlowRead" to true ( #3710 )
...
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
* volume server:set the default value of "hasSlowRead" to true
2022-10-12 21:13:26 -07:00
LHHDZ
bf5e45b66a
add acl helper functionalities ( #3831 )
2022-10-12 20:15:16 -07:00
Konstantin Lebedev
401315f337
master fix interruption through ctrl+c ( #3834 )
2022-10-12 07:18:40 -07:00
zemul
a05725aea6
filer: get directory metadata ( #3833 )
2022-10-12 07:14:49 -07:00
chrislu
e55076c46f
cloud tier: add retry when copying data file
...
fix https://github.com/seaweedfs/seaweedfs/issues/3828
2022-10-12 00:38:32 -07:00
chrislu
cea73ac008
serialize sqlite operations
...
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71
fix error handling
2022-10-12 00:03:14 -07:00
chrislu
8aec786a6d
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-11 21:58:19 -07:00
chrislu
0d817bc347
fix invalid memory address or nil pointer dereference on filer.sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/3826
2022-10-11 21:58:17 -07:00
Guo Lei
84c401e693
Optimiz leveldb metric ( #3830 )
...
* optimiz updating mapmetric for leveldb
* import loading leveldb
* add comments
2022-10-11 21:13:25 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey
to ExtAmzOwnerKey
and ExtAmzAclKey
to avoid unn… ( #3824 )
...
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-11 20:14:14 -07:00
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac ( #3814 )
2022-10-10 07:28:02 -07:00
LHHDZ
3550692afc
change s3_account.go package to avoid cycle dependency ( #3813 )
2022-10-10 06:44:29 -07:00
chrislu
0711870f43
3.31
2022-10-10 00:15:30 -07:00
chrislu
109dc7fdf6
mount: remove mount directory from generating cache directory id
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3811
2022-10-10 00:12:10 -07:00
chrislu
fc54afdba9
tests add parsing ec shard info
2022-10-09 22:47:55 -07:00
chrislu
0623bf582e
include ec shard for capacityByFreeVolumeCount
2022-10-09 22:31:30 -07:00
chrislu
98dc1e5c15
move volume: find target volume server by exiting/max ratio
2022-10-09 22:30:59 -07:00
chrislu
049f040c3c
refactor
2022-10-09 22:28:39 -07:00
chrislu
124b97f6d7
parsing ec volumes
2022-10-09 19:21:23 -07:00
chrislu
1e27d36576
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-09 18:58:14 -07:00
chrislu
f9383aa726
refactor to change capacity data type
2022-10-09 18:58:10 -07:00
binbinshi
b7de4a967e
fix: compact_map get error mismatching cokie ( #3748 )
...
* fix: compact_map get error
* fix: CompactSection delete lock and move test to compact_map
Co-authored-by: shibinbin <shibinbin@megvii.com>
2022-10-09 16:56:40 -07:00
BakerBunker
b9d8a837af
Fix mount -t weed
with extra options ( #3807 )
...
fuse: Ignore "-o" command
2022-10-08 11:25:19 -07:00
Konstantin Lebedev
5db25a8f2a
metric shows who is currently blocking the cluster or not ( #3799 )
...
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452
* fix metric MasterAdminLock
2022-10-07 13:26:29 -07:00
Viktor Kuzmin
f8d3ff466d
Allow parallel volume loading from different dirs during startup. ( #3802 )
2022-10-07 11:27:15 -07:00
Konstantin Lebedev
a522507f95
configure raft metrics ( #3798 )
2022-10-07 04:20:34 -07:00
Konstantin Lebedev
b9933d5589
master server graceful stop ( #3797 )
2022-10-06 09:30:30 -07:00
Konstantin Lebedev
36daa7709d
show raft leader via shell ( #3796 )
2022-10-06 07:10:41 -07:00
chrislu
ea271600ec
fix parameters
2022-10-04 12:36:05 -07:00
chrislu
0452ae6a6c
filer.sync: limit concurrency when fetching file chunks
...
fix https://github.com/seaweedfs/seaweedfs/issues/3787
2022-10-04 11:35:07 -07:00
Konstantin Lebedev
ec46a34f33
avoid increasing the coordinates of metric labels ( #3785 )
2022-10-04 01:20:01 -07:00
chrislu
29632d5a34
3.30
2022-10-02 18:13:44 -07:00
LHHDZ
e9584d9661
add ownership rest apis ( #3765 )
2022-10-01 19:18:00 -07:00
Konstantin Lebedev
6fa3d0cc46
fix delete key panic of remote sync dir ( #3770 )
2022-10-01 02:33:47 -07:00
chrislu
c3487d3da7
better fix for option format
2022-09-30 08:49:38 -07:00
chrislu
ba0db28dbb
mount: fix extra option format
2022-09-30 08:44:41 -07:00
chrislu
8e81619d02
mount: accept all extra mount options
...
fix https://github.com/seaweedfs/seaweedfs/issues/3767
2022-09-30 08:40:37 -07:00
Eric Yang
56c94cc08e
ADHOC: filter deleted files from idx file binary search ( #3763 )
...
* ADHOC: filter deleted files from idx file binary search
* remove unwanted check
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-29 12:48:36 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer ( #3759 )
2022-09-29 12:29:01 -07:00
LHHDZ
e00a12b099
associate Account
and Identity
by accountId ( #3754 )
2022-09-28 13:25:59 -07:00
LHHDZ
aacdcc4cad
s3: add account ( #3753 )
...
associate `Account` and `Identity` by accountId
2022-09-28 12:45:18 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal ( #3755 )
...
Add new volumes to HUP(reload) signal
2022-09-28 12:44:13 -07:00
chrislu
47fd5d3fa1
skip debug message
2022-09-27 13:12:31 -07:00
Konstantin Lebedev
7ae51d1ec6
fix set master options for single server ( #3708 )
...
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-27 12:06:35 -07:00
jerebear12
190afcc3fd
Allow CORS on Filer ( #3750 )
...
Adjusted filer response headers
2022-09-27 11:47:52 -07:00
Konstantin Lebedev
faa6167b6b
fs.meta.load load any dirs with prefix "important" ( #3747 )
...
* fs.meta.load load any dirs with prefix "important"
* replace dirPattern to dirPrefix
* help dirPrefix
2022-09-27 09:58:46 -07:00
chrislu
b6d7556dda
skip truncation on error
...
fix https://github.com/seaweedfs/seaweedfs/issues/3746
2022-09-27 09:48:23 -07:00
chrislu
1ffb1e696e
3.29
2022-09-25 19:20:45 -07:00
chrislu
640c53cda8
add future list
2022-09-25 17:43:42 -07:00
chrislu
cc570a4477
assert results are sorted
2022-09-25 13:49:28 -07:00
chrislu
5c8f1467a1
ordered execution async wait
2022-09-25 13:45:55 -07:00
chrislu
973b8ff067
rename
2022-09-25 13:34:55 -07:00
chrislu
e31fdbc89b
rename file
2022-09-25 12:59:38 -07:00
chrislu
31922b2bf2
s3 to watch specific directories
2022-09-20 09:30:05 -07:00
chrislu
b90d92fca9
grpc: watch metadata changes by directory
2022-09-20 09:25:18 -07:00
chrislu
b463ca1a2f
filer replication: compare content changes directly
...
Fix https://github.com/seaweedfs/seaweedfs/issues/3714
The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks.
For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.
2022-09-20 08:35:10 -07:00
chrislu
b2a171632b
mount: fix 0 file size in linux
2022-09-18 21:42:13 -07:00
chrislu
23f3aea4cf
adjust logs
2022-09-18 18:14:24 -07:00
chrislu
901e8157de
3.28
2022-09-18 17:46:04 -07:00
chrislu
9d230521a4
report write error
2022-09-18 16:49:48 -07:00
chrislu
10d5b4b32b
volume server: rename readBufferSize to readBufferSizeMB
2022-09-17 10:56:28 -07:00
chrislu
3fc261d27c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-16 23:54:39 -07:00
chrislu
06748f2771
remove unused code
2022-09-16 23:54:37 -07:00