Commit graph

6208 commits

Author SHA1 Message Date
chrislu 5149b3d07b filer can proxy to peer filer holding the lock 2023-06-25 15:28:01 -07:00
chrislu 868f7875d7 refactor 2023-06-25 14:30:58 -07:00
chrislu 26eff062c8 refactor 2023-06-25 14:17:58 -07:00
chrislu 464a71a373 add distributed lock manager 2023-06-25 14:14:40 -07:00
chrislu ee4f7cd636 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-06-25 00:58:24 -07:00
chrislu 3fd659df2a add distributed lock manager 2023-06-25 00:58:21 -07:00
Damiano Albani 0d9ba8c612
Support templating name of files stored via Rclone backend (#4606) 2023-06-24 07:08:24 -07:00
chrislu a82c44972b refactor 2023-06-20 00:12:59 -07:00
chrislu 14a19170e2 refactor 2023-06-19 23:51:04 -07:00
chrislu 84ced2abf5 add LockRing 2023-06-19 23:48:33 -07:00
chrislu fe1716ccdf remove deprecated tests 2023-06-19 18:43:51 -07:00
chrislu 1ce725728d minor 2023-06-19 18:43:41 -07:00
chrislu 11549706ef refactor 2023-06-19 18:21:07 -07:00
chrislu 8ec1bc2c99 remove unused cluster node leader 2023-06-19 18:19:13 -07:00
Bai Jie 3b88ab42aa
remove duplicate fileCount query (#4588) 2023-06-18 00:14:14 -07:00
Bai Jie 44b9d72ef0
doIsEmpty() return error if v.DataBackend is nil (#4587) 2023-06-18 00:13:40 -07:00
chrislu 94fbf02ea2 generate jwt token when looking up needle id
fix https://github.com/seaweedfs/seaweedfs/issues/4577
2023-06-17 00:11:23 -07:00
柏杰 0b0fb9b9e4
avoid data race read volume.IsEmpty (#4574)
* avoid data race read volume.IsEmpty

-   avoid phantom read isEmpty for onlyEmpty
-   use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope

* add Destroy(onlyEmpty: true) test

* add Destroy(onlyEmpty: false) test

* remove unused `IsEmpty()`

* change literal `8` to `SuperBlockSize`
2023-06-14 14:39:58 -07:00
Konstantin Lebedev 1e22d5caf2
fix get file stats for IsEmpty (#4576) 2023-06-14 01:43:30 -07:00
Konstantin Lebedev 4dd890d4a2
optional https port for s3 (#4482)
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-13 23:58:49 -07:00
chrislu 3fbf4f6189 Revert "Initial patch for redis version 7 support (#4572)"
This reverts commit a25bca0692.
2023-06-12 22:27:38 -07:00
Konstantin Lebedev 4527ead295
fix from comment delete volume is empty (#4573)
* fix from coments
https://github.com/seaweedfs/seaweedfs/pull/4561

* fix tests

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 22:22:46 -07:00
Aaron Gipson a25bca0692
Initial patch for redis version 7 support (#4572) 2023-06-12 13:30:03 -07:00
Konstantin Lebedev 25535e9c36
Delete volume is empty (#4561)
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559

* fix IsEmpty

* fix test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 10:42:44 -07:00
柏杰 68feffa4e6
avoid overwriting non-nil uploadReaderToChunks.uploadErr with nil value (#4560) 2023-06-10 23:15:27 -07:00
Konstantin Lebedev 5ee04d20fa
Healthz check for deadlocks (#4558) 2023-06-09 09:42:48 -07:00
wusong d6af3a8eb6
fix 404 not found (#4536)
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-06 09:14:49 -07:00
Konstantin Lebedev 5caff2c0f0
fix: deletes newly created volumes (#4551)
https://github.com/seaweedfs/seaweedfs/issues/4531

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-06 09:08:27 -07:00
wusong 9bdbf9c880
revert #4491 (#4550)
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-06 00:17:51 -07:00
chrislu 0bffb3770f support TTL when creating folders
related to https://github.com/seaweedfs/seaweedfs/discussions/4549
2023-06-05 23:29:25 -07:00
wusong 26f15d0079
Fix no more writable volumes by delay judgment (#4548)
* fix nomore writables volumes while disk free space is sufficient by time delay

* reset

---------

Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-05 10:17:21 -07:00
chrislu fb4b61036c 3.52 2023-06-04 22:34:26 -07:00
Konstantin Lebedev a0931be0c0
S3 TLS credentials Refreshing (#4506)
* S3 TLS credentials Refreshing

* fix: logging

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-04 14:27:56 -07:00
wusong 5aec6da8a3
fix deadlock for filer upload (#4527)
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-03 13:38:27 -07:00
chrislu e23f3d6eca set df.File to nil after it is closed
possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530
2023-06-02 17:16:45 -07:00
chrislu ca7cc61319 reduce the window size between unregistering a volume server and creating volumes on that server
fix https://github.com/seaweedfs/seaweedfs/issues/4467
2023-05-30 08:20:00 -07:00
chrislu 68c3cf767c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-05-29 16:45:28 -07:00
chrislu d1d1757520 adjust error message 2023-05-29 16:45:25 -07:00
wusong e7f4418855
add return for error handle (#4502) 2023-05-24 07:45:06 -07:00
chrislu 4310e1fac4 3.51 2023-05-21 23:43:39 -07:00
wusong 8fffe3e822
fix no more writables volumes while disk free space is sufficient (#4491)
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-05-21 22:18:50 -07:00
mervynzhang 1ebb549f77
support swift (#4480) 2023-05-19 06:39:25 -07:00
Mesar Hameed a90b777ff4
Feat: etcd filer store keys should have customizable prefix (#4484)
An etcd cluster is not necessarily only dedicated to seaweedfs.
This security enhancement adds a customizable key_prefix option to the etcd filer store.
This will allow an etcd cluster administrator to limit the seaweedfs etcd user to only read/write a subset of keys under the
key_prefix, instead of all keys on the etcd cluster.
2023-05-18 23:08:56 -07:00
SmsS4 17e91d2917
Use filerGroup for s3 buckets collection prefix (#4465)
* Use filerGroup for s3 buckets collection prefix

* Fix templates

* Remove flags

* Remove s3CollectionPrefix
2023-05-16 09:39:43 -07:00
chrislu b7f011f777 fix test 2023-05-15 17:46:56 -07:00
Mesar Hameed 65484e80ef
Feat: support username/password authentication for etcd filer store s… (#4477)
Feat: support username/password authentication for etcd filer store seaweedfs/seaweedfs#4262

Co-authored-by: Mesar Hameed <mesar.hameed@gmail.com>
2023-05-15 17:45:14 -07:00
chrislu 9204ee2d2d 3.50 2023-05-11 08:03:11 -07:00
chrislu d999f1f0e2 update raft version
fix #4460
2023-05-09 22:54:23 -07:00
chrislu 25b237b83c reset wait time on a successful retry 2023-05-07 23:39:33 -07:00
chrislu 59f55c1a66 3.49 2023-05-07 14:00:19 -07:00
chrislu b2a3860290 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-05-07 13:33:47 -07:00
chrislu 4511edc871 update raft 2023-05-07 13:33:44 -07:00
themarkchen 7592d013fe
fix shell volume.balance bug (#4447) 2023-05-04 07:35:37 -07:00
chrislu 42766bccc8 3.48 2023-04-30 22:20:18 -07:00
zemul e9fda774f4
[Filer] post add param:saveInside (#4434)
* fix:mount deadlock

* feat: filer http upload to metadata

* feat: /etc save inside

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-29 08:31:05 -07:00
wusong 4867aa03ab
Fix: http rename move dir to subdir (#4432)
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-04-26 20:54:36 -07:00
Konstantin Lebedev d75a7b7f62
allow deleting only older empty dir without recursion (#4430) 2023-04-25 08:31:14 -07:00
zemul f352616b7d
Volume range read use sync.pool (#4422) 2023-04-24 07:00:48 -07:00
chrislu 18686b7375 3.47 2023-04-23 22:45:41 -07:00
zemul 68f1cc34a5
[mount]Fix copy file range (#4407)
* fix:mount deadlock

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-23 22:34:38 -07:00
chrislu e1b92bc701 reset local file content if remote file is updated
fix https://github.com/seaweedfs/seaweedfs/issues/4289
2023-04-22 22:32:21 -07:00
chrislu ff7b6d779e avoid overwriting variables
fix https://github.com/seaweedfs/seaweedfs/issues/4365
2023-04-22 11:49:59 -07:00
wusong 2e240704ab
Writables inconsistency (#4417)
fix: inconsistent read and write permissions between master and volume server

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-04-21 00:14:41 -07:00
wusong 19245dde50
mount: add retry for read only case (#4416)
* mount: add retry for read only case

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

* add new util retry function for mount

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

* change error list param

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

---------

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-04-21 00:09:31 -07:00
SmsS4 4131874fa8
use env for flags wtih dot in name (#4415) 2023-04-20 09:33:27 -07:00
Damiano Albani 9f55c7c90e
Rclone storage backend (#4402)
* Add Rclone storage backend

* Support templating the name of files stored via Rclone

* Enable Rclone accounting

* Remove redundant type conversion

* Provide progress information for Rclone download/upload operations

* Log error when Rclone can't instantiate filesystem

* Remove filename templating functionality for Rclone storage

To (maybe) be later reintroduced as a generic functionality for all
storage backends.

* Remove S3 specific check

* Move Rclone config initialisation to init() method
2023-04-18 14:12:12 -07:00
wusong a408b46d95
compilation fail (#4414)
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-04-18 00:36:55 -07:00
chrislu 0a22eea55d collect ec shard from multiple locations
fix https://github.com/seaweedfs/seaweedfs/issues/4365
2023-04-17 22:56:21 -07:00
chrislu ae9388723f adjust error message 2023-04-17 22:01:39 -07:00
chrislu 8ecdf958ab 3.46 2023-04-16 21:57:43 -07:00
Chengyu Liu 0f653d2153
update fuse client log which should output the fh of FileHandle. (#4404) 2023-04-15 22:15:16 -07:00
zemul 0122e022ea
Mount concurrent read (#4400)
* fix:mount deadlock

* feat: concurrent read

* fix

* Remove useless code

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-13 22:32:45 -07:00
Konstantin Lebedev 5614ad0000
fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter (#4399)
* fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter

* fix list with delimiter and start token

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-04-12 16:53:49 -07:00
Konstantin Lebedev 88b2bbadd1
try fix s3test test_bucket_listv2_delimiter_prefix (#4396) 2023-04-12 08:47:20 -07:00
Konstantin Lebedev 095cdb0c7f
fix test prefix is a directory (#4393)
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-04-12 00:08:13 -07:00
Konstantin Lebedev 44ad07276c
s3 fix get list of dir object key with slash suffix (#4391)
* s3 fix get list of dir object key with slash suffix
https://github.com/seaweedfs/seaweedfs/issues/3086

* list only entry dir eq prefix

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-04-11 23:52:35 -07:00
Konstantin Lebedev 130bc3e668
s3 fix get fake dir object key (#4390) 2023-04-11 07:36:22 -07:00
Matt da1d3b5a33
adjusted regex to be from 1 to 255 for the value (#4377)
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation

* adjusted regex to be from 1 to 255 for the value

* added better error message

* fixed regex
2023-04-10 12:24:38 -07:00
Matt 2d213d87e1
File Path Configuration TTL Validation (#4376)
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation
2023-04-04 22:07:17 -07:00
chrislu efef6e94bf use UPSERT for postgres style databases 2023-04-04 11:52:36 -07:00
chrislu 422bfaed69 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-04-02 22:53:02 -07:00
chrislu b9ab65dba5 3.45 2023-04-02 22:52:58 -07:00
wusong 2b82d0f871
style: clerical error (#4361) 2023-03-31 07:36:21 -07:00
LHHDZ 8e80ce9476
fix key corrupt when fs.configure copy path trie (#4353)
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-29 11:29:44 -07:00
chrislu 3cc7b90414 print wait in progress 2023-03-26 12:22:48 -07:00
chrislu f1f14e28bc adjust name 2023-03-26 12:17:23 -07:00
chrislu d54798c3ae Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-03-25 22:03:00 -07:00
chrislu 6bfb91b539 update entry Attributes.Mtime = time.Now().Unix()
fix https://github.com/seaweedfs/seaweedfs/issues/4337
2023-03-25 22:02:57 -07:00
renweijun bbd461b2c6
weed mount default EnableACL,Support chmod chown (#4335) 2023-03-25 04:47:53 -07:00
chrislu 5db9fcccd4 refactoring 2023-03-21 23:01:49 -07:00
Konstantin Lebedev de4545c28b
add validate config for raft (#4332) 2023-03-21 06:36:13 -07:00
chrislu adb90bd252 avoid lower casing the command
fix https://github.com/seaweedfs/seaweedfs/pull/4321
2023-03-19 21:20:46 -07:00
chrislu 5b43c4bb98 3.44 2023-03-19 20:27:55 -07:00
chrislu 50dc2fe96b cleaning variables 2023-03-19 18:48:40 -07:00
Konstantin Lebedev 29d1312625
Fs.verify.concurrency (#4293)
* init

* refactor

* refactor print msg

* param just concurrency

* fix race

* next fix race

* reset variables before each run

* clean

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-03-19 18:38:57 -07:00
chrislu 784daea1fa fix volume not found if marked as read only
fix https://github.com/seaweedfs/seaweedfs/issues/4088
2023-03-19 18:30:13 -07:00
Stewart Miles 264be0d2d4
Retry until a leader is selected. (#4318)
Fixes regression introduced in
https://github.com/seaweedfs/seaweedfs/pull/4313

Related to #4307
2023-03-16 20:50:38 -07:00
LHHDZ d695119073
The filer does not set defaultReplication to the defaultReplication o… (#4315)
The filer does not set defaultReplication to the defaultReplication of the master (it is not necessary, because if the filer is not set, the configuration of the master will be used when calling Assign), otherwise, when the defaultReplication of the master is modified and all master nodes are restarted, the defaultReplication will not take effect (because the filer Keep the previous defaultReplication setting from the master)

Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-15 22:10:39 -07:00
Stewart Miles 57ab1f8516
Use exponential backoff to query leader. (#4313)
`topology.Leader()` was using a backoff that typically
resulted in at least a 5s delay when initially starting
a master and raft server. This changes the backoff
algorithm to use exponential backoff starting with 100ms
and waiting up to 20s for leader selection.

Related to #4307
2023-03-15 17:49:46 -07:00
Stewart Miles dd71f54c6b
Fix -raftHashicorp and -raftBootstrap flag propagation. (#4309)
`weed server` was not correctly propagating
`-master.raftHashicorp` and `-master.raftBootstrap` flags when
starting the master server.

Related to #4307
2023-03-15 13:03:20 -07:00