gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9607 commits 28 branches 278 tags 135 MiB
Commit graph

6218 commits

Author SHA1 Message Date
wusong c44a6552d2
Fix buff use (#4628) 2023-07-03 08:33:34 -07:00
Amir H. Yeganemehr f0d3c874b6
Unix Socket listener for S3 server (#4621) 
Add localSocket option to s3 server
 2023-06-26 16:22:45 -07:00
chrislu 2c4c2f0994 3.53 2023-06-25 22:51:14 -07:00
chrislu f193942adc lock on append metadata operation 2023-06-25 22:07:23 -07:00
chrislu 61c42f9991 adjust lock APIs 2023-06-25 20:30:20 -07:00
chrislu aafb874ad2 lock until cancelled 2023-06-25 19:31:25 -07:00
chrislu c4f44d56f7 add distributed lock client 2023-06-25 17:38:34 -07:00
chrislu f8aa5ea844 adjust filer.proto 2023-06-25 17:38:19 -07:00
chrislu 13a8593ab9 adjust errors 2023-06-25 17:37:54 -07:00
chrislu 06471dac9d init lock ring 2023-06-25 15:28:16 -07:00
chrislu 5149b3d07b filer can proxy to peer filer holding the lock 2023-06-25 15:28:01 -07:00
chrislu 868f7875d7 refactor 2023-06-25 14:30:58 -07:00
chrislu 26eff062c8 refactor 2023-06-25 14:17:58 -07:00
chrislu 464a71a373 add distributed lock manager 2023-06-25 14:14:40 -07:00
chrislu ee4f7cd636 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-06-25 00:58:24 -07:00
chrislu 3fd659df2a add distributed lock manager 2023-06-25 00:58:21 -07:00
Damiano Albani 0d9ba8c612
Support templating name of files stored via Rclone backend (#4606) 2023-06-24 07:08:24 -07:00
chrislu a82c44972b refactor 2023-06-20 00:12:59 -07:00
chrislu 14a19170e2 refactor 2023-06-19 23:51:04 -07:00
chrislu 84ced2abf5 add LockRing 2023-06-19 23:48:33 -07:00
chrislu fe1716ccdf remove deprecated tests 2023-06-19 18:43:51 -07:00
chrislu 1ce725728d minor 2023-06-19 18:43:41 -07:00
chrislu 11549706ef refactor 2023-06-19 18:21:07 -07:00
chrislu 8ec1bc2c99 remove unused cluster node leader 2023-06-19 18:19:13 -07:00
Bai Jie 3b88ab42aa
remove duplicate fileCount query (#4588) 2023-06-18 00:14:14 -07:00
Bai Jie 44b9d72ef0
doIsEmpty() return error if v.DataBackend is nil (#4587) 2023-06-18 00:13:40 -07:00
chrislu 94fbf02ea2 generate jwt token when looking up needle id 
fix https://github.com/seaweedfs/seaweedfs/issues/4577
 2023-06-17 00:11:23 -07:00
柏杰 0b0fb9b9e4
avoid data race read volume.IsEmpty (#4574) 
* avoid data race read volume.IsEmpty

-   avoid phantom read isEmpty for onlyEmpty
-   use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope

* add Destroy(onlyEmpty: true) test

* add Destroy(onlyEmpty: false) test

* remove unused `IsEmpty()`

* change literal `8` to `SuperBlockSize`
 2023-06-14 14:39:58 -07:00
Konstantin Lebedev 1e22d5caf2
fix get file stats for IsEmpty (#4576) 2023-06-14 01:43:30 -07:00
Konstantin Lebedev 4dd890d4a2
optional https port for s3 (#4482) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-13 23:58:49 -07:00
chrislu 3fbf4f6189 Revert "Initial patch for redis version 7 support (#4572)" 
This reverts commit a25bca0692.
 2023-06-12 22:27:38 -07:00
Konstantin Lebedev 4527ead295
fix from comment delete volume is empty (#4573) 
* fix from coments
https://github.com/seaweedfs/seaweedfs/pull/4561

* fix tests

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 22:22:46 -07:00
Aaron Gipson a25bca0692
Initial patch for redis version 7 support (#4572) 2023-06-12 13:30:03 -07:00
Konstantin Lebedev 25535e9c36
Delete volume is empty (#4561) 
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559

* fix IsEmpty

* fix test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 10:42:44 -07:00
柏杰 68feffa4e6
avoid overwriting non-nil uploadReaderToChunks.uploadErr with nil value (#4560) 2023-06-10 23:15:27 -07:00
Konstantin Lebedev 5ee04d20fa
Healthz check for deadlocks (#4558) 2023-06-09 09:42:48 -07:00
wusong d6af3a8eb6
fix 404 not found (#4536) 
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-06 09:14:49 -07:00
Konstantin Lebedev 5caff2c0f0
fix: deletes newly created volumes (#4551) 
https://github.com/seaweedfs/seaweedfs/issues/4531

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-06 09:08:27 -07:00
wusong 9bdbf9c880
revert #4491 (#4550) 
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-06 00:17:51 -07:00
chrislu 0bffb3770f support TTL when creating folders 
related to https://github.com/seaweedfs/seaweedfs/discussions/4549
 2023-06-05 23:29:25 -07:00
wusong 26f15d0079
Fix no more writable volumes by delay judgment (#4548) 
* fix nomore writables volumes while disk free space is sufficient by time delay

* reset

---------

Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-05 10:17:21 -07:00
chrislu fb4b61036c 3.52 2023-06-04 22:34:26 -07:00
Konstantin Lebedev a0931be0c0
S3 TLS credentials Refreshing (#4506) 
* S3 TLS credentials Refreshing

* fix: logging

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-04 14:27:56 -07:00
wusong 5aec6da8a3
fix deadlock for filer upload (#4527) 
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-03 13:38:27 -07:00
chrislu e23f3d6eca set df.File to nil after it is closed 
possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530
 2023-06-02 17:16:45 -07:00
chrislu ca7cc61319 reduce the window size between unregistering a volume server and creating volumes on that server 
fix https://github.com/seaweedfs/seaweedfs/issues/4467
 2023-05-30 08:20:00 -07:00
chrislu 68c3cf767c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-05-29 16:45:28 -07:00
chrislu d1d1757520 adjust error message 2023-05-29 16:45:25 -07:00
wusong e7f4418855
add return for error handle (#4502) 2023-05-24 07:45:06 -07:00
chrislu 4310e1fac4 3.51 2023-05-21 23:43:39 -07:00
wusong 8fffe3e822
fix no more writables volumes while disk free space is sufficient (#4491) 
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-05-21 22:18:50 -07:00
mervynzhang 1ebb549f77
support swift (#4480) 2023-05-19 06:39:25 -07:00
Mesar Hameed a90b777ff4
Feat: etcd filer store keys should have customizable prefix (#4484) 
An etcd cluster is not necessarily only dedicated to seaweedfs.
This security enhancement adds a customizable key_prefix option to the etcd filer store.
This will allow an etcd cluster administrator to limit the seaweedfs etcd user to only read/write a subset of keys under the
key_prefix, instead of all keys on the etcd cluster.
 2023-05-18 23:08:56 -07:00
SmsS4 17e91d2917
Use filerGroup for s3 buckets collection prefix (#4465) 
* Use filerGroup for s3 buckets collection prefix

* Fix templates

* Remove flags

* Remove s3CollectionPrefix
 2023-05-16 09:39:43 -07:00
chrislu b7f011f777 fix test 2023-05-15 17:46:56 -07:00
Mesar Hameed 65484e80ef
Feat: support username/password authentication for etcd filer store s… (#4477) 
Feat: support username/password authentication for etcd filer store seaweedfs/seaweedfs#4262

Co-authored-by: Mesar Hameed <mesar.hameed@gmail.com>
 2023-05-15 17:45:14 -07:00
chrislu 9204ee2d2d 3.50 2023-05-11 08:03:11 -07:00
chrislu d999f1f0e2 update raft version 
fix #4460
 2023-05-09 22:54:23 -07:00
chrislu 25b237b83c reset wait time on a successful retry 2023-05-07 23:39:33 -07:00
chrislu 59f55c1a66 3.49 2023-05-07 14:00:19 -07:00
chrislu b2a3860290 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-05-07 13:33:47 -07:00
chrislu 4511edc871 update raft 2023-05-07 13:33:44 -07:00
themarkchen 7592d013fe
fix shell volume.balance bug (#4447) 2023-05-04 07:35:37 -07:00
chrislu 42766bccc8 3.48 2023-04-30 22:20:18 -07:00
zemul e9fda774f4
[Filer] post add param:saveInside (#4434) 
* fix:mount deadlock

* feat: filer http upload to metadata

* feat: /etc save inside

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-04-29 08:31:05 -07:00
wusong 4867aa03ab
Fix: http rename move dir to subdir (#4432) 
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-04-26 20:54:36 -07:00
Konstantin Lebedev d75a7b7f62
allow deleting only older empty dir without recursion (#4430) 2023-04-25 08:31:14 -07:00
zemul f352616b7d
Volume range read use sync.pool (#4422) 2023-04-24 07:00:48 -07:00
chrislu 18686b7375 3.47 2023-04-23 22:45:41 -07:00
zemul 68f1cc34a5
[mount]Fix copy file range (#4407) 
* fix:mount deadlock

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-04-23 22:34:38 -07:00
chrislu e1b92bc701 reset local file content if remote file is updated 
fix https://github.com/seaweedfs/seaweedfs/issues/4289
 2023-04-22 22:32:21 -07:00
chrislu ff7b6d779e avoid overwriting variables 
fix https://github.com/seaweedfs/seaweedfs/issues/4365
 2023-04-22 11:49:59 -07:00
wusong 2e240704ab
Writables inconsistency (#4417) 
fix: inconsistent read and write permissions between master and volume server

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
 2023-04-21 00:14:41 -07:00
wusong 19245dde50
mount: add retry for read only case (#4416) 
* mount: add retry for read only case

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

* add new util retry function for mount

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

* change error list param

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

---------

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
 2023-04-21 00:09:31 -07:00
SmsS4 4131874fa8
use env for flags wtih dot in name (#4415) 2023-04-20 09:33:27 -07:00
Damiano Albani 9f55c7c90e
Rclone storage backend (#4402) 
* Add Rclone storage backend

* Support templating the name of files stored via Rclone

* Enable Rclone accounting

* Remove redundant type conversion

* Provide progress information for Rclone download/upload operations

* Log error when Rclone can't instantiate filesystem

* Remove filename templating functionality for Rclone storage

To (maybe) be later reintroduced as a generic functionality for all
storage backends.

* Remove S3 specific check

* Move Rclone config initialisation to init() method
 2023-04-18 14:12:12 -07:00
wusong a408b46d95
compilation fail (#4414) 
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
 2023-04-18 00:36:55 -07:00
chrislu 0a22eea55d collect ec shard from multiple locations 
fix https://github.com/seaweedfs/seaweedfs/issues/4365
 2023-04-17 22:56:21 -07:00
chrislu ae9388723f adjust error message 2023-04-17 22:01:39 -07:00
chrislu 8ecdf958ab 3.46 2023-04-16 21:57:43 -07:00
Chengyu Liu 0f653d2153
update fuse client log which should output the fh of FileHandle. (#4404) 2023-04-15 22:15:16 -07:00
zemul 0122e022ea
Mount concurrent read (#4400) 
* fix:mount deadlock

* feat: concurrent read

* fix

* Remove useless code

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-04-13 22:32:45 -07:00
Konstantin Lebedev 5614ad0000
fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter (#4399) 
* fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter

* fix list with delimiter and start token

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-12 16:53:49 -07:00
Konstantin Lebedev 88b2bbadd1
try fix s3test test_bucket_listv2_delimiter_prefix (#4396) 2023-04-12 08:47:20 -07:00
Konstantin Lebedev 095cdb0c7f
fix test prefix is a directory (#4393) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-12 00:08:13 -07:00
Konstantin Lebedev 44ad07276c
s3 fix get list of dir object key with slash suffix (#4391) 
* s3 fix get list of dir object key with slash suffix
https://github.com/seaweedfs/seaweedfs/issues/3086

* list only entry dir eq prefix

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-11 23:52:35 -07:00
Konstantin Lebedev 130bc3e668
s3 fix get fake dir object key (#4390) 2023-04-11 07:36:22 -07:00
Matt da1d3b5a33
adjusted regex to be from 1 to 255 for the value (#4377) 
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation

* adjusted regex to be from 1 to 255 for the value

* added better error message

* fixed regex
 2023-04-10 12:24:38 -07:00
Matt 2d213d87e1
File Path Configuration TTL Validation (#4376) 
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation
 2023-04-04 22:07:17 -07:00
chrislu efef6e94bf use UPSERT for postgres style databases 2023-04-04 11:52:36 -07:00
chrislu 422bfaed69 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-04-02 22:53:02 -07:00
chrislu b9ab65dba5 3.45 2023-04-02 22:52:58 -07:00
wusong 2b82d0f871
style: clerical error (#4361) 2023-03-31 07:36:21 -07:00
LHHDZ 8e80ce9476
fix key corrupt when fs.configure copy path trie (#4353) 
Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2023-03-29 11:29:44 -07:00
chrislu 3cc7b90414 print wait in progress 2023-03-26 12:22:48 -07:00
chrislu f1f14e28bc adjust name 2023-03-26 12:17:23 -07:00
chrislu d54798c3ae Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-03-25 22:03:00 -07:00
chrislu 6bfb91b539 update entry Attributes.Mtime = time.Now().Unix() 
fix https://github.com/seaweedfs/seaweedfs/issues/4337
 2023-03-25 22:02:57 -07:00
renweijun bbd461b2c6
weed mount default EnableACL，Support chmod chown (#4335) 2023-03-25 04:47:53 -07:00
chrislu 5db9fcccd4 refactoring 2023-03-21 23:01:49 -07:00