Commit graph

766 commits

Author SHA1 Message Date
Seyed Mahdi Sadegh Shobeiri 97236389e8
Add modifyTimeAgo to volume.fsck (#5133)
* Add modifyTimeAgo to volume.fsck

* Fix AppendAtNs
2023-12-23 12:17:30 -08:00
Seyed Mahdi Sadegh Shobeiri 54ba2c8868
Fix cutoffTimeAgo in findMissingChunksInFiler (#5132) 2023-12-23 09:18:16 -08:00
Chris Lu 580940bf82
Merge accumulated changes related to message queue (#5098)
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
2023-12-11 12:05:54 -08:00
zehweh d818ca9631 fix TestMisplacedChecking() and add test-case 2023-11-21 08:24:34 -08:00
Amir H. Yeganemehr 6f559fe838 fs.mergeVolumes: change dir to a named parameter 2023-11-13 08:25:44 -08:00
Amir H. Yeganemehr d3556479be Fix typo (dist => dest) 2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr bb274ea8a3 fs.mergeVolumes: Make a plan based on volumes size 2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr b05d682c48 make help command in shell case insensitive 2023-11-13 06:06:45 -08:00
jsh 47112917ff ec.decode: mount the collected ec shards 2023-11-10 00:04:42 -08:00
zehweh 2b9dda7d2e
fix isMisplaced() in command_volume_fix_replication.go (#4988) 2023-11-07 07:58:19 -08:00
chrislu 2bc05f70e7 log full percentage 2023-10-22 12:59:34 -07:00
Konstantin Lebedev 1cac5d983d
fix: disallow file name too long when writing a file (#4881)
* fix: disallow file name too long when writing a file

* bool LongerName to MaxFilenameLength

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 14:29:55 -07:00
Konstantin Lebedev edee91ef0e
fix has changes false if deleted result size eq zero (#4909)
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 08:15:20 -07:00
Amir H. Yeganemehr 3e8604a4c5
add fs.mergeVolumes command into shell (#4907)
* add fs.compact command into shell

* rename fs.compact to fs.mergeVolumes
2023-10-10 12:47:18 -07:00
Konstantin Lebedev 2b3e39397e
fix: skipping checking active volumes with the same number of files at the moment (#4893)
* fix: skipping checking active volumes with the same number of files at the moment
 https://github.com/seaweedfs/seaweedfs/issues/4140

* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140

* add TestShouldSkipVolume

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-09 09:57:26 -07:00
chrislu 0fd7222d65 default to skip if less than 4 nodes 2023-10-05 11:13:48 -07:00
Konstantin Lebedev dffe00a822
fix: logger place msg (#4880) 2023-10-02 08:29:09 -07:00
Konstantin Lebedev dd580190b4
fix: avoid deleting one replica without sync (#4875)
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647

* Update weed/shell/command_volume_fix_replication.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* fix: revert this existing do option to positive

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-27 23:12:10 -07:00
chrislu 9d589b48e6 rename function 2023-09-26 15:17:33 -07:00
Konstantin Lebedev df4ded758e
fix: avoid deleting more than one replica (#4873)
https://github.com/seaweedfs/seaweedfs/issues/4647

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-26 00:20:48 -07:00
chrislu 19505c1cf4 describe a topic 2023-09-25 20:46:00 -07:00
Tobias Gurtzick 78dbac7702
fix(ec): volumes created by foreign collection due to bug in ec balance (#4864)
* fix(ec): ignore 0 byte data files

refers to parts of #4861

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>

* fix(ec): ignore volumes not from the current collection during balance

fixes #4861

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>

---------

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
2023-09-25 10:35:43 -07:00
chrislu 645ae8c57b Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs""
This reverts commit 8cb42c39
2023-09-25 09:35:16 -07:00
chrislu 3cf9b8d621 create topic and report topic 2023-09-24 22:00:43 -07:00
chrislu b3f94feede do create topic 2023-09-24 22:00:43 -07:00
chrislu 0361c321b4 add CreateTopic API 2023-09-24 22:00:43 -07:00
chrislu 8cb42c39ad Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs"
This reverts commit 2e5aa06026, reversing
changes made to 4d414f54a2.
2023-09-18 16:12:50 -07:00
dependabot[bot] a04bd4d26f
Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 (#4850)
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0

Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.63.1 to 1.64.0.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* API changes

* go mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2023-09-18 14:43:05 -07:00
Nico D'Cotta 796b7508f3
Implement SRV lookups for filer (#4767) 2023-08-24 07:08:56 -07:00
SmsS4 1e72579ac7
Fix s3.clean.uploads (#4766) 2023-08-19 06:30:18 -07:00
chrislu 3365468d0d added an error message 2023-08-08 20:35:21 -07:00
wusong 3ec977cd6c
minor (#4723) 2023-08-01 02:34:59 -07:00
chrislu 31b2751aff clone volume locations in case they are changed
fix https://github.com/seaweedfs/seaweedfs/issues/4642
2023-07-06 00:32:58 -07:00
chrislu 1ce725728d minor 2023-06-19 18:43:41 -07:00
chrislu 8ec1bc2c99 remove unused cluster node leader 2023-06-19 18:19:13 -07:00
柏杰 0b0fb9b9e4
avoid data race read volume.IsEmpty (#4574)
* avoid data race read volume.IsEmpty

-   avoid phantom read isEmpty for onlyEmpty
-   use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope

* add Destroy(onlyEmpty: true) test

* add Destroy(onlyEmpty: false) test

* remove unused `IsEmpty()`

* change literal `8` to `SuperBlockSize`
2023-06-14 14:39:58 -07:00
Konstantin Lebedev 25535e9c36
Delete volume is empty (#4561)
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559

* fix IsEmpty

* fix test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 10:42:44 -07:00
Konstantin Lebedev 5caff2c0f0
fix: deletes newly created volumes (#4551)
https://github.com/seaweedfs/seaweedfs/issues/4531

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-06 09:08:27 -07:00
mervynzhang 1ebb549f77
support swift (#4480) 2023-05-19 06:39:25 -07:00
SmsS4 17e91d2917
Use filerGroup for s3 buckets collection prefix (#4465)
* Use filerGroup for s3 buckets collection prefix

* Fix templates

* Remove flags

* Remove s3CollectionPrefix
2023-05-16 09:39:43 -07:00
themarkchen 7592d013fe
fix shell volume.balance bug (#4447) 2023-05-04 07:35:37 -07:00
chrislu e1b92bc701 reset local file content if remote file is updated
fix https://github.com/seaweedfs/seaweedfs/issues/4289
2023-04-22 22:32:21 -07:00
Matt da1d3b5a33
adjusted regex to be from 1 to 255 for the value (#4377)
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation

* adjusted regex to be from 1 to 255 for the value

* added better error message

* fixed regex
2023-04-10 12:24:38 -07:00
Matt 2d213d87e1
File Path Configuration TTL Validation (#4376)
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation
2023-04-04 22:07:17 -07:00
chrislu f1f14e28bc adjust name 2023-03-26 12:17:23 -07:00
chrislu 50dc2fe96b cleaning variables 2023-03-19 18:48:40 -07:00
Konstantin Lebedev 29d1312625
Fs.verify.concurrency (#4293)
* init

* refactor

* refactor print msg

* param just concurrency

* fix race

* next fix race

* reset variables before each run

* clean

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-03-19 18:38:57 -07:00
chrislu 2268d38263 fix naming
fix https://github.com/seaweedfs/seaweedfs/issues/4305
2023-03-14 08:38:02 -07:00
chrislu 98bc93cf31 fix tests 2023-03-13 10:00:27 -07:00
Konstantin Lebedev e17429223e
shell script unclean variables (#4298) 2023-03-13 07:16:31 -07:00