Commit graph

501 commits

Author SHA1 Message Date
chrislu c950a40aad subscriber keep connected to the balancer 2023-12-28 11:56:37 -08:00
Chris Lu 580940bf82
Merge accumulated changes related to message queue (#5098)
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
2023-12-11 12:05:54 -08:00
chrislu deb86ff4a6 upgrading grpc 2023-10-12 21:38:34 -07:00
chrislu de0b969b36 Revert "rename"
This reverts commit 35b5264ab7.
2023-10-12 20:28:11 -07:00
Konstantin Lebedev 1cac5d983d
fix: disallow file name too long when writing a file (#4881)
* fix: disallow file name too long when writing a file

* bool LongerName to MaxFilenameLength

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 14:29:55 -07:00
chrislu 35b5264ab7 rename 2023-10-11 21:44:56 -07:00
Konstantin Lebedev 2b3e39397e
fix: skipping checking active volumes with the same number of files at the moment (#4893)
* fix: skipping checking active volumes with the same number of files at the moment
 https://github.com/seaweedfs/seaweedfs/issues/4140

* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140

* add TestShouldSkipVolume

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-09 09:57:26 -07:00
chrislu b02fdeabff RetryForever => RetryUntil 2023-10-01 11:33:56 -07:00
chrislu ebd6f96d35 adjust mq.proto 2023-09-30 21:47:57 -07:00
chrislu 358cba43ef update proto generated files 2023-09-30 13:19:25 -07:00
chrislu d7c211fe8c adjust mq proto 2023-09-30 13:18:49 -07:00
chrislu 9d589b48e6 rename function 2023-09-26 15:17:33 -07:00
Konstantin Lebedev f8b94cac0e
[s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859)
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface

https://github.com/seaweedfs/seaweedfs/issues/4519

* fix: test bucket acl default and
adjust the variable names

* fix: s3 api config test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-25 08:34:12 -07:00
chrislu 3cf9b8d621 create topic and report topic 2023-09-24 22:00:43 -07:00
chrislu b3f94feede do create topic 2023-09-24 22:00:43 -07:00
chrislu 0361c321b4 add CreateTopic API 2023-09-24 22:00:43 -07:00
chrislu 6c6655de9d wip 2023-09-19 14:03:42 -07:00
chrislu 4827425146 balancer works 2023-09-16 15:05:38 -07:00
chrislu 436d99443b receive broker stats 2023-09-14 23:49:05 -07:00
chrislu b771fefa37 Merge branch 'master' into sub 2023-09-13 00:03:47 -07:00
chrislu 2492a45499 ack interval 2023-09-06 22:39:46 -07:00
chrislu ba67e6ca29 api for sub 2023-09-04 21:43:50 -07:00
chrislu cb470d44df can pub and sub 2023-09-01 00:36:51 -07:00
chrislu 504ae8383a protoc version 2023-08-28 09:01:25 -07:00
chrislu dbcba75271 rename to lookup 2023-08-27 18:59:04 -07:00
chrislu 4d6c18d86f pub sub initial tests 2023-08-27 17:50:59 -07:00
chrislu 1990456670 sub 2023-08-27 13:13:14 -07:00
chrislu 905911853d adjust proto 2023-08-26 13:39:21 -07:00
Nico D'Cotta 796b7508f3
Implement SRV lookups for filer (#4767) 2023-08-24 07:08:56 -07:00
chrislu 010c5e91e3 add stream assign proto 2023-08-22 09:53:54 -07:00
chrislu 01d70c21f3 Squashed commit of the following:
commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 22:52:19 2023 -0700

    fix compilation

commit e77ad33b7ca0423138fbae26a4433b60923a9588
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 22:46:44 2023 -0700

    pub

commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 13:27:39 2023 -0700

    fix generic type

commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d
Merge: 30f942580 16e3f2d52
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 12:47:14 2023 -0700

    Merge branch 'master' into pubsub

commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 11:10:58 2023 -0700

    wip

commit f8b00980bc2f3879bb43decffd9a08d842f196f2
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:14:35 2023 -0700

    add design document

commit 08d2bebe42a26ebc39f1542f54d99e73620727dd
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:14:06 2023 -0700

    minor

commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:13:49 2023 -0700

    rename
2023-08-20 22:53:05 -07:00
chrislu f8aa5ea844 adjust filer.proto 2023-06-25 17:38:19 -07:00
chrislu 5149b3d07b filer can proxy to peer filer holding the lock 2023-06-25 15:28:01 -07:00
chrislu 464a71a373 add distributed lock manager 2023-06-25 14:14:40 -07:00
chrislu 3fd659df2a add distributed lock manager 2023-06-25 00:58:21 -07:00
chrislu 8ec1bc2c99 remove unused cluster node leader 2023-06-19 18:19:13 -07:00
Konstantin Lebedev 25535e9c36
Delete volume is empty (#4561)
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559

* fix IsEmpty

* fix test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 10:42:44 -07:00
wusong 26f15d0079
Fix no more writable volumes by delay judgment (#4548)
* fix nomore writables volumes while disk free space is sufficient by time delay

* reset

---------

Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-05 10:17:21 -07:00
mervynzhang 1ebb549f77
support swift (#4480) 2023-05-19 06:39:25 -07:00
Konstantin Lebedev d75a7b7f62
allow deleting only older empty dir without recursion (#4430) 2023-04-25 08:31:14 -07:00
chrislu 5db9fcccd4 refactoring 2023-03-21 23:01:49 -07:00
Muhammad Hallaj bin Subery 9bd422d2c9
adding support for B2 region (#4177)
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
2023-02-05 21:24:21 -08:00
chrislu 81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
Guo Lei d8cfa1552b
support enable/disable vacuum (#4087)
* stop vacuum

* suspend/resume vacuum

* remove unused code

* rename

* rename param
2022-12-28 01:36:44 -08:00
chrislu e1ca6308cb add chunk etag when downloading from remote storage
fix https://github.com/seaweedfs/seaweedfs/issues/3987
2022-12-10 21:49:07 -08:00
James Hartig 81624de27b
Include name/mime in ReadAllNeedles (#4005) 2022-11-23 15:59:38 -08:00
James Hartig 4c85da7844
Include meta in ReadAllNeedles (#3991)
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.

Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
2022-11-20 20:19:41 -08:00
chrislu 70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
Eric Yang 51d462f204
ADHOC: volume fsck using append at ns (#3906)
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00