gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
10026 commits 28 branches 278 tags 135 MiB
Commit graph

67 commits

Author SHA1 Message Date
chrislu 35869b5c80 subscriber can be notified of the assignment change when topic is just configured 
Next: Subscriber needs to read by the timestamp offset.
 2024-01-03 13:30:30 -08:00
chrislu efb695fd93 lookup existing topic partitions 2024-01-02 17:29:35 -08:00
chrislu 093fdc1621 subscriber can get assignments 2023-12-28 20:35:15 -08:00
chrislu c950a40aad subscriber keep connected to the balancer 2023-12-28 11:56:37 -08:00
chrislu e0727071c8 go fmt 2023-12-22 11:33:50 -08:00
chrislu 034db049a0 comment fix 2023-12-22 11:33:00 -08:00
Chris Lu 580940bf82
Merge accumulated changes related to message queue (#5098) 
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
 2023-12-11 12:05:54 -08:00
chrislu 17710e1ecb adjust logs 2023-11-15 18:11:35 -08:00
chrislu de0b969b36 Revert "rename" 
This reverts commit 35b5264ab7.
 2023-10-12 20:28:11 -07:00
chrislu 35b5264ab7 rename 2023-10-11 21:44:56 -07:00
chrislu 734178093e refactor TopicPartition struct 2023-10-02 01:01:45 -07:00
chrislu ebd6f96d35 adjust mq.proto 2023-09-30 21:47:57 -07:00
chrislu 02cb799481 add a test framework for grpc 2023-09-30 21:36:58 -07:00
chrislu c4e6545197 fix compilation 2023-09-30 21:26:09 -07:00
chrislu 9d589b48e6 rename function 2023-09-26 15:17:33 -07:00
chrislu 23f334dd13 comment 2023-09-24 23:36:43 -07:00
chrislu c7e05e4e71 ensure latest stats are reported 2023-09-24 23:34:31 -07:00
chrislu dff2ce5d2f read existing broker stats 2023-09-24 23:10:33 -07:00
chrislu 1492bf7552 fix listing topics 2023-09-24 23:05:41 -07:00
chrislu 3cf9b8d621 create topic and report topic 2023-09-24 22:00:43 -07:00
chrislu b3f94feede do create topic 2023-09-24 22:00:43 -07:00
chrislu d74348048a implement create topic 2023-09-24 22:00:43 -07:00
chrislu 0361c321b4 add CreateTopic API 2023-09-24 22:00:43 -07:00
chrislu b18112ae10 fix compilation bugs during merge 2023-09-19 14:08:17 -07:00
chrislu 6c6655de9d wip 2023-09-19 14:03:42 -07:00
chrislu 27af11f1e8 Revert "Revert "Merge branch 'master' into sub"" 
This reverts commit 0bb97709d4.
 2023-09-18 18:47:34 -07:00
chrislu 0bb97709d4 Revert "Merge branch 'master' into sub" 
This reverts commit 4d414f54a2, reversing
changes made to 4827425146.
 2023-09-18 16:13:20 -07:00
chrislu 4d414f54a2 Merge branch 'master' into sub 2023-09-18 16:04:59 -07:00
chrislu a65088d739 broker report stats to balancer 2023-09-16 23:00:51 -07:00
chrislu 89a1fd1751 Squashed commit of the following: 
commit 4827425146
Author: chrislu <chris.lu@gmail.com>
Date:   Sat Sep 16 15:05:38 2023 -0700

    balancer works

commit 3b50139f68
Author: chrislu <chris.lu@gmail.com>
Date:   Fri Sep 15 22:22:32 2023 -0700

    comments

commit 7f685ce7ba
Author: chrislu <chris.lu@gmail.com>
Date:   Fri Sep 15 22:20:05 2023 -0700

    adjust APIs

commit 436d99443b
Author: chrislu <chris.lu@gmail.com>
Date:   Thu Sep 14 23:49:05 2023 -0700

    receive broker stats

commit b771fefa37
Merge: 0a851ec00 890881037
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 13 00:03:47 2023 -0700

    Merge branch 'master' into sub

commit 0a851ec00b
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Sep 10 22:01:25 2023 -0700

    Create balancer.go

commit 39941edc0b
Author: chrislu <chris.lu@gmail.com>
Date:   Thu Sep 7 23:55:19 2023 -0700

    add publisher shutdown

commit 875f562779
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 6 23:16:41 2023 -0700

    server side send response at least once per second

commit 984b6c54cf
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 6 23:15:29 2023 -0700

    ack interval 128

commit 2492a45499
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 6 22:39:46 2023 -0700

    ack interval

commit ba67e6ca29
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Sep 4 21:43:50 2023 -0700

    api for sub

commit 9e4f985698
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Sep 4 21:43:30 2023 -0700

    publish, benchmark

commit cb470d44df
Author: chrislu <chris.lu@gmail.com>
Date:   Fri Sep 1 00:36:51 2023 -0700

    can pub and sub

commit 1eb2da46d5
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Aug 28 09:02:12 2023 -0700

    connect and publish

commit 504ae8383a
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Aug 28 09:01:25 2023 -0700

    protoc version

commit dbcba75271
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 18:59:04 2023 -0700

    rename to lookup

commit c9caf33119
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 18:33:46 2023 -0700

    move functions

commit 4d6c18d86f
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 17:50:59 2023 -0700

    pub sub initial tests

commit 4eb8e8624d
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 13:14:39 2023 -0700

    rename

commit 1990456670
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 13:13:14 2023 -0700

    sub

commit 905911853d
Author: chrislu <chris.lu@gmail.com>
Date:   Sat Aug 26 13:39:21 2023 -0700

    adjust proto
 2023-09-16 15:06:16 -07:00
chrislu 4827425146 balancer works 2023-09-16 15:05:38 -07:00
chrislu 436d99443b receive broker stats 2023-09-14 23:49:05 -07:00
chrislu b771fefa37 Merge branch 'master' into sub 2023-09-13 00:03:47 -07:00
chrislu 39941edc0b add publisher shutdown 2023-09-07 23:55:19 -07:00
chrislu 875f562779 server side send response at least once per second 2023-09-06 23:16:41 -07:00
chrislu 2492a45499 ack interval 2023-09-06 22:39:46 -07:00
chrislu ba67e6ca29 api for sub 2023-09-04 21:43:50 -07:00
chrislu 9e4f985698 publish, benchmark 2023-09-04 21:43:30 -07:00
chrislu cb470d44df can pub and sub 2023-09-01 00:36:51 -07:00
chrislu 1eb2da46d5 connect and publish 2023-08-28 09:02:12 -07:00
chrislu dbcba75271 rename to lookup 2023-08-27 18:59:04 -07:00
chrislu c9caf33119 move functions 2023-08-27 18:33:46 -07:00
chrislu 4d6c18d86f pub sub initial tests 2023-08-27 17:50:59 -07:00
chrislu 4eb8e8624d rename 2023-08-27 13:14:39 -07:00
chrislu 1990456670 sub 2023-08-27 13:13:14 -07:00
chrislu 905911853d adjust proto 2023-08-26 13:39:21 -07:00
Nico D'Cotta 796b7508f3
Implement SRV lookups for filer (#4767) 2023-08-24 07:08:56 -07:00
chrislu 01d70c21f3 Squashed commit of the following: 
commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 22:52:19 2023 -0700

    fix compilation

commit e77ad33b7ca0423138fbae26a4433b60923a9588
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 22:46:44 2023 -0700

    pub

commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 13:27:39 2023 -0700

    fix generic type

commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d
Merge: 30f942580 16e3f2d52
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 12:47:14 2023 -0700

    Merge branch 'master' into pubsub

commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 11:10:58 2023 -0700

    wip

commit f8b00980bc2f3879bb43decffd9a08d842f196f2
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:14:35 2023 -0700

    add design document

commit 08d2bebe42a26ebc39f1542f54d99e73620727dd
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:14:06 2023 -0700

    minor

commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:13:49 2023 -0700

    rename
 2023-08-20 22:53:05 -07:00
chrislu 8ec1bc2c99 remove unused cluster node leader 2023-06-19 18:19:13 -07:00
chrislu 81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00