seaweedfs/weed/mq/pub_balancer/balance_action_split.go

44 lines
1.2 KiB
Go
Raw Permalink Normal View History

Merge accumulated changes related to message queue (#5098) * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * tracking topic=>broker * merge * comment
2023-12-11 20:05:54 +00:00
package pub_balancer
/*
Sequence of operations to ensure ordering
Assuming Publisher P10 is publishing to Topic Partition TP10, and Subscriber S10 is subscribing to Topic TP10.
After splitting Topic TP10 into Topic Partition TP11 and Topic Partition TP21,
Publisher P11 is publishing to Topic Partition TP11, and Publisher P21 is publishing to Topic Partition TP21.
Subscriber S12 is subscribing to Topic Partition TP11, and Subscriber S21 is subscribing to Topic Partition TP21.
(The last digit is ephoch generation number, which is increasing when the topic partitioning is changed.)
The diagram is as follows:
P10 -> TP10 -> S10
||
\/
P11 -> TP11 -> S11
P21 -> TP21 -> S21
The following is the sequence of events:
1. Create Topic Partition TP11 and TP21
2. Close Publisher(s) P10
3. Close Subscriber(s) S10
4. Close Topic Partition TP10
5. Start Publisher P11, P21
6. Start Subscriber S11, S21
The dependency is as follows:
2 => 3 => 4
| |
v v
1 => (5 | 6)
And also:
2 => 5
3 => 6
For brokers:
1. Close all publishers for a topic partition
2. Close all subscribers for a topic partition
3. Close the topic partition
*/