Commit graph

202 commits

Author SHA1 Message Date
chrislu 88d3f51648 fix rocksdb build 2024-01-01 22:41:23 -08:00
Chris Lu 580940bf82
Merge accumulated changes related to message queue (#5098)
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
2023-12-11 12:05:54 -08:00
chrislu 556a242acf remove unused file 2023-11-30 08:20:01 -08:00
Konstantin Lebedev ce9b822a8e
golang up version to 1.21.3 (#4911) 2023-10-13 05:31:43 -07:00
Konstantin Lebedev f8b94cac0e
[s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859)
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface

https://github.com/seaweedfs/seaweedfs/issues/4519

* fix: test bucket acl default and
adjust the variable names

* fix: s3 api config test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-25 08:34:12 -07:00
Konstantin Lebedev a40888d343
fix: make dev_tls with volume https (#4866) 2023-09-25 06:47:38 -07:00
chrislu 1494221b08 reduce binary size 2023-09-21 11:58:03 -07:00
Konstantin Lebedev a46f873edd
[s3acl] Step 0: Put bucket ACL only responds success if the ACL is private. (#4856)
* Passing test:
test_bucket_acl_default
test_bucket_acl_canned_private_to_private

https://github.com/seaweedfs/seaweedfs/issues/4519

* Update weed/s3api/s3api_bucket_handlers.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-21 08:20:05 -07:00
chrislu b0da8788a1 upgrade rocksdb versions 2023-05-29 23:14:45 -07:00
chrislu 1f7934a89c enable gocdk in the generated local image
fix https://github.com/seaweedfs/seaweedfs/issues/4256
2023-02-27 19:20:02 -08:00
Chris Lu bac45912e3
go 1.20 (#4192) 2023-02-08 16:41:23 -08:00
CommanderRoot c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094) 2023-01-01 05:07:53 -08:00
Ryan Russell 949607fe90
refactor(compose)!: upgrade to v2 closes #3699 (#3705)
* docs(compose): Update readme for compose v2 upgrade

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(compose): Upgrade `Makefile` for compose v2

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(compose): Upgrade yml from version 2 -> 3.9

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-10-16 14:02:33 -07:00
Konstantin Lebedev b9933d5589
master server graceful stop (#3797) 2022-10-06 09:30:30 -07:00
Ryan Russell b51880ed3b
docs(docker): readability fixes (#3684)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:27:35 -07:00
chrislu 5afc20dbd3 add tmux for dev 2022-09-01 14:47:45 -07:00
chrislu 04300dc2ac build local rocksdb 2022-09-01 09:44:48 -07:00
chrislu 466bbe4cdd dockerfile fix directory 2022-09-01 09:43:41 -07:00
chrislu 9c76896e2d fix building rocksdb image 2022-09-01 00:20:23 -07:00
Patrick Schmidt a73e177ecf
Add an End-to-End workflow for FUSE mount (#3562)
* Add an e2e workflow to test FUSE mount

* Fix deadlocks during concurrent r/w
2022-08-31 09:27:53 -07:00
Konstantin Lebedev 913a5d0dd4
add target dev_race to MakeFile (#3522) 2022-08-26 01:28:03 -07:00
Konstantin Lebedev d8c54b527e
avoid Dockerfile Entrypoint: "file not found" (#3430)
https://www.koskila.net/how-to-fix-exec-user-process-caused-no-such-file-or-directory-in-docker-on-windows/
2022-08-11 15:07:44 -07:00
chrislu 8a61c124c7 go 1.19 2022-08-02 12:17:31 -07:00
chrislu 5fbe437ee9 update rocksdb version to 7.4.4 2022-07-31 19:47:27 -07:00
chrislu 1d05e7fcbd site move 2022-07-29 15:47:14 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev bcbdc4cb37 use const multipart uploads folder
avoid error bucket NotEmpty if multipart uploads folder exist
2022-06-29 16:21:16 +05:00
Chris Lu e22c2c8012
Merge pull request #3188 from kmlebedev/fix_make_dev
fix ip.bind cmd for docker compose
2022-06-29 01:36:59 -07:00
Konstantin Lebedev 5b388ed6c1 initial advancedtls 2022-06-23 23:32:15 +05:00
Konstantin Lebedev a9f6c4b43d fix ip.bind cmd for docker compose 2022-06-15 16:49:53 +05:00
Konstantin Lebedev fb57aa431c stats master_replica_placement_mismatch 2022-06-10 15:30:40 +05:00
Chris Lu f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud 2022-06-07 08:56:55 -07:00
Kaiwalya Joshi 47d335cf8c
feat: Send commands to weed shell from the docker image.
Add the ability to send commands to weed shell from the docker image.

Allows an operator to perform maintenance commands like so:
```
docker run \
  --rm \
  -e SHELL_FILER=localhost:8888 \
  -e SHELL_MASTER=localhost:9333 \
  chrislusf/seaweedfs:local \
  "shell" \
  "fs.configure -locationPrefix=/buckets/foo -volumeGrowthCount=3 -replication=002 -apply"
```
2022-06-01 15:47:10 -07:00
Konstantin Lebedev b86628f85d fix show empty dir
https://github.com/chrislusf/seaweedfs/issues/3086
2022-06-01 20:10:52 +05:00
Konstantin Lebedev a14728d508 docker compose nextcloud rm mysql 2022-05-23 20:11:32 +05:00
Konstantin Lebedev a34438384e docker compose file for nextcloud testing
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-23 18:53:35 +05:00
chrislu ca85b66511 fix build 2022-05-16 00:58:11 -07:00
Konstantin Lebedev 59435e4a2e rocksdb up version to v7.2.2 2022-05-15 01:22:55 +05:00
Konstantin Lebedev 8f762a1a06 build release containers for all tags 2022-05-14 18:40:32 +05:00
Konstantin Lebedev d12a423aa0 refactor
add some tests

https://github.com/chrislusf/seaweedfs/pull/2996
2022-05-12 12:23:02 +05:00
chrislu bbd3c586f6 docker compose: use ip.bind=0.0.0.0 2022-05-06 22:58:46 -07:00
chrislu 22fbce8fd5 add docker compose file to sync 2 mounts 2022-05-06 03:53:36 -07:00
chrislu b2a43dd044 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-06 01:51:28 -07:00
chrislu 362dec87a3 add tests 2022-05-06 01:50:34 -07:00
Konstantin Lebedev 8342f651f3 fix scanNamed 2022-05-03 15:18:28 +05:00
Konstantin Lebedev d8925b4e83 Merge branch 'new_master' into ydb
# Conflicts:
#	go.mod
#	go.sum
2022-05-03 00:13:57 +05:00
Konstantin Lebedev 319d300d48 fix createDB 2022-05-02 22:23:07 +05:00
Konstantin Lebedev 275c3bb19c ydb initial
https://github.com/chrislusf/seaweedfs/issues/2942
2022-05-01 17:28:58 +05:00
chrislu 192983b464 s3 backend support customizing storage class 2022-04-30 17:36:40 -07:00
chrislu cdb31c5b61 go 1.18 for rocksdb 2022-04-18 03:03:08 -07:00