gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
10051 commits 28 branches 278 tags 135 MiB
Commit graph

6452 commits

Author SHA1 Message Date
chrislu 6324062acc fix 2023-12-22 11:37:15 -08:00
chrislu e0727071c8 go fmt 2023-12-22 11:33:50 -08:00
chrislu ee1c9bc314 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-12-22 11:33:02 -08:00
chrislu 034db049a0 comment fix 2023-12-22 11:33:00 -08:00
chrislu 6df44d3c95 option to filer.copy with public volume server url 
fix https://github.com/seaweedfs/seaweedfs/issues/5126
 2023-12-22 11:32:43 -08:00
chrislu 7a0d1e7fc1 minor 2023-12-22 11:29:54 -08:00
Farbod c278f49bca
Using filer.remote.sync concurrency in filer.remote.gateway (#5123) 
- Chnaged ProcessEvenFn to be concurrent just like filer.remote.sync

Co-authored-by: itsfarbod <itsfarbod@hamravesh.com>
 2023-12-20 16:25:54 -08:00
jerebear12 06343f8976
Set allowed origins in config (#5109) 
* Add a way to use a JWT in an HTTP only cookie

If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.

* Added a way to specify allowed origins header from config

* Removed unecessary log

* Check list of domains from config or command flag

* Handle default wildcard and change name of config value to cors
 2023-12-20 16:21:11 -08:00
Konstantin Lebedev 3c9bcfb864
chore: add dsn for connection to mysql (#5060) 
* chore: add dsn for connection to mysql

* add comment

* new comment

* fix: validate dsn and adapt password
 2023-12-20 16:20:58 -08:00
chrislu 915416a277 fix wrong parameter 
fix https://github.com/seaweedfs/seaweedfs/pull/5102
 2023-12-13 12:02:57 -08:00
Farbod d5c01e5ceb
Remote gateway Bug fix (#5100) 2023-12-12 07:46:45 -08:00
chrislu 61b05fad95 fix test 2023-12-11 20:57:29 -08:00
Chris Lu 580940bf82
Merge accumulated changes related to message queue (#5098) 
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
 2023-12-11 12:05:54 -08:00
Konstantin Lebedev 125ad8fe63 falls back to update only if error contains msg "duplicate entry" 
https://github.com/seaweedfs/seaweedfs/issues/5062
 2023-12-09 10:18:18 -08:00
chrislu c1a6e624e5 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-12-07 08:13:46 -08:00
chrislu 5fe4c04a1a add missing error in the log 
related to https://github.com/seaweedfs/seaweedfs/issues/5084
 2023-12-07 08:13:42 -08:00
Lars Lehtonen 43a5973a66 weed/storage: fix dropped test errors 2023-12-06 12:05:22 -08:00
jerebear12 d5d9fbb8aa Add a way to use a JWT in an HTTP only cookie 
If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.
 2023-12-05 08:57:01 -08:00
chrislu 756bcc032d adjust logs 2023-11-27 12:57:29 -08:00
Konstantin Lebedev dd78397fea logging PickForWrite error 
https://github.com/seaweedfs/seaweedfs/issues/3886
 2023-11-27 12:56:15 -08:00
chrislu 81f11883e3 go fmt 2023-11-26 11:47:20 -08:00
chrislu 7a50762b9e reduce binary size by skipping rclone 2023-11-23 00:48:27 -08:00
zehweh d818ca9631 fix TestMisplacedChecking() and add test-case 2023-11-21 08:24:34 -08:00
julor 2fa37e0287 Update filer_server_handlers_write_autochunk.go 2023-11-20 11:17:14 -08:00
julor 9e03b916d8 verify content-md5 in header 
if upload content by s3 client and set content-md5 in header,then will verify recived and saved content,if not equal delete thos chunks
 2023-11-20 11:17:14 -08:00
SmsS4 6c1ce18541 Fix weed fix 2023-11-20 11:14:19 -08:00
Lars Lehtonen 57801687de weed/command: fix dropped error 2023-11-20 08:05:24 -08:00
Viktor Kuzmin 06e723f629 Waiting for writes to complete with acquired locks may lead to deadlock, fixes #4952 2023-11-17 07:38:14 -08:00
Konstantin Lebedev 2b229e98ce fix: add doDeleteFile option for filer backup 2023-11-17 07:37:28 -08:00
chrislu 17710e1ecb adjust logs 2023-11-15 18:11:35 -08:00
Amir H. Yeganemehr 6f559fe838 fs.mergeVolumes: change dir to a named parameter 2023-11-13 08:25:44 -08:00
Amir H. Yeganemehr d3556479be Fix typo (dist => dest) 2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr bb274ea8a3 fs.mergeVolumes: Make a plan based on volumes size 2023-11-13 06:41:24 -08:00
Konstantin Lebedev e55fa68848 fix: excludeFileName is empty 2023-11-13 06:23:46 -08:00
Konstantin Lebedev 3c5295a1a6 filer backup add option for exclude file names that match the regexp to sync on filer 2023-11-13 06:23:46 -08:00
Konstantin Lebedev 5e41ab1370 s3 api add not implemented response for PutBucketVersioning 2023-11-13 06:22:23 -08:00
Konstantin Lebedev 7f6152f130 s3 api add default response for GetBucketVersioning 2023-11-13 06:22:23 -08:00
Amir H. Yeganemehr b05d682c48 make help command in shell case insensitive 2023-11-13 06:06:45 -08:00
chrislu 27b34f3793 3.59 2023-11-12 22:12:08 -08:00
jsh 47112917ff ec.decode: mount the collected ec shards 2023-11-10 00:04:42 -08:00
chrislu 85bed4d92e Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-11-09 08:02:57 -08:00
chrislu c563386675 rename parameters 2023-11-09 08:02:39 -08:00
Lars Lehtonen 5db82e594a weed/server: fix dropped webdav error 2023-11-07 19:05:21 -08:00
zehweh 2b9dda7d2e
fix isMisplaced() in command_volume_fix_replication.go (#4988) 2023-11-07 07:58:19 -08:00
Konstantin Lebedev d693beea87
Replace an invalid method in a metric with an invalid one (#4987) 2023-11-06 10:51:47 -08:00
chrislu a667f0ad99 fix error reporting 
related to https://github.com/seaweedfs/seaweedfs/issues/4978
 2023-11-05 07:17:16 -08:00
chrislu b40ba7c7fe Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-11-01 16:28:05 -07:00
chrislu 310eec9a9c skip if path changed 
mitigate https://github.com/seaweedfs/seaweedfs/issues/4968
 2023-11-01 16:27:59 -07:00
Nico D'Cotta 3834ad5c48
refactor webdav subdirectory, fixes #4967 (#4969) 
* refactor webdav subdirectory, fixes #4967

* fix bug where Name() was not called on delegate wrappedFileInfo
 2023-11-01 13:12:49 -07:00
tired-engineer 4ed06e9ba5
Pass correct Filer's gRPC port to S3 server. (#4972) 
Fixes seaweedfs/seaweedfs#4971
 2023-11-01 10:11:41 -07:00
SmsS4 ba11331cc1
Fix twice increase metric counter (#4961) 2023-10-30 08:05:27 -07:00
SmsS4 f95848ba7d
Add ErrorGetNotFound and ErrorGetInternal to volume server metrics (#4960) 2023-10-30 07:38:03 -07:00
chrislu d1e83a3b4d 3.58 2023-10-29 23:04:20 -07:00
duanhongyi bf839651ec
Fix: issues #4950 (#4952) 2023-10-25 22:09:39 -07:00
zehweh 8b39bbbe2f
fix copying .vif file in VolumeCopy (#4943) 
closes #4934
fixes #2633
might fix #3528
 2023-10-24 09:54:24 -07:00
chrislu 2bc05f70e7 log full percentage 2023-10-22 12:59:34 -07:00
chrislu 612fd7206e fix 2023-10-19 08:10:33 -07:00
Konstantin Lebedev 6c82d7ddfc
[s3] do reload s3 static config (#4923) 
* do reload s3 config

* print error on reload s3 config

* print success msg

* Update weed/s3api/s3api_server.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-10-18 23:26:49 -07:00
lakeland1990 3df0e11ad3
remove compare to fix #4668 issue (#4924) 2023-10-18 09:06:56 -07:00
chrislu cbc24c7b24 disconnect from old subscribers 2023-10-13 16:48:14 -07:00
James Hedley 6acb43bbbb
Add optional flags to enable mTLS with verification of client certificate (#4910) 2023-10-13 09:02:24 -07:00
chrislu b62066d38f adding DiskType to /dir/status output 2023-10-12 21:39:11 -07:00
chrislu deb86ff4a6 upgrading grpc 2023-10-12 21:38:34 -07:00
chrislu de0b969b36 Revert "rename" 
This reverts commit 35b5264ab7.
 2023-10-12 20:28:11 -07:00
Konstantin Lebedev 1cac5d983d
fix: disallow file name too long when writing a file (#4881) 
* fix: disallow file name too long when writing a file

* bool LongerName to MaxFilenameLength

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 14:29:55 -07:00
Konstantin Lebedev edee91ef0e
fix has changes false if deleted result size eq zero (#4909) 
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 08:15:20 -07:00
chrislu 35b5264ab7 rename 2023-10-11 21:44:56 -07:00
Amir H. Yeganemehr 3e8604a4c5
add fs.mergeVolumes command into shell (#4907) 
* add fs.compact command into shell

* rename fs.compact to fs.mergeVolumes
 2023-10-10 12:47:18 -07:00
Konstantin Lebedev 2b3e39397e
fix: skipping checking active volumes with the same number of files at the moment (#4893) 
* fix: skipping checking active volumes with the same number of files at the moment
 https://github.com/seaweedfs/seaweedfs/issues/4140

* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140

* add TestShouldSkipVolume

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-09 09:57:26 -07:00
Yuval Yacoby 3fe00996b2
added healthz endpoint to filer (#4899) 2023-10-08 07:03:18 -07:00
Andrew Garrett 082f67bfe9
Fix filer.backup local sink to propagate file mode changes (#4896) 2023-10-06 05:40:20 -07:00
chrislu 0fd7222d65 default to skip if less than 4 nodes 2023-10-05 11:13:48 -07:00
chrislu 7b3c63a8af handle wrapped errors 2023-10-05 08:15:11 -07:00
chrislu 4e493d596c minor 2023-10-05 08:13:53 -07:00
Lars Lehtonen 530bdedf76
weed/operation: fix dropped error (#4888) 2023-10-03 10:55:20 -07:00
chrislu 6ebe26a765 Revert "Revert "Revert "Add disk type to prometheus metrics" (#4777)"" 
This reverts commit 567d788928.
 2023-10-03 08:28:52 -07:00
chrislu 7540d43ee9 Revert "Revert "fix compilation"" 
This reverts commit f9abfd0b03.
 2023-10-03 08:28:02 -07:00
chrislu 249c0e06ef Revert "fix compilation" 
This reverts commit 451ec6504d.
 2023-10-03 08:27:50 -07:00
chrislu 451ec6504d fix compilation 2023-10-03 08:15:18 -07:00
chrislu f9abfd0b03 Revert "fix compilation" 
This reverts commit 0483ba3889.
 2023-10-03 07:58:11 -07:00
chrislu 0483ba3889 fix compilation 2023-10-02 11:55:50 -07:00
chrislu 567d788928 Revert "Revert "Add disk type to prometheus metrics" (#4777)" 
This reverts commit 9215ba24be.
 2023-10-02 11:49:54 -07:00
chrislu b5070d6280 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-10-02 10:52:40 -07:00
Konstantin Lebedev dffe00a822
fix: logger place msg (#4880) 2023-10-02 08:29:09 -07:00
chrislu bf13f3ced7 add subscriber coordinator 2023-10-02 01:02:27 -07:00
chrislu 734178093e refactor TopicPartition struct 2023-10-02 01:01:45 -07:00
chrislu 2a578b9033 refactor 2023-10-01 11:59:19 -07:00
chrislu 321fda552c comment 2023-10-01 11:36:27 -07:00
chrislu b02fdeabff RetryForever => RetryUntil 2023-10-01 11:33:56 -07:00
chrislu ebd6f96d35 adjust mq.proto 2023-09-30 21:47:57 -07:00
chrislu 02cb799481 add a test framework for grpc 2023-09-30 21:36:58 -07:00
chrislu c4e6545197 fix compilation 2023-09-30 21:26:09 -07:00
chrislu 358cba43ef update proto generated files 2023-09-30 13:19:25 -07:00
chrislu 597ab1e4bf add design for mq balancer 2023-09-30 13:19:05 -07:00
chrislu d7c211fe8c adjust mq proto 2023-09-30 13:18:49 -07:00
chrislu 40888b5612 align 64bit atomic operation 
fix https://github.com/seaweedfs/seaweedfs/issues/4878
 2023-09-28 07:58:43 -07:00
Konstantin Lebedev dd580190b4
fix: avoid deleting one replica without sync (#4875) 
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647

* Update weed/shell/command_volume_fix_replication.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* fix: revert this existing do option to positive

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-27 23:12:10 -07:00
Konstantin Lebedev 44906f1f3b
fix: avoid error file name too long when writing a file (#4876) 2023-09-27 05:40:51 -07:00
chrislu 9d589b48e6 rename function 2023-09-26 15:17:33 -07:00
Konstantin Lebedev df4ded758e
fix: avoid deleting more than one replica (#4873) 
https://github.com/seaweedfs/seaweedfs/issues/4647

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-26 00:20:48 -07:00