gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6734 commits 28 branches 278 tags 135 MiB
Commit graph

4432 commits

Author SHA1 Message Date
Chris Lu 7348d2ae63 randomize traffic if replicated 2021-11-05 00:03:13 -07:00
Chris Lu 00ae965d8d randomize a bit for ec shards distribution 2021-11-04 09:23:40 -07:00
Chris Lu 77f90ae288 add leader election in master 2021-11-04 00:54:38 -07:00
Chris Lu 35c37562bc check auth only when enabled 2021-11-03 12:11:36 -07:00
Chris Lu 3c245c69d3 handle possible race condition 2021-11-03 01:36:39 -07:00
Chris Lu af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu ab97b17e62 better printout 2021-11-02 23:45:47 -07:00
Chris Lu e66865a8c5 adjust master client log level 2021-11-02 23:45:28 -07:00
Chris Lu 0c8dea9de8 go fmt 2021-11-02 23:39:16 -07:00
Chris Lu 5160eb08f7 shell: optionally read filer address from master 2021-11-02 23:38:45 -07:00
Chris Lu 18bfbf62fc add one unit test 2021-11-02 23:36:43 -07:00
Chris Lu e1ab8b01d0 s3: adjust permission for HEAD bucket operation 
fix https://github.com/chrislusf/seaweedfs/issues/2417#issuecomment-958391856
 2021-11-02 16:28:55 -07:00
Chris Lu b25661c6df s3: skip permission checking for creating bucket if the bucket already exists 
fix https://github.com/chrislusf/seaweedfs/issues/2417

Rclone was trying to create the bucket even though the bucket already exists.
 2021-11-02 13:13:36 -07:00
Chris Lu 1b90d60765 filer: paginate large directories to delete file chunks 2021-11-02 01:04:50 -07:00
Chris Lu 794375ca0a adjust help message since both fullPercent and quietFor are needed. 2021-11-01 17:22:47 -07:00
Chris Lu fc9e246592 2.76 2021-10-31 18:08:28 -07:00
Chris Lu 9cf756b188 S3: support CORS 
related to https://github.com/chrislusf/seaweedfs/issues/1271
 2021-10-31 18:06:43 -07:00
Chris Lu f17fa400d5 refactoring 2021-10-31 18:05:34 -07:00
Chris Lu 006b4ec0f8 refactoring 2021-10-31 18:02:08 -07:00
Chris Lu edbf6d297b filer.meta.tail: add example to send metadata to elastic search 2021-10-31 18:01:33 -07:00
Chris Lu 24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu f5824f369c avoid reporting error when it is a user request error 2021-10-29 14:14:29 -07:00
Chris Lu c857cc7286 cloud tier: remove tagging since not all s3 vendors support this 2021-10-29 12:39:19 -07:00
Chris Lu d04cdcf40d s3 header add user agent 2021-10-29 12:28:24 -07:00
Konstantin Lebedev dc3ae5e8a6 DeleteBucketLifecycleConfiguration return 204 2021-10-28 19:34:37 +05:00
Konstantin Lebedev 2afb5a13af fix DeleteBucketLifecycleConfiguration 2021-10-28 18:30:33 +05:00
Chris Lu 2e76834e4d filer store: redis2 fix wrong pagination 2021-10-27 23:46:25 -07:00
Chris Lu 54b6e0f3fd adjust logs 2021-10-27 23:46:07 -07:00
Chris Lu 900e888695 filer: avoid possible prefixed pagination loop on unsupported filer stores 2021-10-27 23:45:48 -07:00
Chris Lu d774fa6c9a rename variable 2021-10-25 14:39:20 -07:00
Chris Lu 2539ba0b62 fix compilation 2021-10-25 14:38:48 -07:00
Chris Lu 5f2d7c1589 erasure coding: skip erasure coding if less than recommended 4 nodes 2021-10-25 14:38:11 -07:00
Chris Lu c9d3fb4a30 2.75 2021-10-24 18:15:59 -07:00
Chris Lu 5435027ff0 volume copy: stream out copying progress and avoid grpc request timeout 
fix https://github.com/chrislusf/seaweedfs/issues/2386
 2021-10-24 02:52:56 -07:00
Chris Lu 3be3c17f59 volume vacuum: avoid timeout with streaming progress report 
fix https://github.com/chrislusf/seaweedfs/issues/2396
 2021-10-24 01:55:34 -07:00
Konstantin Lebedev 235329a92a fix Tagging add xmlns 2021-10-20 19:40:22 +05:00
Konstantin Lebedev dc7e525cb9 fix Tagging test 2021-10-20 19:12:00 +05:00
Konstantin Lebedev 0b834600d5 fix PutObjectTaggingHandler Unmarshal 2021-10-20 17:58:06 +05:00
Konstantin Lebedev c78220a7f2 fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389 2021-10-20 16:01:06 +05:00
Chris Lu 182f43ae5f 2.74 2021-10-18 14:23:54 -07:00
Chris Lu 309f46e3af S3: fix upload limit if the size is more than 8GB 2021-10-18 14:14:18 -07:00
Chris Lu cd4fa7561b 2.73 2021-10-18 10:47:48 -07:00
Chris Lu a6bb509cb7 adjust mime detection 2021-10-18 10:47:39 -07:00
Konstantin Lebedev dd2cc1acaf fix https://github.com/chrislusf/seaweedfs/issues/2387 2021-10-18 16:27:57 +05:00
Chris Lu 97c963bac9 2.72 2021-10-17 17:40:27 -07:00
Chris Lu 3833dac3f7 continue to read from memory if there is no flush 2021-10-17 13:53:04 -07:00
Chris Lu 29fa1b9fdc add logs 2021-10-17 13:50:34 -07:00
Chris Lu cca62fdb30 mount: streaming renaming folders 2021-10-17 04:22:42 -07:00
Chris Lu 004e56c1a6 save updated lastTsNs 2021-10-17 02:01:20 -07:00
Chris Lu 71a8c80f6f s3: continuously listen to file updates 
fix https://github.com/chrislusf/seaweedfs/issues/2342
 2021-10-17 01:04:19 -07:00
Chris Lu 93bb7869b8 Revert "mount: fix renaming a deep directory with unvisited directories" 
This reverts commit 0ccdb937bb.
 2021-10-16 23:52:30 -07:00
Chris Lu 8e2c9713a3 turn on new faster algorithm to translate into visible chunks 2021-10-16 23:35:20 -07:00
Chris Lu e7d1f183cc ensure file id is populated 2021-10-16 23:34:27 -07:00
Chris Lu fc1694b321 add tests 2021-10-16 23:34:12 -07:00
Chris Lu 0ccdb937bb mount: fix renaming a deep directory with unvisited directories 2021-10-16 23:33:45 -07:00
Chris Lu b9a2efd69b temporarily reverting 2021-10-16 16:29:10 -07:00
Chris Lu 8a124ef9fc Revert "remove deprecated code" 
This reverts commit de7688c539.
 2021-10-16 16:09:33 -07:00
Chris Lu 93624635b4 Revert "go fmt" 
This reverts commit 6c704eb2ba.
 2021-10-16 16:09:30 -07:00
Chris Lu 6c704eb2ba go fmt 2021-10-16 16:06:18 -07:00
Chris Lu de7688c539 remove deprecated code 2021-10-16 16:06:12 -07:00
Chris Lu 7336990639 faster file read for large files 2021-10-16 16:03:16 -07:00
Chris Lu 8965a53c4d add warning error 2021-10-16 15:57:30 -07:00
Chris Lu 8e3075d051 ensure file entry chunks have populated the file id 2021-10-16 01:05:48 -07:00
Chris Lu 822ee2475b remove unnecessary file 2021-10-16 01:05:06 -07:00
Konstantin Lebedev 8de7915950 s3 mime detect 2021-10-14 15:03:11 +05:00
Chris Lu 5fd4b05c5e
Merge pull request #2381 from Juneezee/deprecate-ioutil 
refactor: move from io/ioutil to io and os package
 2021-10-13 22:38:58 -07:00
Chris Lu 46a09c6074 adjust test 2021-10-13 22:38:47 -07:00
Eng Zer Jun a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu 4cbd390fbe test: add fail message 2021-10-13 20:42:20 -07:00
Chris Lu b693a8d6b9
Merge pull request #2380 from kmlebedev/bucket_policy 
Bucket policy
 2021-10-13 15:29:53 -07:00
Konstantin Lebedev 9d6ffa0ea1 GetBucketLifecycleConfigurationHandler 2021-10-14 01:35:33 +05:00
Konstantin Lebedev 458145425e S3 MultipartUpload pass contentType to meta 2021-10-12 17:14:54 +05:00
Chris Lu de1ec050e8 fier/s3: save "Content-Disposition" to extended properties 
related to https://github.com/chrislusf/seaweedfs/issues/2371
 2021-10-12 00:04:28 -07:00
Chris Lu 34f764007f escape file name only when necessary 2021-10-11 23:33:45 -07:00
Chris Lu 8e9273db99 s3: use "response-content-disposition" to overwrite default content-disposition 
fix one part of https://github.com/chrislusf/seaweedfs/issues/2371

see https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
 2021-10-11 23:29:52 -07:00
Chris Lu f0b928ff5e go fmt 2021-10-11 23:23:46 -07:00
Chris Lu 3afa451cdc volume: find a non-empty offset when binary searching by timestamp 2021-10-11 22:00:41 -07:00
Chris Lu b530f12327 volume: find a non-empty offset when binary searching by timestamp 
fix https://github.com/chrislusf/seaweedfs/issues/2364
 2021-10-11 16:11:50 -07:00
Konstantin Lebedev be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu e6ef7b2387 return error early 
fix https://github.com/chrislusf/seaweedfs/issues/2370
 2021-10-11 01:24:30 -07:00
Chris Lu 3d586be552 2.71 2021-10-10 22:40:44 -07:00
Chris Lu f4676824a7 fix test code compilation 2021-10-10 22:14:13 -07:00
Chris Lu 8205166009 disable testing with redis server 
--- FAIL: TestNameList (0.00s)
panic: exec: "redis-server": executable file not found in $PATH [recovered]
	panic: exec: "redis-server": executable file not found in $PATH

goroutine 37 [running]:
testing.tRunner.func1.2({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1209 +0x24e
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1212 +0x218
panic({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/panic.go:1038 +0x215
github.com/chrislusf/seaweedfs/weed/filer/redis3.TestNameList(0x407c59)
	/home/runner/work/seaweedfs/seaweedfs/weed/filer/redis3/kv_directory_children_test.go:45 +0x376
testing.tRunner(0xc0003e2680, 0xf57f48)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x35a
 2021-10-10 21:57:55 -07:00
Chris Lu 3e2acf677c removing tikv to resolve "go mod tidy" problem 
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back.

go mod tidy
go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker
go: finding module for package cloud.google.com/go/kms/apiv1
go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/coreos/etcd/clientv3/credentials
go: finding module for package github.com/coreos/etcd/clientv3/balancer
go: finding module for package github.com/d4l3k/messagediff
go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible
go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0
go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1
go: finding module for package google.golang.org/grpc/naming
github.com/chrislusf/seaweedfs/weed/filer/tikv imports
	github.com/tikv/client-go/v2/tikv imports
	go.etcd.io/etcd/clientv3 tested by
	go.etcd.io/etcd/clientv3.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
 2021-10-10 19:27:02 -07:00
Chris Lu bf218cd59d removing etcd sequencer 
causing go mod tidy problem. If anyone wants this, please help to resolve this first.

github.com/chrislusf/seaweedfs/weed/sequence imports
	go.etcd.io/etcd/client tested by
	go.etcd.io/etcd/client.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
 2021-10-10 19:10:46 -07:00
Chris Lu 17e08a16f4 fix redis3 deletion 2021-10-09 04:54:48 -07:00
Chris Lu ce1efeb0eb adds more error message 2021-10-09 04:54:14 -07:00
Chris Lu e4830bd93d go fmt 2021-10-07 21:13:31 -07:00
Chris Lu 0a856241fe avoid int bigger than math.MaxInt32 
fix https://github.com/chrislusf/seaweedfs/issues/2363
 2021-10-07 21:12:57 -07:00
Chris Lu d688e10ed1 do not try to compress if not sure about the file content 2021-10-07 13:29:00 -07:00
Chris Lu 332f5ad3a8 revert temporary changes that disabled compression 
fix https://github.com/chrislusf/seaweedfs/issues/2362
 2021-10-07 13:24:16 -07:00
Chris Lu f3d8232e14 reduce one redis lookup on hot path 2021-10-06 22:01:19 -07:00
Chris Lu 2336a397dc use pipeline to save some time 2021-10-06 20:40:18 -07:00
Chris Lu 371fead8a5 redis3 using redis native sorted set 2021-10-06 18:18:24 -07:00
Chris Lu 8668d49c9d test with real redis 
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkNameList-12    	    1789	    760599 ns/op
BenchmarkRedis-12       	   17539	     64122 ns/op
PASS
 2021-10-06 01:25:37 -07:00
Chris Lu 6b31f3c97a add benchmark test 2021-10-06 00:37:57 -07:00
Chris Lu 893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu f0d1e7bd05 skip ec volumes when loading normal volumes 2021-10-05 02:31:44 -07:00
Chris Lu 332d49432d reduce concurrent volume grow requests 2021-10-05 01:58:30 -07:00
Chris Lu 96119eab00 refactor 2021-10-05 00:40:04 -07:00
Chris Lu 8a66306064 calculate disk usage in case of race condition 
related to https://github.com/chrislusf/seaweedfs/issues/2357
 2021-10-04 23:32:07 -07:00
Chris Lu 4ed2994555 use tsMemory to determine whether read from disk or memory 
remove lastFlushTime
 2021-10-04 16:02:56 -07:00
Chris Lu 6a030547a2 server: remove peer check if not starting master 
more fix https://github.com/chrislusf/seaweedfs/issues/2352
 2021-10-04 03:27:10 -07:00
Chris Lu 62c2732fd1 add back logic to check master peers 
fix https://github.com/chrislusf/seaweedfs/issues/2352
 2021-10-04 02:51:26 -07:00
Chris Lu d4bb16e20e
Merge pull request #2354 from chrislusf/bptree 
Add Redis3
 2021-10-04 02:43:15 -07:00
Chris Lu 947add39e6 clean up *SkipListElementReference loaded from Redis 2021-10-04 02:31:38 -07:00
Chris Lu 513fed323a SkipListElementReference can be an empty object 2021-10-04 02:30:44 -07:00
Chris Lu 280ab7f95c add test 2021-10-04 02:30:24 -07:00
Chris Lu 04662126bb add redis3 2021-10-04 01:04:27 -07:00
Chris Lu 2b9aab3442 use 1000 per batch 2021-10-04 01:03:40 -07:00
Chris Lu 366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu ba7fbac07f rename 2021-10-03 19:23:34 -07:00
Chris Lu e6196cdc50 add name list 2021-10-03 17:54:25 -07:00
Chris Lu a481c4a45e return previous element if visited 2021-10-03 13:50:52 -07:00
Chris Lu 22d8684e88 refactor out listStore 2021-10-03 02:19:21 -07:00
Chris Lu d343b0db57 update value 2021-10-03 01:15:14 -07:00
Chris Lu 4f50f8c2ca insert key and value 2021-10-03 01:07:35 -07:00
Chris Lu 69b84bb771 TestFindGreaterOrEqual 2021-10-02 14:15:49 -07:00
Chris Lu 57e2fd3f9b remove bptree 2021-10-02 14:03:54 -07:00
Chris Lu 4c1741fdbb working skiplist 2021-10-02 14:02:56 -07:00
Chris Lu b297849147 typo 2021-10-01 23:24:54 -07:00
Chris Lu af207bbaf0 retry both assign volume and uploading data 
fix https://github.com/chrislusf/seaweedfs/issues/2351
 2021-10-01 23:23:39 -07:00
Chris Lu b6694279d7 Merge branch 'master' into bptree 2021-10-01 16:55:44 -07:00
Chris Lu 1e3fdf366f go fmt 2021-10-01 12:10:24 -07:00
Chris Lu e862b2529a refactor 2021-10-01 12:10:11 -07:00
Chris Lu 88e006bc6d
Merge pull request #2349 from kmlebedev/fix_repl_volumes_per_step 
Topology update for every Nth id volume
 2021-10-01 12:05:51 -07:00
Konstantin Lebedev 5e64b22b45 check that the topology has been updated 2021-10-01 18:51:22 +05:00
Chris Lu f58ea6a2ee add source name to error message 2021-10-01 02:19:30 -07:00
chrislusf 3ffbaaa071 use github.com/linxGnu/grocksdb 2021-10-01 03:16:03 +00:00
Konstantin Lebedev 2cecde89c3 rename opt volumesPerStep 2021-10-01 00:17:54 +05:00
Konstantin Lebedev fc51ffce2b https://github.com/chrislusf/seaweedfs/issues/1846 2021-09-30 20:24:24 +05:00
Chris Lu a067deaabc avoid possible modified location list 
fix issue 1 of https://github.com/chrislusf/seaweedfs/issues/2345
 2021-09-28 16:54:18 -07:00
Chris Lu 040443e2d1 fix possible error case 2021-09-27 23:59:45 -07:00
Chris Lu 2e9372dcf7 volume stream read skips deleted content 2021-09-27 03:07:44 -07:00
Chris Lu 225b019fe0 stream read multiple volumes in a volume server 2021-09-27 02:51:31 -07:00
Chris Lu 1904448d4e adjust starting offset 2021-09-27 02:01:29 -07:00
Chris Lu 5956a8b05a adjust comment 2021-09-27 01:58:03 -07:00
Chris Lu aa64f2ac4c send needle data instead of raw needle body 2021-09-27 01:53:41 -07:00
Chris Lu c4d7ee6c5c volume server: read all files in a volume 2021-09-27 01:45:32 -07:00
Chris Lu 4a1d4d7462 s3: default to allow empty folder for better performance 2021-09-26 22:34:14 -07:00
Chris Lu cee4d20bc1 2.70 2021-09-26 17:37:46 -07:00
Chris Lu 603ea2db73 avoid looping forever if there are no more metadata updates 2021-09-26 11:55:27 -07:00
Chris Lu 9887610b54 log tsNs should be processing time 2021-09-26 11:54:13 -07:00
Chris Lu 2baed2e1e9 avoid possible metadata subscription data loss 
Previous implementation append filer logs into one file. So one file is not always sorted, which can lead to miss reading some entries, especially when different filers have different write throughput.
 2021-09-25 01:18:44 -07:00
Chris Lu a814f3f0a8 adjust metadata tail output 2021-09-25 01:04:51 -07:00
Chris Lu 52fe86df45 use default 10000 for grpc port 2021-09-20 14:05:59 -07:00
Chris Lu b3d88180ca Merge branch 'master' into bptree 2021-09-19 23:56:59 -07:00
Chris Lu fa7c65bd4b 2.69 2021-09-19 21:44:06 -07:00
Chris Lu 84fdda85e7 go fmt 2021-09-19 12:06:15 -07:00
Chris Lu ad5099e570 refactor 2021-09-19 12:02:23 -07:00