gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6939 commits 28 branches 278 tags 135 MiB
Commit graph

477 commits

Author SHA1 Message Date
chrislu 9405eaefdb filer.sync: fix replicating partially updated file 
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
 2022-02-07 03:46:28 -08:00
chrislu 4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu 606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
guol-fnst da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
banjiaojuhao 45e9c83421 padding zero for sparse file 2022-01-13 22:21:22 +08:00
chrislu 1453263b63 remove dead code 2022-01-13 02:02:04 -08:00
chrislu 10ecf80ca1 add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
chrislu 34742be029 remove duplicated metadata subscription in filer 
https://github.com/chrislusf/seaweedfs/issues/2545
 2021-12-30 01:51:52 -08:00
chrislu 9a00c17555 reader: avoid wrong pattern detection due to lock waiting 2021-12-28 16:30:33 -08:00
chenkai 47c30e3add filer list entries use context to break job 2021-12-28 15:03:41 +08:00
chrislu 9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
banjiaojuhao 083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
chrislu b541e39a2c fix tests 2021-12-22 16:17:30 -08:00
chrislu 0ec7bc6710 detect non streaming mode on the first read 2021-12-22 16:05:38 -08:00
chrislu 4c1368d621 fix test 2021-12-22 16:05:08 -08:00
chrislu 0cb9036f66 mount: only cache the first chunk on stream read 2021-12-19 23:06:03 -08:00
chrislu a152f17937 mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
Bl1tz23 83bffca7e6 fix MongodbStore.ListDirectoryEntries panic on find failure 2021-12-03 13:58:37 +03:00
Chris Lu 689f5513a9 redis3 supports sentinel 2021-11-29 01:09:51 -08:00
Chris Lu 2f72c24498 skip the rest logic 2021-11-29 00:32:21 -08:00
Chris Lu 96c66ca2aa read deleted chunks when replicating data 2021-11-28 23:33:03 -08:00
limd ec03f22cc3 Merge remote-tracking branch 'origin/master' 2021-11-25 16:07:14 +08:00
limd 220797bd71 support redis sentinel 2021-11-25 15:57:03 +08:00
Chris Lu 7bf891c00a randomize same-dc servers and other-dc servers 2021-11-12 11:30:11 -08:00
Chris Lu 4729a57cc0 use constants 2021-11-08 17:47:56 -08:00
Chris Lu e0fc2898e9 auto updated filer peer list 2021-11-06 14:23:35 -07:00
Chris Lu 5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu 7348d2ae63 randomize traffic if replicated 2021-11-05 00:03:13 -07:00
Chris Lu af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu 0c8dea9de8 go fmt 2021-11-02 23:39:16 -07:00
Chris Lu 1b90d60765 filer: paginate large directories to delete file chunks 2021-11-02 01:04:50 -07:00
Chris Lu 24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu f5824f369c avoid reporting error when it is a user request error 2021-10-29 14:14:29 -07:00
Chris Lu 2e76834e4d filer store: redis2 fix wrong pagination 2021-10-27 23:46:25 -07:00
Chris Lu 900e888695 filer: avoid possible prefixed pagination loop on unsupported filer stores 2021-10-27 23:45:48 -07:00
Chris Lu 8e2c9713a3 turn on new faster algorithm to translate into visible chunks 2021-10-16 23:35:20 -07:00
Chris Lu e7d1f183cc ensure file id is populated 2021-10-16 23:34:27 -07:00
Chris Lu fc1694b321 add tests 2021-10-16 23:34:12 -07:00
Chris Lu b9a2efd69b temporarily reverting 2021-10-16 16:29:10 -07:00
Chris Lu 8a124ef9fc Revert "remove deprecated code" 
This reverts commit de7688c539.
 2021-10-16 16:09:33 -07:00
Chris Lu 93624635b4 Revert "go fmt" 
This reverts commit 6c704eb2ba.
 2021-10-16 16:09:30 -07:00
Chris Lu 6c704eb2ba go fmt 2021-10-16 16:06:18 -07:00
Chris Lu de7688c539 remove deprecated code 2021-10-16 16:06:12 -07:00
Chris Lu 7336990639 faster file read for large files 2021-10-16 16:03:16 -07:00
Chris Lu 8e3075d051 ensure file entry chunks have populated the file id 2021-10-16 01:05:48 -07:00
Eng Zer Jun a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Konstantin Lebedev 9d6ffa0ea1 GetBucketLifecycleConfigurationHandler 2021-10-14 01:35:33 +05:00
Konstantin Lebedev be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu f4676824a7 fix test code compilation 2021-10-10 22:14:13 -07:00
Chris Lu 8205166009 disable testing with redis server 
--- FAIL: TestNameList (0.00s)
panic: exec: "redis-server": executable file not found in $PATH [recovered]
	panic: exec: "redis-server": executable file not found in $PATH

goroutine 37 [running]:
testing.tRunner.func1.2({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1209 +0x24e
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1212 +0x218
panic({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/panic.go:1038 +0x215
github.com/chrislusf/seaweedfs/weed/filer/redis3.TestNameList(0x407c59)
	/home/runner/work/seaweedfs/seaweedfs/weed/filer/redis3/kv_directory_children_test.go:45 +0x376
testing.tRunner(0xc0003e2680, 0xf57f48)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x35a
 2021-10-10 21:57:55 -07:00