seaweedfs/weed/filer
chrislu 9405eaefdb filer.sync: fix replicating partially updated file
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
2022-02-07 03:46:28 -08:00
..
abstract_sql filer: fix mysql, postgres batch delete error 2021-07-22 08:23:20 -07:00
cassandra add gocql timeout setting 2022-01-18 15:21:13 +08:00
elastic/v7 filer: fix mysql, postgres batch delete error 2021-07-22 08:23:20 -07:00
etcd go mod 2021-09-03 23:25:33 -07:00
hbase filer: fix mysql, postgres batch delete error 2021-07-22 08:23:20 -07:00
leveldb add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
leveldb2 refactor: move from io/ioutil to io and os package 2021-10-14 12:27:58 +08:00
leveldb3 refactor: move from io/ioutil to io and os package 2021-10-14 12:27:58 +08:00
mongodb fix MongodbStore.ListDirectoryEntries panic on find failure 2021-12-03 13:58:37 +03:00
mysql Fix mysql sql for batch delete 2021-07-25 11:06:14 +09:00
mysql2 go fmt 2021-04-02 02:22:26 -07:00
postgres go fmt 2021-04-02 02:22:26 -07:00
postgres2 go fmt 2021-04-02 02:22:26 -07:00
redis filer: redis store needs to clean up batch deleted sub folders 2021-09-18 14:05:16 -07:00
redis2 support redis sentinel 2021-11-25 15:57:03 +08:00
redis3 redis3 supports sentinel 2021-11-29 01:09:51 -08:00
rocksdb refactor: move from io/ioutil to io and os package 2021-10-14 12:27:58 +08:00
sqlite go fmt 2021-09-01 02:45:42 -07:00
configuration.go Change DeleteFolderChildren to DeleteRange api 2021-08-26 17:49:56 +08:00
entry.go able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
entry_codec.go able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
filechunk_manifest.go mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
filechunk_manifest_test.go rename filer2 to filer 2020-09-01 00:21:19 -07:00
filechunks.go filer.sync: fix replicating partially updated file 2022-02-07 03:46:28 -08:00
filechunks2_test.go rename filer2 to filer 2020-09-01 00:21:19 -07:00
filechunks_read.go go fmt 2021-11-02 23:39:16 -07:00
filechunks_read_test.go go fmt 2021-11-02 23:39:16 -07:00
filechunks_test.go optimization: improve random range query for large files 2021-07-19 23:07:22 -07:00
filer.go filer list entries use context to break job 2021-12-28 15:03:41 +08:00
filer_buckets.go s3: skip hidden directories in /buckets folder 2021-08-01 12:28:08 -07:00
filer_conf.go use streaming mode for long poll grpc calls 2021-12-26 00:15:03 -08:00
filer_conf_test.go filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
filer_delete_entry.go use streaming mode for long poll grpc calls 2021-12-26 00:15:03 -08:00
filer_deletion.go skip the rest logic 2021-11-29 00:32:21 -08:00
filer_notify.go avoid possible metadata subscription data loss 2021-09-25 01:18:44 -07:00
filer_notify_append.go fix possible error case 2021-09-27 23:59:45 -07:00
filer_notify_test.go rename filer2 to filer 2020-09-01 00:21:19 -07:00
filer_on_meta_event.go Merge branch 'master' into add_remote_storage 2021-08-01 15:44:21 -07:00
filer_rename.go check cross device rename error 2020-12-10 23:50:32 -08:00
filer_search.go avoid int bigger than math.MaxInt32 2021-10-07 21:12:57 -07:00
filerstore.go add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
filerstore_hardlink.go refactor 2020-12-21 21:46:32 -08:00
filerstore_translate_path.go avoid int bigger than math.MaxInt32 2021-10-07 21:12:57 -07:00
filerstore_wrapper.go add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
meta_aggregator.go dedup local metadata subscribers 2022-01-23 16:14:22 -08:00
meta_replay.go rename filer2 to filer 2020-09-01 00:21:19 -07:00
read_remote.go use streaming mode for long poll grpc calls 2021-12-26 00:15:03 -08:00
read_write.go s3 config read via grpc 2021-07-09 02:48:03 -07:00
reader_at.go padding zero for sparse file 2022-01-13 22:21:22 +08:00
reader_at_test.go fix test 2021-12-22 16:05:08 -08:00
reader_pattern.go detect non streaming mode on the first read 2021-12-22 16:05:38 -08:00
remote_mapping.go use streaming mode for long poll grpc calls 2021-12-26 00:15:03 -08:00
remote_storage.go use streaming mode for long poll grpc calls 2021-12-26 00:15:03 -08:00
remote_storage_test.go rename 2021-09-04 05:03:51 -07:00
s3iam_conf.go rename 2021-07-09 03:19:21 -07:00
s3iam_conf_test.go rename 2021-07-09 03:19:21 -07:00
stream.go padding zero for sparse file 2022-01-13 22:21:22 +08:00
topics.go rename filer2 to filer 2020-09-01 00:21:19 -07:00