gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
7204 commits 28 branches 278 tags 135 MiB
Commit graph

510 commits

Author SHA1 Message Date
chrislu 2f4cd20f90 tests skip reader pattern monitoring 2022-03-13 18:15:53 -07:00
chrislu 53513475bf mount: add back random read support 
avoid too much memory used also
 2022-03-13 01:38:52 -08:00
zzq09494 2ea18fca48 fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command' 2022-03-10 13:37:34 +08:00
banjiaojuhao f7f2a597dd minor 2022-03-08 16:22:55 +08:00
banjiaojuhao d61bea9038 [bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized). 2022-03-08 16:22:55 +08:00
chrislu 6f3ec989a7 fix manifest batch size 2022-03-07 00:07:53 -08:00
banjiaojuhao a07c93f4dd filer: reset bytesBuffer before use 2022-03-06 21:07:36 +08:00
chrislu da76af187f mount: avoid possible index out of bounds error 2022-03-04 22:36:01 -08:00
chrislu 6fbbc78574 stream reading a whole chunk 2022-03-02 13:50:46 -08:00
chrislu 941ced60a4 download 2 chunks if at the beginning of a file 2022-02-27 03:57:24 -08:00
chrislu 551d00d51a prefetch other chunks when stream reading 2022-02-26 23:20:45 -08:00
chrislu 7b1a713d2a remove dead code 2022-02-26 03:23:15 -08:00
chrislu 3a58b7bac6 a little safer 2022-02-26 03:23:06 -08:00
chrislu 3345a50d9b prefetch 2 chunks 2022-02-26 03:06:17 -08:00
chrislu 86ce69f709 remove logs 2022-02-26 03:00:20 -08:00
chrislu 28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu 72c0233938 less logs 2022-02-25 15:34:24 -08:00
chrislu 03466f955e rename: delete source entry metadata only, skipping hard links 2022-02-25 02:57:54 -08:00
chrislu 8080fe4cc1 logs 2022-02-25 02:56:23 -08:00
chrislu 202a29d014 refactoring 2022-02-25 01:17:26 -08:00
chrislu be3fc77391 mount2: use consistent inode 2022-02-25 00:53:27 -08:00
chrislu ceaf993a27 mount2: add rdev 2022-02-24 14:51:25 -08:00
chrislu b05962b90e rename: handle hard links 2022-02-24 02:59:00 -08:00
chrislu 56da3494cb leveldb3: adjust memory allocation for each bucket 2022-02-20 13:40:13 -08:00
chrislu 7bc67399e4 listing for filer stores without prefixed query: break if no more progress 2022-02-19 10:12:51 -08:00
chrislu 248c0c8087 minor 2022-02-19 09:54:09 -08:00
chrislu 4ee0a6f47b filer store: reduce one possible listing operation 2022-02-19 00:43:42 -08:00
chrislu b3594278c9 optimize a bit 2022-02-18 23:36:10 -08:00
chrislu df51e0c042 mongodb: remove ErrNoDocuments checking since it is only for creating single record, not for querying 2022-02-15 16:18:36 -08:00
chrislu 98cce6a150 mongodb: ensure closing cursor 2022-02-15 16:15:44 -08:00
banjiaojuhao b5ec346700 FilerStore: add redis_lua 2022-02-15 20:54:57 +08:00
Eng Zer Jun b92df1654c
test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-02-14 10:38:13 +08:00
chrislu 85c1615b43 filer read empty file may cause OOM in some cases 
fix https://github.com/chrislusf/seaweedfs/issues/2641
 2022-02-07 23:08:54 -08:00
chrislu 9405eaefdb filer.sync: fix replicating partially updated file 
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
 2022-02-07 03:46:28 -08:00
chrislu 4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu 606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
guol-fnst da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
banjiaojuhao 45e9c83421 padding zero for sparse file 2022-01-13 22:21:22 +08:00
chrislu 1453263b63 remove dead code 2022-01-13 02:02:04 -08:00
chrislu 10ecf80ca1 add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
chrislu 34742be029 remove duplicated metadata subscription in filer 
https://github.com/chrislusf/seaweedfs/issues/2545
 2021-12-30 01:51:52 -08:00
chrislu 9a00c17555 reader: avoid wrong pattern detection due to lock waiting 2021-12-28 16:30:33 -08:00
chenkai 47c30e3add filer list entries use context to break job 2021-12-28 15:03:41 +08:00
chrislu 9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
banjiaojuhao 083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
chrislu b541e39a2c fix tests 2021-12-22 16:17:30 -08:00
chrislu 0ec7bc6710 detect non streaming mode on the first read 2021-12-22 16:05:38 -08:00
chrislu 4c1368d621 fix test 2021-12-22 16:05:08 -08:00
chrislu 0cb9036f66 mount: only cache the first chunk on stream read 2021-12-19 23:06:03 -08:00
chrislu a152f17937 mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00