gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
7342 commits 28 branches 278 tags 135 MiB
Commit graph

521 commits

Author SHA1 Message Date
Chris Lu 371fead8a5 redis3 using redis native sorted set 2021-10-06 18:18:24 -07:00
Chris Lu 8668d49c9d test with real redis 
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkNameList-12    	    1789	    760599 ns/op
BenchmarkRedis-12       	   17539	     64122 ns/op
PASS
 2021-10-06 01:25:37 -07:00
Chris Lu 6b31f3c97a add benchmark test 2021-10-06 00:37:57 -07:00
Chris Lu 893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu 947add39e6 clean up *SkipListElementReference loaded from Redis 2021-10-04 02:31:38 -07:00
Chris Lu 2b9aab3442 use 1000 per batch 2021-10-04 01:03:40 -07:00
Chris Lu 366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu b6694279d7 Merge branch 'master' into bptree 2021-10-01 16:55:44 -07:00
chrislusf 3ffbaaa071 use github.com/linxGnu/grocksdb 2021-10-01 03:16:03 +00:00
Chris Lu 040443e2d1 fix possible error case 2021-09-27 23:59:45 -07:00
Chris Lu 9887610b54 log tsNs should be processing time 2021-09-26 11:54:13 -07:00
Chris Lu 2baed2e1e9 avoid possible metadata subscription data loss 
Previous implementation append filer logs into one file. So one file is not always sorted, which can lead to miss reading some entries, especially when different filers have different write throughput.
 2021-09-25 01:18:44 -07:00
Chris Lu 198fa58e3c Merge branch 'master' into bptree 2021-09-18 14:06:16 -07:00
Chris Lu 49d971e602 filer: redis store needs to clean up batch deleted sub folders 2021-09-18 14:05:16 -07:00
Chris Lu 8f2e4be074 wip 2021-09-18 14:04:30 -07:00
Chris Lu 96514f0f00 fix tests 2021-09-13 02:19:48 -07:00
Chris Lu e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
joshuafc 5654d0d60d CompactionTableSizeMultiplier of leveldb use default value. #2325 
To improve performance of leveldb find key in condition of large directory(millions of files) which use uuid as filename.
 2021-09-09 10:42:34 +08:00
Chris Lu 574485ec69 better IP v6 support 2021-09-07 19:29:42 -07:00
zhoub 6a7ed1bd0e add bloom filter to leveldb_store to improve fuse performance. 2021-09-07 21:09:10 +08:00
Chris Lu 6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Chris Lu 28f45f8fa6 re-org 2021-09-05 14:38:31 -07:00
Chris Lu df29281536 rename 2021-09-04 05:11:06 -07:00
Chris Lu c3db389e42 refactor 2021-09-04 05:09:38 -07:00
Chris Lu e9ce099453 rename 2021-09-04 05:03:51 -07:00
Chris Lu 8707ef00a6 refactor 2021-09-04 05:02:20 -07:00
Chris Lu 8ec357b3d3 go mod 2021-09-03 23:25:33 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
yulai.li b17b81529e Add build tags for TiKV filerstore 2021-08-30 15:59:25 +08:00
Rain Li e5f9ff983d
Merge branch 'master' into filerstore-tikv 2021-08-30 15:52:54 +08:00
Chris Lu dc481c081c remove unused function 2021-08-27 22:18:54 -07:00
Chris Lu 9242f3aaeb fix build 2021-08-26 17:33:57 -07:00
Chris Lu 05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
yulai.li c1dc5ab4ac Add deleterange_concurrency to filer configuration file 2021-08-26 18:25:08 +08:00
yulai.li 318757ef8c Change DeleteFolderChildren to DeleteRange api 2021-08-26 17:49:56 +08:00
yulai.li 546efeba8f Fix build bug 2021-08-26 16:20:35 +08:00
yulai.li 2088f28424 init post 2021-08-26 15:20:18 +08:00
Chris Lu 9bcf94b2b1 ensure multi-threaded correctness 2021-08-25 17:28:50 -07:00
Chris Lu 40dc283b2d fix locating data chunks 2021-08-15 23:07:58 -07:00
Chris Lu 72eb6d5b9d ensure no writes to remote storage if content is not changed 2021-08-15 20:23:41 -07:00
Chris Lu 5d5a21ba2d adjust log format 2021-08-15 19:46:45 -07:00
Chris Lu 49b5e47bd1 retry forever with filer.remote.sync, and some refactoring 2021-08-15 12:38:26 -07:00
Chris Lu 8f7d2d317f readerAt need to use the right offset 
fix https://github.com/chrislusf/seaweedfs/issues/2259
 2021-08-15 11:55:58 -07:00
Chris Lu 9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu 0c66b173a4 fix 2021-08-13 11:31:43 -07:00
Chris Lu e02a317d3d adjust retry logic in case some data is partially written 2021-08-13 11:30:38 -07:00
Chris Lu 2d519c6cb6 adjust the retry logic 2021-08-13 11:13:30 -07:00
Chris Lu b961fcd338 filer: stream read from volume server, reduce memory usage 2021-08-13 11:00:11 -07:00
Chris Lu 5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Chris Lu bfac55e6c0 avoid integer overflow 
fix https://github.com/chrislusf/seaweedfs/issues/2254
 2021-08-11 22:22:49 -07:00
Chris Lu 9d85569c55 ensure using local quorum consistency 2021-08-10 05:10:57 -07:00
Chris Lu 713c035a6e shell: remote.cache remote.uncache 2021-08-09 14:35:18 -07:00
Chris Lu c5f38c365d go fmt 2021-08-08 22:30:36 -07:00
Chris Lu c0b12da4ef shell: add filer.remote.unmount 2021-08-08 22:26:37 -07:00
Chris Lu 13e45e1605 filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
Chris Lu 46b0cb8c86 Merge branch 'master' into add_remote_storage 2021-08-07 15:41:27 -07:00
Chris Lu de730b079d ChunkStreamReader implenents io.ReaderAt 2021-08-07 15:41:07 -07:00
Chris Lu 59732a0529 refactoring 2021-08-07 15:35:27 -07:00
Chris Lu ecb234f75a refactor 2021-08-07 14:46:23 -07:00
Chris Lu 679f800caa Merge branch 'master' into add_remote_storage 2021-08-06 20:27:12 -07:00
byunghwa.yun f3dc909b21 Change default permissions 2021-08-06 12:24:35 +09:00
Chris Lu 6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu f6a9ad8001 fix tests 2021-08-04 00:31:06 -07:00
Chris Lu 81b255df8b Merge branch 'master' into add_remote_storage 2021-08-01 15:44:21 -07:00
Chris Lu 89933c46d2 s3: skip hidden directories in /buckets folder 2021-08-01 12:28:08 -07:00
Chris Lu 767edd3c08 rename 2021-07-31 23:52:09 -07:00
Chris Lu 9df7d16791 read <- remote_storage 2021-07-31 22:39:38 -07:00
Chris Lu 899963ac20 remote storage location changed to struct 2021-07-29 02:08:55 -07:00
Chris Lu c090d6bb25 add ReadRemote(), add read remote setup when filer starts 2021-07-28 22:43:12 -07:00
Chris Lu 1752eeb538 remote.mount saves the mapping 2021-07-27 03:26:35 -07:00
Chris Lu 4b94b03d90 directory to remote storage mapping 2021-07-27 01:16:28 -07:00
Chris Lu 99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00
byunghwa.yun 2b28a818f1 Fix mysql sql for batch delete 2021-07-25 11:06:14 +09:00
Chris Lu bdb632fa62
Merge pull request #2211 from qieqieplus/filer-bloom-filter 
add bloom filter for filer leveldb/rocksdb
 2021-07-22 23:35:54 -07:00
qieqieplus 2b46df13f8 add bloom filter for filer leveldb/rocksdb 2021-07-23 14:05:59 -06:00
Chris Lu 182288f860 filer: fix mysql, postgres batch delete error 2021-07-22 08:23:20 -07:00
Chris Lu 7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu 70effac0d3 configure and store remote configurations 2021-07-21 02:24:34 -07:00
Chris Lu a125c8fbe2
Merge pull request #2204 from nivekuil/tokenpolicy 
cassandra: use LocalQuorum for all queries
 2021-07-20 15:37:31 -07:00
nivekuil b9a67d46c5 cassandra: use LocalQuorum for all queries 
This changes this filer store from eventual to strong consistency at the cost
of read performance.
 2021-07-20 14:47:39 -07:00
Chris Lu 54c8bc8673 Merge branch 'remote_overlay' 2021-07-19 23:18:35 -07:00
Chris Lu 7ab389e7ec optimization: improve random range query for large files 2021-07-19 23:07:22 -07:00
Chris Lu 93c37cfded
Update filechunk_manifest.go 2021-07-19 18:41:41 -07:00
Chris Lu 450222dd64 add remote to filer.Entry and filer_pb entry, add RemoteConf 2021-07-19 02:47:27 -07:00
nivekuil 2faf96f002 cassandra: Use LocalOne instead of One consistency 2021-07-16 04:29:46 -07:00
Chris Lu 3d624d1e16 rename 2021-07-09 03:19:21 -07:00
Chris Lu ecce300964 s3 config read via grpc 2021-07-09 02:48:03 -07:00
Chris Lu 2cb8b31ea7
Merge pull request #2179 from nivekuil/tokenpolicy 
Cassandra: Use TokenAwareHostPolicy with fallback to localDC by default
 2021-07-02 13:51:53 -07:00
nivekuil 8425705643 Cassandra: Use TokenAwareHostPolicy by default with fallback 
See https://pkg.go.dev/github.com/gocql/gocql#hdr-Data_center_awareness_and_query_routing
 2021-07-02 13:50:01 -07:00
陈杨文 6f683e6572
also failed on mips64 
alpine support mips64, will try to build on next version
 2021-06-30 04:36:00 +08:00
Chris Lu 41db292332 skip s390 ppc64le 
due to https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22703
 2021-06-29 10:06:38 -07:00
Chris Lu cc7714fdbe logging changes to debug 2021-06-27 05:54:16 -07:00
Konstantin Lebedev 6aa1a56ec8 avoid crashes Galera Cluster 
https://github.com/chrislusf/seaweedfs/issues/2125
 2021-06-15 18:12:39 +05:00
Chris Lu 28a4a1f8d6 fix for mysql2 postgres2 on fast dropping buckets 2021-06-13 07:31:56 -07:00
Chris Lu 9357911a95 remove all bucket metadata 
fix https://github.com/chrislusf/seaweedfs/issues/2118
 2021-06-10 23:37:54 -07:00
Chris Lu 21ad9a4ac2 filer: mongodb avoids E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key 2021-06-06 13:12:01 -07:00
Chris Lu ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu 67dd094b35 sqlite: does not support non-linux/darwin/windows 2021-05-25 17:36:15 -07:00
Konstantin Lebedev 84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Konstantin Lebedev 0e404cd84b Chunk download stats 2021-05-24 12:14:50 +05:00
Konstantin Lebedev bb42633522 rm chunk Download stats 2021-05-24 12:08:12 +05:00
Chris Lu 064269bb57 filer: Support sqlite as filer meta store 2021-05-23 23:58:28 -07:00
Chris Lu 2d0a0733ec filer: bootstrap from peer filer 
fix https://github.com/chrislusf/seaweedfs/issues/1861
 2021-05-22 03:17:02 -07:00
Konstantin Lebedev 3325b850cc rm func CheckAllChunkViews 2021-05-21 16:05:59 +05:00
Konstantin Lebedev 26a4f34a57 del checks 
url err logging
stats chunk fetch
 2021-05-21 15:59:12 +05:00
Konstantin Lebedev 42f631f549 fix https://github.com/chrislusf/seaweedfs/issues/2046 2021-05-20 13:19:35 +05:00
Konstantin Lebedev 03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" 
This reverts commit ac71117e
 2021-05-20 11:45:21 +05:00
Chris Lu 1737af480a adjust logs 2021-05-10 21:47:51 -07:00
Chris Lu 55a8f57381 go fmt 2021-05-06 03:37:51 -07:00
Chris Lu ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu fa0d973113
Merge pull request #2037 from utsl42/reader_at 
make reader_at handle random reads more efficiently for FUSE
 2021-04-28 17:31:46 -07:00
Nathan Hawkins 042de9359c make reader_at handle random reads more efficiently for FUSE 2021-04-28 19:13:37 -04:00
Konstantin Lebedev c2269123d3 fix aws style Etag for chunks 2021-04-28 22:28:05 +05:00
Chris Lu 79f2e780c1 ensure name pattern checking is case sensitive 2021-04-24 11:51:23 -07:00
Chris Lu ddc8643ee0 filer: directory listing adds namePatternExclude 
fix https://github.com/chrislusf/seaweedfs/issues/2023
 2021-04-24 11:49:03 -07:00
Chris Lu 89eb9f6e70 clean up .uploads directory in mysql and postgres tables 
fix https://github.com/chrislusf/seaweedfs/issues/1957

When no uploads are running, you can run this SQL to clean up.
delete from <bucket_name> where directory like '/.uploads/%'
 2021-04-22 23:23:23 -07:00
Chris Lu d41e6826d3 adjust logging 2021-04-18 13:06:38 -07:00
Konstantin Lebedev 5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Konstantin Lebedev 011e6e90ee Merge branch 'upstreamMaster' into iamapipr 2021-04-06 13:50:33 +05:00
Chris Lu 5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu 8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
LazyDBA247-Anyvision 7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision 4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
LazyDBA247-Anyvision 4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu bd7471d877 refactor 2021-03-25 12:05:59 -07:00
Konstantin Lebedev 9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Chris Lu 2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Chris Lu d48dd0c738 use []byte directly instead of bytes.Buffer 2021-03-22 22:12:57 -07:00
Konstantin Lebedev eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Chris Lu 69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Konstantin Lebedev 06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu 4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Konstantin Lebedev 46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Konstantin Lebedev 7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Chris Lu 35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu 2d4c2db81d filer: leveldb, rocksdb auto create store directory 
fix https://github.com/chrislusf/seaweedfs/issues/1901
 2021-03-14 13:20:14 -07:00
Chris Lu 6d3a96eb56 filer: mysql2, postgres2 trigger actions on bucket creation and deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1877
 2021-03-13 22:07:39 -08:00
Chris Lu 7d57664c2d mount: internals switch to filer.Entry instead of protobuf 2021-03-12 00:36:38 -08:00
Chris Lu d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Konstantin Lebedev 58cdcc6d6e error if urls not found 2021-03-11 23:34:36 +05:00
Konstantin Lebedev 4db21012c1 error if read chunk zero data size 2021-03-11 22:38:59 +05:00
Konstantin Lebedev 15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Chris Lu 5be83dd4a9 add comments 2021-03-09 14:13:48 -08:00
Chris Lu 9d8ca513d9 refactor 2021-03-09 13:21:26 -08:00
Chris Lu 9abb041763 filer source: support filerProxy mode 2021-02-28 16:19:47 -08:00
Chris Lu 776f497469 filer: fs.configure should try to read from entry.content also 
related to https://github.com/chrislusf/seaweedfs/issues/1792
 2021-02-18 17:07:02 -08:00
Chris Lu 3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu 6daa932f5c refactoring to get master function, instead of passing master values directly 
this will enable retrying later
 2021-02-17 20:55:55 -08:00
Chris Lu 3f8b0da677 filer: do not print password on error 
fix https://github.com/chrislusf/seaweedfs/issues/1809
 2021-02-17 02:13:52 -08:00