gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6491 commits 28 branches 278 tags 135 MiB
Commit graph

430 commits

Author SHA1 Message Date
Chris Lu 49b5e47bd1 retry forever with filer.remote.sync, and some refactoring 2021-08-15 12:38:26 -07:00
Chris Lu 8f7d2d317f readerAt need to use the right offset 
fix https://github.com/chrislusf/seaweedfs/issues/2259
 2021-08-15 11:55:58 -07:00
Chris Lu 9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu 0c66b173a4 fix 2021-08-13 11:31:43 -07:00
Chris Lu e02a317d3d adjust retry logic in case some data is partially written 2021-08-13 11:30:38 -07:00
Chris Lu 2d519c6cb6 adjust the retry logic 2021-08-13 11:13:30 -07:00
Chris Lu b961fcd338 filer: stream read from volume server, reduce memory usage 2021-08-13 11:00:11 -07:00
Chris Lu 5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Chris Lu bfac55e6c0 avoid integer overflow 
fix https://github.com/chrislusf/seaweedfs/issues/2254
 2021-08-11 22:22:49 -07:00
Chris Lu 9d85569c55 ensure using local quorum consistency 2021-08-10 05:10:57 -07:00
Chris Lu 713c035a6e shell: remote.cache remote.uncache 2021-08-09 14:35:18 -07:00
Chris Lu c5f38c365d go fmt 2021-08-08 22:30:36 -07:00
Chris Lu c0b12da4ef shell: add filer.remote.unmount 2021-08-08 22:26:37 -07:00
Chris Lu 13e45e1605 filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
Chris Lu 46b0cb8c86 Merge branch 'master' into add_remote_storage 2021-08-07 15:41:27 -07:00
Chris Lu de730b079d ChunkStreamReader implenents io.ReaderAt 2021-08-07 15:41:07 -07:00
Chris Lu 59732a0529 refactoring 2021-08-07 15:35:27 -07:00
Chris Lu ecb234f75a refactor 2021-08-07 14:46:23 -07:00
Chris Lu 679f800caa Merge branch 'master' into add_remote_storage 2021-08-06 20:27:12 -07:00
byunghwa.yun f3dc909b21 Change default permissions 2021-08-06 12:24:35 +09:00
Chris Lu 6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu f6a9ad8001 fix tests 2021-08-04 00:31:06 -07:00
Chris Lu 81b255df8b Merge branch 'master' into add_remote_storage 2021-08-01 15:44:21 -07:00
Chris Lu 89933c46d2 s3: skip hidden directories in /buckets folder 2021-08-01 12:28:08 -07:00
Chris Lu 767edd3c08 rename 2021-07-31 23:52:09 -07:00
Chris Lu 9df7d16791 read <- remote_storage 2021-07-31 22:39:38 -07:00
Chris Lu 899963ac20 remote storage location changed to struct 2021-07-29 02:08:55 -07:00
Chris Lu c090d6bb25 add ReadRemote(), add read remote setup when filer starts 2021-07-28 22:43:12 -07:00
Chris Lu 1752eeb538 remote.mount saves the mapping 2021-07-27 03:26:35 -07:00
Chris Lu 4b94b03d90 directory to remote storage mapping 2021-07-27 01:16:28 -07:00
Chris Lu 99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00
byunghwa.yun 2b28a818f1 Fix mysql sql for batch delete 2021-07-25 11:06:14 +09:00
Chris Lu bdb632fa62
Merge pull request #2211 from qieqieplus/filer-bloom-filter 
add bloom filter for filer leveldb/rocksdb
 2021-07-22 23:35:54 -07:00
qieqieplus 2b46df13f8 add bloom filter for filer leveldb/rocksdb 2021-07-23 14:05:59 -06:00
Chris Lu 182288f860 filer: fix mysql, postgres batch delete error 2021-07-22 08:23:20 -07:00
Chris Lu 7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu 70effac0d3 configure and store remote configurations 2021-07-21 02:24:34 -07:00
Chris Lu a125c8fbe2
Merge pull request #2204 from nivekuil/tokenpolicy 
cassandra: use LocalQuorum for all queries
 2021-07-20 15:37:31 -07:00
nivekuil b9a67d46c5 cassandra: use LocalQuorum for all queries 
This changes this filer store from eventual to strong consistency at the cost
of read performance.
 2021-07-20 14:47:39 -07:00
Chris Lu 54c8bc8673 Merge branch 'remote_overlay' 2021-07-19 23:18:35 -07:00
Chris Lu 7ab389e7ec optimization: improve random range query for large files 2021-07-19 23:07:22 -07:00
Chris Lu 93c37cfded
Update filechunk_manifest.go 2021-07-19 18:41:41 -07:00
Chris Lu 450222dd64 add remote to filer.Entry and filer_pb entry, add RemoteConf 2021-07-19 02:47:27 -07:00
nivekuil 2faf96f002 cassandra: Use LocalOne instead of One consistency 2021-07-16 04:29:46 -07:00
Chris Lu 3d624d1e16 rename 2021-07-09 03:19:21 -07:00
Chris Lu ecce300964 s3 config read via grpc 2021-07-09 02:48:03 -07:00
Chris Lu 2cb8b31ea7
Merge pull request #2179 from nivekuil/tokenpolicy 
Cassandra: Use TokenAwareHostPolicy with fallback to localDC by default
 2021-07-02 13:51:53 -07:00
nivekuil 8425705643 Cassandra: Use TokenAwareHostPolicy by default with fallback 
See https://pkg.go.dev/github.com/gocql/gocql#hdr-Data_center_awareness_and_query_routing
 2021-07-02 13:50:01 -07:00
陈杨文 6f683e6572
also failed on mips64 
alpine support mips64, will try to build on next version
 2021-06-30 04:36:00 +08:00
Chris Lu 41db292332 skip s390 ppc64le 
due to https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22703
 2021-06-29 10:06:38 -07:00
Chris Lu cc7714fdbe logging changes to debug 2021-06-27 05:54:16 -07:00
Konstantin Lebedev 6aa1a56ec8 avoid crashes Galera Cluster 
https://github.com/chrislusf/seaweedfs/issues/2125
 2021-06-15 18:12:39 +05:00
Chris Lu 28a4a1f8d6 fix for mysql2 postgres2 on fast dropping buckets 2021-06-13 07:31:56 -07:00
Chris Lu 9357911a95 remove all bucket metadata 
fix https://github.com/chrislusf/seaweedfs/issues/2118
 2021-06-10 23:37:54 -07:00
Chris Lu 21ad9a4ac2 filer: mongodb avoids E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key 2021-06-06 13:12:01 -07:00
Chris Lu ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu 67dd094b35 sqlite: does not support non-linux/darwin/windows 2021-05-25 17:36:15 -07:00
Konstantin Lebedev 84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Konstantin Lebedev 0e404cd84b Chunk download stats 2021-05-24 12:14:50 +05:00
Konstantin Lebedev bb42633522 rm chunk Download stats 2021-05-24 12:08:12 +05:00
Chris Lu 064269bb57 filer: Support sqlite as filer meta store 2021-05-23 23:58:28 -07:00
Chris Lu 2d0a0733ec filer: bootstrap from peer filer 
fix https://github.com/chrislusf/seaweedfs/issues/1861
 2021-05-22 03:17:02 -07:00
Konstantin Lebedev 3325b850cc rm func CheckAllChunkViews 2021-05-21 16:05:59 +05:00
Konstantin Lebedev 26a4f34a57 del checks 
url err logging
stats chunk fetch
 2021-05-21 15:59:12 +05:00
Konstantin Lebedev 42f631f549 fix https://github.com/chrislusf/seaweedfs/issues/2046 2021-05-20 13:19:35 +05:00
Konstantin Lebedev 03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" 
This reverts commit ac71117e
 2021-05-20 11:45:21 +05:00
Chris Lu 1737af480a adjust logs 2021-05-10 21:47:51 -07:00
Chris Lu 55a8f57381 go fmt 2021-05-06 03:37:51 -07:00
Chris Lu ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu fa0d973113
Merge pull request #2037 from utsl42/reader_at 
make reader_at handle random reads more efficiently for FUSE
 2021-04-28 17:31:46 -07:00
Nathan Hawkins 042de9359c make reader_at handle random reads more efficiently for FUSE 2021-04-28 19:13:37 -04:00
Konstantin Lebedev c2269123d3 fix aws style Etag for chunks 2021-04-28 22:28:05 +05:00
Chris Lu 79f2e780c1 ensure name pattern checking is case sensitive 2021-04-24 11:51:23 -07:00
Chris Lu ddc8643ee0 filer: directory listing adds namePatternExclude 
fix https://github.com/chrislusf/seaweedfs/issues/2023
 2021-04-24 11:49:03 -07:00
Chris Lu 89eb9f6e70 clean up .uploads directory in mysql and postgres tables 
fix https://github.com/chrislusf/seaweedfs/issues/1957

When no uploads are running, you can run this SQL to clean up.
delete from <bucket_name> where directory like '/.uploads/%'
 2021-04-22 23:23:23 -07:00
Chris Lu d41e6826d3 adjust logging 2021-04-18 13:06:38 -07:00
Konstantin Lebedev 5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Konstantin Lebedev 011e6e90ee Merge branch 'upstreamMaster' into iamapipr 2021-04-06 13:50:33 +05:00
Chris Lu 5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu 8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
LazyDBA247-Anyvision 7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision 4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
LazyDBA247-Anyvision 4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu bd7471d877 refactor 2021-03-25 12:05:59 -07:00
Konstantin Lebedev 9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Chris Lu 2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Chris Lu d48dd0c738 use []byte directly instead of bytes.Buffer 2021-03-22 22:12:57 -07:00
Konstantin Lebedev eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Chris Lu 69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Konstantin Lebedev 06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu 4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Konstantin Lebedev 46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Konstantin Lebedev 7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Chris Lu 35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu 2d4c2db81d filer: leveldb, rocksdb auto create store directory 
fix https://github.com/chrislusf/seaweedfs/issues/1901
 2021-03-14 13:20:14 -07:00
Chris Lu 6d3a96eb56 filer: mysql2, postgres2 trigger actions on bucket creation and deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1877
 2021-03-13 22:07:39 -08:00
Chris Lu 7d57664c2d mount: internals switch to filer.Entry instead of protobuf 2021-03-12 00:36:38 -08:00
Chris Lu d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Konstantin Lebedev 58cdcc6d6e error if urls not found 2021-03-11 23:34:36 +05:00