Chris Lu
8707ef00a6
refactor
2021-09-04 05:02:20 -07:00
Chris Lu
8ec357b3d3
go mod
2021-09-03 23:25:33 -07:00
Chris Lu
7ce97b59d8
go fmt
2021-09-01 02:45:42 -07:00
yulai.li
b17b81529e
Add build tags for TiKV filerstore
2021-08-30 15:59:25 +08:00
Rain Li
e5f9ff983d
Merge branch 'master' into filerstore-tikv
2021-08-30 15:52:54 +08:00
Chris Lu
dc481c081c
remove unused function
2021-08-27 22:18:54 -07:00
Chris Lu
9242f3aaeb
fix build
2021-08-26 17:33:57 -07:00
Chris Lu
05a648bb96
refactor: separating out remote.proto
2021-08-26 15:18:34 -07:00
yulai.li
c1dc5ab4ac
Add deleterange_concurrency to filer configuration file
2021-08-26 18:25:08 +08:00
yulai.li
318757ef8c
Change DeleteFolderChildren to DeleteRange api
2021-08-26 17:49:56 +08:00
yulai.li
546efeba8f
Fix build bug
2021-08-26 16:20:35 +08:00
yulai.li
2088f28424
init post
2021-08-26 15:20:18 +08:00
Chris Lu
9bcf94b2b1
ensure multi-threaded correctness
2021-08-25 17:28:50 -07:00
Chris Lu
40dc283b2d
fix locating data chunks
2021-08-15 23:07:58 -07:00
Chris Lu
72eb6d5b9d
ensure no writes to remote storage if content is not changed
2021-08-15 20:23:41 -07:00
Chris Lu
5d5a21ba2d
adjust log format
2021-08-15 19:46:45 -07:00
Chris Lu
49b5e47bd1
retry forever with filer.remote.sync, and some refactoring
2021-08-15 12:38:26 -07:00
Chris Lu
8f7d2d317f
readerAt need to use the right offset
...
fix https://github.com/chrislusf/seaweedfs/issues/2259
2021-08-15 11:55:58 -07:00
Chris Lu
9462f5129a
shell: add "remote.meta.sync"
2021-08-15 01:53:46 -07:00
Chris Lu
0c66b173a4
fix
2021-08-13 11:31:43 -07:00
Chris Lu
e02a317d3d
adjust retry logic in case some data is partially written
2021-08-13 11:30:38 -07:00
Chris Lu
2d519c6cb6
adjust the retry logic
2021-08-13 11:13:30 -07:00
Chris Lu
b961fcd338
filer: stream read from volume server, reduce memory usage
2021-08-13 11:00:11 -07:00
Chris Lu
5a0f92423e
use grpc and jwt
2021-08-12 21:40:33 -07:00
Chris Lu
bfac55e6c0
avoid integer overflow
...
fix https://github.com/chrislusf/seaweedfs/issues/2254
2021-08-11 22:22:49 -07:00
Chris Lu
9d85569c55
ensure using local quorum consistency
2021-08-10 05:10:57 -07:00
Chris Lu
713c035a6e
shell: remote.cache remote.uncache
2021-08-09 14:35:18 -07:00
Chris Lu
c5f38c365d
go fmt
2021-08-08 22:30:36 -07:00
Chris Lu
c0b12da4ef
shell: add filer.remote.unmount
2021-08-08 22:26:37 -07:00
Chris Lu
13e45e1605
filer.remote.sync can work now
2021-08-08 01:21:42 -07:00
Chris Lu
46b0cb8c86
Merge branch 'master' into add_remote_storage
2021-08-07 15:41:27 -07:00
Chris Lu
de730b079d
ChunkStreamReader implenents io.ReaderAt
2021-08-07 15:41:07 -07:00
Chris Lu
59732a0529
refactoring
2021-08-07 15:35:27 -07:00
Chris Lu
ecb234f75a
refactor
2021-08-07 14:46:23 -07:00
Chris Lu
679f800caa
Merge branch 'master' into add_remote_storage
2021-08-06 20:27:12 -07:00
byunghwa.yun
f3dc909b21
Change default permissions
2021-08-06 12:24:35 +09:00
Chris Lu
6b743dbbf9
refactor client subscribe metadata
2021-08-04 16:25:46 -07:00
Chris Lu
f6a9ad8001
fix tests
2021-08-04 00:31:06 -07:00
Chris Lu
81b255df8b
Merge branch 'master' into add_remote_storage
2021-08-01 15:44:21 -07:00
Chris Lu
89933c46d2
s3: skip hidden directories in /buckets folder
2021-08-01 12:28:08 -07:00
Chris Lu
767edd3c08
rename
2021-07-31 23:52:09 -07:00
Chris Lu
9df7d16791
read <- remote_storage
2021-07-31 22:39:38 -07:00
Chris Lu
899963ac20
remote storage location changed to struct
2021-07-29 02:08:55 -07:00
Chris Lu
c090d6bb25
add ReadRemote(), add read remote setup when filer starts
2021-07-28 22:43:12 -07:00
Chris Lu
1752eeb538
remote.mount saves the mapping
2021-07-27 03:26:35 -07:00
Chris Lu
4b94b03d90
directory to remote storage mapping
2021-07-27 01:16:28 -07:00
Chris Lu
99b599aa8a
remote.mount
2021-07-26 22:53:44 -07:00
byunghwa.yun
2b28a818f1
Fix mysql sql for batch delete
2021-07-25 11:06:14 +09:00
Chris Lu
bdb632fa62
Merge pull request #2211 from qieqieplus/filer-bloom-filter
...
add bloom filter for filer leveldb/rocksdb
2021-07-22 23:35:54 -07:00
qieqieplus
2b46df13f8
add bloom filter for filer leveldb/rocksdb
2021-07-23 14:05:59 -06:00
Chris Lu
182288f860
filer: fix mysql, postgres batch delete error
2021-07-22 08:23:20 -07:00
Chris Lu
7359193e97
go fmt
2021-07-21 14:38:12 -07:00
Chris Lu
70effac0d3
configure and store remote configurations
2021-07-21 02:24:34 -07:00
Chris Lu
a125c8fbe2
Merge pull request #2204 from nivekuil/tokenpolicy
...
cassandra: use LocalQuorum for all queries
2021-07-20 15:37:31 -07:00
nivekuil
b9a67d46c5
cassandra: use LocalQuorum for all queries
...
This changes this filer store from eventual to strong consistency at the cost
of read performance.
2021-07-20 14:47:39 -07:00
Chris Lu
54c8bc8673
Merge branch 'remote_overlay'
2021-07-19 23:18:35 -07:00
Chris Lu
7ab389e7ec
optimization: improve random range query for large files
2021-07-19 23:07:22 -07:00
Chris Lu
93c37cfded
Update filechunk_manifest.go
2021-07-19 18:41:41 -07:00
Chris Lu
450222dd64
add remote to filer.Entry and filer_pb entry, add RemoteConf
2021-07-19 02:47:27 -07:00
nivekuil
2faf96f002
cassandra: Use LocalOne instead of One consistency
2021-07-16 04:29:46 -07:00
Chris Lu
3d624d1e16
rename
2021-07-09 03:19:21 -07:00
Chris Lu
ecce300964
s3 config read via grpc
2021-07-09 02:48:03 -07:00
Chris Lu
2cb8b31ea7
Merge pull request #2179 from nivekuil/tokenpolicy
...
Cassandra: Use TokenAwareHostPolicy with fallback to localDC by default
2021-07-02 13:51:53 -07:00
nivekuil
8425705643
Cassandra: Use TokenAwareHostPolicy by default with fallback
...
See https://pkg.go.dev/github.com/gocql/gocql#hdr-Data_center_awareness_and_query_routing
2021-07-02 13:50:01 -07:00
陈杨文
6f683e6572
also failed on mips64
...
alpine support mips64, will try to build on next version
2021-06-30 04:36:00 +08:00
Chris Lu
41db292332
skip s390 ppc64le
...
due to https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22703
2021-06-29 10:06:38 -07:00
Chris Lu
cc7714fdbe
logging changes to debug
2021-06-27 05:54:16 -07:00
Konstantin Lebedev
6aa1a56ec8
avoid crashes Galera Cluster
...
https://github.com/chrislusf/seaweedfs/issues/2125
2021-06-15 18:12:39 +05:00
Chris Lu
28a4a1f8d6
fix for mysql2 postgres2 on fast dropping buckets
2021-06-13 07:31:56 -07:00
Chris Lu
9357911a95
remove all bucket metadata
...
fix https://github.com/chrislusf/seaweedfs/issues/2118
2021-06-10 23:37:54 -07:00
Chris Lu
21ad9a4ac2
filer: mongodb avoids E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key
2021-06-06 13:12:01 -07:00
Chris Lu
ab606dec2a
filer: add path-specific option to enforce readonly
2021-06-04 01:03:41 -07:00
Chris Lu
67dd094b35
sqlite: does not support non-linux/darwin/windows
2021-05-25 17:36:15 -07:00
Konstantin Lebedev
84dce32a57
Merge branch 'master' into head_check_all_chunks
2021-05-24 12:28:19 +05:00
Konstantin Lebedev
0e404cd84b
Chunk download stats
2021-05-24 12:14:50 +05:00
Konstantin Lebedev
bb42633522
rm chunk Download stats
2021-05-24 12:08:12 +05:00
Chris Lu
064269bb57
filer: Support sqlite as filer meta store
2021-05-23 23:58:28 -07:00
Chris Lu
2d0a0733ec
filer: bootstrap from peer filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1861
2021-05-22 03:17:02 -07:00
Konstantin Lebedev
3325b850cc
rm func CheckAllChunkViews
2021-05-21 16:05:59 +05:00
Konstantin Lebedev
26a4f34a57
del checks
...
url err logging
stats chunk fetch
2021-05-21 15:59:12 +05:00
Konstantin Lebedev
42f631f549
fix https://github.com/chrislusf/seaweedfs/issues/2046
2021-05-20 13:19:35 +05:00
Konstantin Lebedev
03d1199d5f
Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader"
...
This reverts commit ac71117e
2021-05-20 11:45:21 +05:00
Chris Lu
1737af480a
adjust logs
2021-05-10 21:47:51 -07:00
Chris Lu
55a8f57381
go fmt
2021-05-06 03:37:51 -07:00
Chris Lu
ac71117ee6
revert PR #1903 avoid http error: superfluous response.WriteHeader
2021-05-05 15:11:39 -07:00
Chris Lu
fa0d973113
Merge pull request #2037 from utsl42/reader_at
...
make reader_at handle random reads more efficiently for FUSE
2021-04-28 17:31:46 -07:00
Nathan Hawkins
042de9359c
make reader_at handle random reads more efficiently for FUSE
2021-04-28 19:13:37 -04:00
Konstantin Lebedev
c2269123d3
fix aws style Etag for chunks
2021-04-28 22:28:05 +05:00
Chris Lu
79f2e780c1
ensure name pattern checking is case sensitive
2021-04-24 11:51:23 -07:00
Chris Lu
ddc8643ee0
filer: directory listing adds namePatternExclude
...
fix https://github.com/chrislusf/seaweedfs/issues/2023
2021-04-24 11:49:03 -07:00
Chris Lu
89eb9f6e70
clean up .uploads directory in mysql and postgres tables
...
fix https://github.com/chrislusf/seaweedfs/issues/1957
When no uploads are running, you can run this SQL to clean up.
delete from <bucket_name> where directory like '/.uploads/%'
2021-04-22 23:23:23 -07:00
Chris Lu
d41e6826d3
adjust logging
2021-04-18 13:06:38 -07:00
Konstantin Lebedev
5021bea698
GetUserPolicy
2021-04-10 23:57:45 +05:00
Konstantin Lebedev
011e6e90ee
Merge branch 'upstreamMaster' into iamapipr
2021-04-06 13:50:33 +05:00
Chris Lu
5e64f65632
ensure tables are created
...
fix https://github.com/chrislusf/seaweedfs/issues/1957
2021-04-02 12:02:26 -07:00
Chris Lu
8e404a1433
go fmt
2021-04-02 02:22:26 -07:00
Chris Lu
e06807a55b
adjust logs
2021-03-30 20:36:06 -07:00
LazyDBA247-Anyvision
7f44d953b5
fix GetBool
2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660
add enableUpsert=true
...
and rename config to upsertQuery
2021-03-30 00:32:03 +03:00
LazyDBA247-Anyvision
4a02389eb0
Adding custom insertQuery support for postgres/2 mysql/2
2021-03-29 09:58:13 +03:00
Chris Lu
bd7471d877
refactor
2021-03-25 12:05:59 -07:00
Konstantin Lebedev
9f26f2815c
SaveAs S3 Configuration
2021-03-25 12:34:11 +05:00
Chris Lu
2f7c7afdec
refactor: remove unused parameter
2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738
use []byte directly instead of bytes.Buffer
2021-03-22 22:12:57 -07:00
Konstantin Lebedev
eb54993a4e
Merge branch 'upstreamMaster' into check_chunkviews_mr
...
# Conflicts:
# weed/filer/filechunk_manifest.go
# weed/filer/stream.go
# weed/replication/repl_util/replication_util.go
# weed/util/fasthttp_util.go
2021-03-16 15:29:49 +05:00
Chris Lu
69694a17be
reverting 7d57664c2d
2021-03-16 02:59:26 -07:00
Konstantin Lebedev
06da02739d
CheckAllChunkViews() for HEAD requests only
2021-03-16 14:15:17 +05:00
Chris Lu
4b1ed227d1
revert fasthttp changes
...
related to https://github.com/chrislusf/seaweedfs/issues/1907
2021-03-16 00:33:14 -07:00
Konstantin Lebedev
46b9f5cff4
add debug logging
2021-03-15 19:33:33 +05:00
Konstantin Lebedev
7194a5e7bf
avoid http error: superfluous response.WriteHeader
...
https://github.com/chrislusf/seaweedfs/issues/1838
2021-03-15 18:52:59 +05:00
Chris Lu
35d939a1d2
go fmt
2021-03-14 13:21:02 -07:00
Chris Lu
2d4c2db81d
filer: leveldb, rocksdb auto create store directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1901
2021-03-14 13:20:14 -07:00
Chris Lu
6d3a96eb56
filer: mysql2, postgres2 trigger actions on bucket creation and deletion
...
fix https://github.com/chrislusf/seaweedfs/issues/1877
2021-03-13 22:07:39 -08:00
Chris Lu
7d57664c2d
mount: internals switch to filer.Entry instead of protobuf
2021-03-12 00:36:38 -08:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder
...
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-11 11:01:44 -08:00
Konstantin Lebedev
58cdcc6d6e
error if urls not found
2021-03-11 23:34:36 +05:00
Konstantin Lebedev
4db21012c1
error if read chunk zero data size
2021-03-11 22:38:59 +05:00
Konstantin Lebedev
15b018da34
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-10 19:41:35 +05:00
Chris Lu
5be83dd4a9
add comments
2021-03-09 14:13:48 -08:00
Chris Lu
9d8ca513d9
refactor
2021-03-09 13:21:26 -08:00
Chris Lu
9abb041763
filer source: support filerProxy mode
2021-02-28 16:19:47 -08:00
Chris Lu
776f497469
filer: fs.configure should try to read from entry.content also
...
related to https://github.com/chrislusf/seaweedfs/issues/1792
2021-02-18 17:07:02 -08:00
Chris Lu
3575d41009
go fmt
2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c
refactoring to get master function, instead of passing master values directly
...
this will enable retrying later
2021-02-17 20:55:55 -08:00
Chris Lu
3f8b0da677
filer: do not print password on error
...
fix https://github.com/chrislusf/seaweedfs/issues/1809
2021-02-17 02:13:52 -08:00
Chris Lu
b9b5b932c5
filer: postgres2 mysql2 avoid repeatedly creating tables
2021-02-15 16:19:24 -08:00
LazyDBA247-Anyvision
7f458d5e78
better postgres connection pool management
...
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime )
to enable refresh of stale connections.
2021-02-15 07:45:09 +02:00
LazyDBA247-Anyvision
51b4963e2e
postgres2 & memsql2
...
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
2021-02-14 13:14:36 +02:00
Chris Lu
7d9dc3c6a2
use fasthttp lib to read
2021-02-12 05:59:52 -08:00
Konstantin Lebedev
94eac4f00e
Do reconnect to RabbitMQ
...
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
2021-02-11 13:59:36 +05:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
Chris Lu
c3af72d950
fier store: fix elastic search regression
...
fix https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-04 01:30:14 -08:00
Chris Lu
2396ac234c
filer: fix elastic search pagination
...
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-02 11:34:12 -08:00
Chris Lu
4be51c0701
filer: leveldb and hbase may miss files when listing large directories more than 1024
...
fix https://github.com/chrislusf/seaweedfs/issues/1768
2021-01-31 20:11:44 -08:00
Chris Lu
314dc1c957
filer: etcd store fix listing
...
fix https://github.com/chrislusf/seaweedfs/issues/1767
2021-01-30 13:38:44 -08:00
Chris Lu
990fa69bfe
add back AdjustedUrl() related code
2021-01-28 14:36:29 -08:00
Chris Lu
00707ec00f
mount: outsideContainerClusterMode proxy through filer
...
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
2021-01-24 19:01:58 -08:00
Chris Lu
6ca10725b8
Revert "mount: when outside cluster network, use filer as proxy to access volume servers"
...
This reverts commit 096e088d7b
.
2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b
mount: when outside cluster network, use filer as proxy to access volume servers
2021-01-24 01:41:38 -08:00
Chris Lu
d5add83e85
filer store: add postgres2
2021-01-19 18:07:29 -08:00
Chris Lu
52a8f1470e
filer store: add mysql2
2021-01-19 17:21:50 -08:00
Chris Lu
93b3adba98
fix bucket creation
2021-01-19 15:55:51 -08:00
Chris Lu
fa0c8d5283
fix error
2021-01-19 14:05:48 -08:00
Chris Lu
4c5b752b04
restructuring sql stores
2021-01-19 13:53:16 -08:00
Chris Lu
ca8f793978
refactor: prepare for bucket specific tables
2021-01-19 12:34:58 -08:00
Chris Lu
90ce1eec19
add more input params
2021-01-19 11:32:55 -08:00
Chris Lu
de876c795d
minor fix
2021-01-18 01:14:27 -08:00
Chris Lu
09f49d1c04
refactoring
2021-01-16 19:52:15 -08:00
Chris Lu
a4063a5437
add stream list directory entries
2021-01-15 23:56:24 -08:00
Chris Lu
01dc8a43ba
fix rocksdb
2021-01-15 18:31:29 -08:00