Chris Lu
cf1586a34d
add logs for writing to remote file
2021-11-27 22:09:23 -08:00
limd
8805c04128
fix redis2 sentinel config example
2021-11-25 19:20:02 +08:00
limd
ec03f22cc3
Merge remote-tracking branch 'origin/master'
2021-11-25 16:07:14 +08:00
limd
220797bd71
support redis sentinel
2021-11-25 15:57:03 +08:00
Chris Lu
6c27845be0
add retries when writing to remote s3
...
fix https://github.com/chrislusf/seaweedfs/issues/2465
2021-11-22 21:48:04 -08:00
Chris Lu
1f75f1f9dc
filer: fix mysql2 SQL template
2021-11-11 22:28:28 -08:00
Chris Lu
3abbaccb70
filer: fix mysql command to upsert
2021-11-11 22:27:13 -08:00
Chris Lu
c4e22b5a9a
filer: deprecate "-peers" option
2021-11-06 14:36:45 -07:00
Chris Lu
5ea86ef1da
Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()"
...
This reverts commit af71ae11aa
.
2021-11-05 17:52:15 -07:00
Chris Lu
af71ae11aa
master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()
2021-11-03 01:09:48 -07:00
Chris Lu
ab97b17e62
better printout
2021-11-02 23:45:47 -07:00
Chris Lu
5160eb08f7
shell: optionally read filer address from master
2021-11-02 23:38:45 -07:00
Chris Lu
edbf6d297b
filer.meta.tail: add example to send metadata to elastic search
2021-10-31 18:01:33 -07:00
Chris Lu
3be3c17f59
volume vacuum: avoid timeout with streaming progress report
...
fix https://github.com/chrislusf/seaweedfs/issues/2396
2021-10-24 01:55:34 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
...
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
3e2acf677c
removing tikv to resolve "go mod tidy" problem
...
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back.
go mod tidy
go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker
go: finding module for package cloud.google.com/go/kms/apiv1
go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/coreos/etcd/clientv3/credentials
go: finding module for package github.com/coreos/etcd/clientv3/balancer
go: finding module for package github.com/d4l3k/messagediff
go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible
go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0
go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1
go: finding module for package google.golang.org/grpc/naming
github.com/chrislusf/seaweedfs/weed/filer/tikv imports
github.com/tikv/client-go/v2/tikv imports
go.etcd.io/etcd/clientv3 tested by
go.etcd.io/etcd/clientv3.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/proxy/grpcproxy imports
google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:27:02 -07:00
Chris Lu
bf218cd59d
removing etcd sequencer
...
causing go mod tidy problem. If anyone wants this, please help to resolve this first.
github.com/chrislusf/seaweedfs/weed/sequence imports
go.etcd.io/etcd/client tested by
go.etcd.io/etcd/client.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/proxy/grpcproxy imports
google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:10:46 -07:00
Chris Lu
6a030547a2
server: remove peer check if not starting master
...
more fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 03:27:10 -07:00
Chris Lu
62c2732fd1
add back logic to check master peers
...
fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 02:51:26 -07:00
Chris Lu
d4bb16e20e
Merge pull request #2354 from chrislusf/bptree
...
Add Redis3
2021-10-04 02:43:15 -07:00
Chris Lu
04662126bb
add redis3
2021-10-04 01:04:27 -07:00
Chris Lu
366f522a2d
add redis3
2021-10-04 01:01:31 -07:00
Chris Lu
b297849147
typo
2021-10-01 23:24:54 -07:00
Chris Lu
af207bbaf0
retry both assign volume and uploading data
...
fix https://github.com/chrislusf/seaweedfs/issues/2351
2021-10-01 23:23:39 -07:00
Chris Lu
4a1d4d7462
s3: default to allow empty folder for better performance
2021-09-26 22:34:14 -07:00
Chris Lu
a814f3f0a8
adjust metadata tail output
2021-09-25 01:04:51 -07:00
Chris Lu
52fe86df45
use default 10000 for grpc port
2021-09-20 14:05:59 -07:00
Chris Lu
84fdda85e7
go fmt
2021-09-19 12:06:15 -07:00
Chris Lu
94a01fcfcb
filer.remote.gateway: add options to include or exclude new bucket names to mirror
2021-09-15 23:04:16 -07:00
Chris Lu
b5f4910412
adjust help messages
2021-09-15 22:53:10 -07:00
Chris Lu
f0907eb83c
add command filre.remote.gateway
2021-09-15 22:48:04 -07:00
Chris Lu
63da4bbb54
separate filer.remote.gateway command to avoid confusion
2021-09-15 22:47:17 -07:00
Chris Lu
2789d10342
go fmt
2021-09-14 10:37:06 -07:00
Chris Lu
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
Chris Lu
2c9d4c8f43
custom grpc port: filer
2021-09-12 02:28:37 -07:00
Chris Lu
e690a2be16
custom grpc port: volume server
2021-09-12 02:25:15 -07:00
Chris Lu
232ad2fe65
custom grpc port: master follower
2021-09-12 02:25:00 -07:00
Chris Lu
bd1f800efe
custom grpc port: master
2021-09-12 02:19:10 -07:00
Chris Lu
2b8ab1534a
mysql table DDL: minor, not necessary, but a little more strict
...
the directory is already hashed into dirhash. So this change is not strictly required.
2021-09-08 23:55:18 -07:00
Chris Lu
574485ec69
better IP v6 support
2021-09-07 19:29:42 -07:00
Chris Lu
0128239c0f
handle ipv6 addresses
2021-09-07 16:43:54 -07:00
Chris Lu
6923af7280
refactoring
2021-09-06 16:20:49 -07:00
Chris Lu
64f6532fbe
update help message
2021-09-06 16:16:22 -07:00
Chris Lu
308d4b2dae
filer.remote.sync: createBucketWithRandomSuffix defaults to true
2021-09-06 15:14:26 -07:00
Chris Lu
c218ef20c7
filer.remote.sync: automatically detect the primary remote storage
2021-09-06 15:10:55 -07:00
Chris Lu
1702ce5395
remove unused code
2021-09-06 12:01:44 -07:00
Chris Lu
60573fd3e2
option to map remote bucket to trimmed bucket name
2021-09-05 11:55:52 -07:00
Chris Lu
bdefdee4e6
filer.remote.sync: add option to add randomized suffix to buckets to avoid conflicts
2021-09-04 22:46:28 -07:00
Chris Lu
98c68d7a79
filer.remote.sync fix upload logic
2021-09-04 18:46:28 -07:00
Chris Lu
16a342eb8a
re-ordering
2021-09-04 13:46:44 -07:00
Chris Lu
796b2eb929
remember sync time
2021-09-04 13:46:22 -07:00
Chris Lu
99d4594ee8
update remote storage mount mapping
2021-09-04 05:22:34 -07:00
Chris Lu
13cc2ef723
when creating bucket, use remote storage client for default storage
2021-09-04 04:54:55 -07:00
Chris Lu
4b28c5f6c3
filer.remote.sync: split into buckets mode and single directory mode
2021-09-04 04:35:46 -07:00
Chris Lu
03a31587ce
go fmt
2021-09-03 20:42:28 -07:00
Chris Lu
7f2a8246a9
minor
2021-09-02 12:32:59 -07:00
Chris Lu
9a73b0e3c9
refactor
2021-09-02 07:07:16 -07:00
Chris Lu
e281f0fa82
refactor
2021-09-02 06:53:21 -07:00
Chris Lu
7ce97b59d8
go fmt
2021-09-01 02:45:42 -07:00
Chris Lu
3bd48c4f29
filer.remote.sync: exit when directory is unmounted
...
this will not propagate the deletions back to the cloud
2021-09-01 01:29:22 -07:00
Chris Lu
8e125339d5
line wrap
2021-08-31 23:30:28 -07:00
Chris Lu
68ea99190f
add additionalPathPrefixes
2021-08-31 23:26:26 -07:00
Rain Li
e5f9ff983d
Merge branch 'master' into filerstore-tikv
2021-08-30 15:52:54 +08:00
Chris Lu
6deee4c0b9
import hdfs
2021-08-29 18:49:43 -07:00
Chris Lu
a31f2907f0
cloud drive: filer.remote.sync supports remove folder
2021-08-29 18:46:28 -07:00
Chris Lu
05d2774117
refactoring
2021-08-28 22:49:38 -07:00
Chris Lu
a9b41900d8
handle cases when mounted directory does not exist
2021-08-28 01:56:56 -07:00
Chris Lu
05a648bb96
refactor: separating out remote.proto
2021-08-26 15:18:34 -07:00
yulai.li
c1dc5ab4ac
Add deleterange_concurrency to filer configuration file
2021-08-26 18:25:08 +08:00
yulai.li
de8ef28460
Update filer.toml scaffold to support tikv store
2021-08-26 16:26:27 +08:00
yulai.li
546efeba8f
Fix build bug
2021-08-26 16:20:35 +08:00
Chris Lu
6bab20d862
cat entry content
2021-08-24 02:38:32 -07:00
Chris Lu
e9ebe24f2e
cloud drive: add support for Azure
2021-08-24 01:18:30 -07:00
Chris Lu
258063de26
cloud drive: add google cloud storage
2021-08-23 00:29:27 -07:00
Chris Lu
05fc7db755
filer.copy: large files pack file ids into chunks
2021-08-18 23:46:54 -07:00
Chris Lu
72eb6d5b9d
ensure no writes to remote storage if content is not changed
2021-08-15 20:23:41 -07:00
Chris Lu
5a7c40510f
format output
2021-08-15 20:07:13 -07:00
Chris Lu
5d5a21ba2d
adjust log format
2021-08-15 19:46:45 -07:00
Chris Lu
c45c12aedc
metadata updates
2021-08-15 19:27:30 -07:00
Chris Lu
49b5e47bd1
retry forever with filer.remote.sync, and some refactoring
2021-08-15 12:38:26 -07:00
Chris Lu
c34747c79d
rename, fix wrong logic.
2021-08-14 21:46:34 -07:00
Chris Lu
0db2517994
go fmt
2021-08-14 02:55:44 -07:00
Chris Lu
7937db52e1
Filer locationPrefix configure does not exec replication #2257
...
fix https://github.com/chrislusf/seaweedfs/issues/2257
2021-08-14 02:54:13 -07:00
Chris Lu
5a0f92423e
use grpc and jwt
2021-08-12 21:40:33 -07:00
Chris Lu
6238644c35
remove gateway command
2021-08-12 20:52:04 -07:00
Chris Lu
d1d1fc772c
move some volume lookup operations to grpc
...
jwt related lookup will come in next commit
2021-08-12 20:33:00 -07:00
Chris Lu
5571f4f70a
master: add master.follower to handle read file id lookup requests
2021-08-12 18:10:59 -07:00
Chris Lu
5469019852
adjust data type
2021-08-12 17:54:34 -07:00
Chris Lu
713c035a6e
shell: remote.cache remote.uncache
2021-08-09 14:35:18 -07:00
Chris Lu
734c980040
volume: support concurrent download data size limit
2021-08-08 23:25:16 -07:00
Chris Lu
c5f38c365d
go fmt
2021-08-08 22:30:36 -07:00
Chris Lu
df85f7a1eb
adjust help message
2021-08-08 22:30:12 -07:00
Chris Lu
7412ccdf88
write back remote entry to local entry after uploading to remote
2021-08-08 17:55:03 -07:00
Chris Lu
dcf614a8c3
skip if the remote entry update is because of internal managerial operations
2021-08-08 15:58:10 -07:00
Chris Lu
13e45e1605
filer.remote.sync can work now
2021-08-08 01:21:42 -07:00
Chris Lu
6b743dbbf9
refactor client subscribe metadata
2021-08-04 16:25:46 -07:00
Chris Lu
d84c311699
refactoring
2021-08-04 12:30:18 -07:00
Chris Lu
d2ddf1dbdb
typo
2021-08-03 09:32:31 -07:00
Chris Lu
3afbf04007
add TLS grpc support for filer meta clients
2021-08-03 01:30:35 -07:00
Chris Lu
fe60c6ef9a
minor changes
2021-08-02 11:49:40 -07:00