Chris Lu
2539ba0b62
fix compilation
2021-10-25 14:38:48 -07:00
Chris Lu
5f2d7c1589
erasure coding: skip erasure coding if less than recommended 4 nodes
2021-10-25 14:38:11 -07:00
Chris Lu
5435027ff0
volume copy: stream out copying progress and avoid grpc request timeout
...
fix https://github.com/chrislusf/seaweedfs/issues/2386
2021-10-24 02:52:56 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
...
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Konstantin Lebedev
be4b3ed509
AclHandlers
2021-10-11 15:03:56 +05:00
Chris Lu
e862b2529a
refactor
2021-10-01 12:10:11 -07:00
Konstantin Lebedev
5e64b22b45
check that the topology has been updated
2021-10-01 18:51:22 +05:00
Konstantin Lebedev
2cecde89c3
rename opt volumesPerStep
2021-10-01 00:17:54 +05:00
Konstantin Lebedev
fc51ffce2b
https://github.com/chrislusf/seaweedfs/issues/1846
2021-09-30 20:24:24 +05:00
Chris Lu
ad5099e570
refactor
2021-09-19 12:02:23 -07:00
Chris Lu
2789d10342
go fmt
2021-09-14 10:37:06 -07:00
Chris Lu
119d5908dd
shell: do not need to lock to see volume -h
2021-09-13 22:13:34 -07:00
Chris Lu
6cd1ce8b74
erasure coding: add cleanup step if anything goes wrong
2021-09-13 01:55:49 -07:00
Chris Lu
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
Chris Lu
0b4269b6a8
remember commands even if failed
2021-09-08 15:55:19 -07:00
Chris Lu
0128239c0f
handle ipv6 addresses
2021-09-07 16:43:54 -07:00
Chris Lu
8e4b43a017
minor
2021-09-06 15:13:38 -07:00
Chris Lu
57a95887d2
remote.cache remote.uncache supports all mounted directories
2021-09-05 14:47:06 -07:00
Chris Lu
c735608685
obfuscate secret keys on display
2021-09-05 14:23:49 -07:00
Chris Lu
32e94de86a
default auto trimming suffix to true
2021-09-05 13:27:35 -07:00
Chris Lu
60573fd3e2
option to map remote bucket to trimmed bucket name
2021-09-05 11:55:52 -07:00
Chris Lu
d57d4c5f8f
shell: add remote.mount.buckets
2021-09-04 21:37:25 -07:00
Chris Lu
d983aa4c7d
correct filtering
2021-09-04 13:58:14 -07:00
Chris Lu
49952ba905
fix cache/uncache filters
2021-09-04 13:45:39 -07:00
Chris Lu
df29281536
rename
2021-09-04 05:11:06 -07:00
Chris Lu
c3db389e42
refactor
2021-09-04 05:09:38 -07:00
Chris Lu
8707ef00a6
refactor
2021-09-04 05:02:20 -07:00
Chris Lu
63c84584bb
refactor
2021-09-04 04:53:36 -07:00
Chris Lu
da49d25950
auto list of storage types
2021-09-04 00:18:21 -07:00
Chris Lu
38f73c8324
add gcs project id
2021-09-03 23:38:33 -07:00
Chris Lu
7ce97b59d8
go fmt
2021-09-01 02:45:42 -07:00
Chris Lu
3bd48c4f29
filer.remote.sync: exit when directory is unmounted
...
this will not propagate the deletions back to the cloud
2021-09-01 01:29:22 -07:00
Chris Lu
cca588429b
shell: fs.meta.save skip saving system logs
2021-08-31 23:05:56 -07:00
Chris Lu
0ee208c1a4
adjust example
2021-08-31 02:04:32 -07:00
Chris Lu
97b7948cd9
adjust error message
2021-08-31 01:07:34 -07:00
Chris Lu
c9205458a1
cloud drive: create mount directory if not exists
2021-08-31 01:07:19 -07:00
Chris Lu
1dfcdc53b7
cloud drive: add storj, filebase
2021-08-30 17:43:14 -07:00
Chris Lu
701cc133f3
cloud drive: s3 add option for v4 signature
2021-08-30 17:28:33 -07:00
Chris Lu
f117f7bf54
adjust comment
2021-08-30 15:56:14 -07:00
Chris Lu
44d0ac95f6
add cluster id if already set
2021-08-29 22:19:46 -07:00
Chris Lu
18bf556d93
fix comments
2021-08-29 22:19:25 -07:00
Chris Lu
21f37a0a75
remove unused code
2021-08-29 22:19:11 -07:00
Chris Lu
001a472057
cloud mount: remote storage support hdfs
2021-08-29 18:41:29 -07:00
Chris Lu
975fbc9ee3
add wasabi example
2021-08-28 22:49:38 -07:00
Chris Lu
65a29ad42a
add warning on unmount a folder
2021-08-28 02:25:48 -07:00
Chris Lu
a22f37b01c
remove tencent region
2021-08-28 02:23:03 -07:00
Chris Lu
0d8936f6cd
add aliyun example
2021-08-28 01:09:48 -07:00
Chris Lu
49a8dfb976
adjust default concurrent level
2021-08-26 17:05:56 -07:00
Chris Lu
6a0bb7106b
cloud drive: parallelize remote storage downloading
2021-08-26 16:16:26 -07:00
Chris Lu
05a648bb96
refactor: separating out remote.proto
2021-08-26 15:18:34 -07:00
Chris Lu
c08ac536ed
cloud drive: add support for Wasabi
...
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
2021-08-25 17:34:29 -07:00
Chris Lu
a7a914f120
cloud drive: add support for Baidu BOS
2021-08-24 23:46:33 -07:00
Chris Lu
2d805beed4
add help mesages
2021-08-24 23:28:35 -07:00
Chris Lu
c6c97bd83d
add default env variables
2021-08-24 23:25:36 -07:00
Chris Lu
a19c728034
cloud drive: add support for Tencent COS
2021-08-24 23:19:45 -07:00
Chris Lu
19a81d25af
cloud drive: add support for Aliyun OSS
2021-08-24 23:14:24 -07:00
Chris Lu
47d775cf68
cloud drive: add support for BackBlaze
2021-08-24 22:30:06 -07:00
Chris Lu
6bab20d862
cat entry content
2021-08-24 02:38:32 -07:00
Chris Lu
e9ebe24f2e
cloud drive: add support for Azure
2021-08-24 01:18:30 -07:00
Chris Lu
00c4e06caa
cloud drive: s3 configurable force path style
2021-08-23 03:30:41 -07:00
Chris Lu
2836a58d87
cloud drive: S3 supports storage class
2021-08-23 02:18:59 -07:00
Chris Lu
ca35a77e85
adjust help message
2021-08-23 00:37:55 -07:00
Chris Lu
258063de26
cloud drive: add google cloud storage
2021-08-23 00:29:27 -07:00
Chris Lu
2158d4fe4d
adjust help message
2021-08-21 02:17:10 -07:00
Chris Lu
ccb3df41f8
elide secret key
2021-08-17 11:27:08 -07:00
Chris Lu
49b5e47bd1
retry forever with filer.remote.sync, and some refactoring
2021-08-15 12:38:26 -07:00
Chris Lu
a539d64896
refactor
2021-08-15 12:09:54 -07:00
Chris Lu
ec989b0377
remove println
2021-08-15 02:10:27 -07:00
Chris Lu
9462f5129a
shell: add "remote.meta.sync"
2021-08-15 01:53:46 -07:00
Chris Lu
c34747c79d
rename, fix wrong logic.
2021-08-14 21:46:34 -07:00
Chris Lu
cb53802752
adjust help message
2021-08-14 15:55:53 -07:00
Chris Lu
889b143fa7
adjust modification detection logic
2021-08-14 15:44:47 -07:00
Chris Lu
708debca14
remote.cache and uncache: more flexible options to select files to cache or uncache
2021-08-14 15:11:55 -07:00
Chris Lu
0db2517994
go fmt
2021-08-14 02:55:44 -07:00
Chris Lu
0f7d4556d8
shell: volume.tier.move makes up changes if volume move failed
2021-08-13 03:09:28 -07:00
Chris Lu
333cdce485
add verbose message
2021-08-13 02:57:14 -07:00
Chris Lu
057ef429ac
format
2021-08-10 12:33:29 -07:00
Chris Lu
b63b042afc
dedup keeps the largest replica
2021-08-10 12:30:41 -07:00
Chris Lu
0526db12e2
do not treat read only volumes differently
2021-08-10 11:37:12 -07:00
Chris Lu
e50a5b8e28
minor: print disk type
2021-08-10 11:10:09 -07:00
Chris Lu
db6275a0c8
print out balance ratio
2021-08-10 10:43:42 -07:00
Chris Lu
85832d02c0
wait for goroutines
2021-08-10 04:13:12 -07:00
Chris Lu
8ff6c9a0c6
output format
2021-08-10 03:25:18 -07:00
Chris Lu
48f448ee09
parallelize tier move
2021-08-10 03:08:29 -07:00
Chris Lu
69a6da7969
avoid fail on tail error
2021-08-10 02:50:28 -07:00
Chris Lu
18228f3044
fix help message
2021-08-10 02:48:41 -07:00
Chris Lu
02f728cb54
fix bug with remote.uncache
2021-08-09 15:24:21 -07:00
Chris Lu
402315f117
go fmt
2021-08-09 14:37:34 -07:00
Chris Lu
713c035a6e
shell: remote.cache remote.uncache
2021-08-09 14:35:18 -07:00
Chris Lu
c5f38c365d
go fmt
2021-08-08 22:30:36 -07:00
Chris Lu
df85f7a1eb
adjust help message
2021-08-08 22:30:12 -07:00
Chris Lu
c0b12da4ef
shell: add filer.remote.unmount
2021-08-08 22:26:37 -07:00
Chris Lu
882a93dacd
fix tests
2021-08-08 17:56:26 -07:00
Chris Lu
eed26af266
Merge branch 'master' into add_remote_storage
2021-08-08 15:48:04 -07:00
Chris Lu
26c222f596
shell: volume.tier.move avoid moving all volumes to one destination
2021-08-08 15:12:39 -07:00
Chris Lu
13e45e1605
filer.remote.sync can work now
2021-08-08 01:21:42 -07:00
Chris Lu
679f800caa
Merge branch 'master' into add_remote_storage
2021-08-06 20:27:12 -07:00
Chris Lu
0c0f77e2ae
skip not found error on deletion
2021-08-06 19:35:47 -07:00
Chris Lu
8828f485c0
print volume deletion error
2021-08-06 19:30:22 -07:00
Chris Lu
e23ca3845b
Merge pull request #2237 from combineads/add_mtime
...
Add crtime and mtime
2021-08-05 21:27:13 -07:00
Chris Lu
286e5dd375
Merge branch 'master' into add_remote_storage
2021-08-05 21:07:04 -07:00
Chris Lu
1e22166939
adjust error message
2021-08-05 21:06:55 -07:00
byunghwa.yun
e84fad9acf
Add crtime and mtime
2021-08-06 12:13:37 +09:00
Chris Lu
1a5d29520c
add tests
2021-08-05 14:49:24 -07:00
Chris Lu
6b743dbbf9
refactor client subscribe metadata
2021-08-04 16:25:46 -07:00
Chris Lu
b9ecf1e3a8
refacotring
2021-08-04 14:56:13 -07:00
Chris Lu
d84c311699
refactoring
2021-08-04 12:30:18 -07:00
Chris Lu
f6a9ad8001
fix tests
2021-08-04 00:31:06 -07:00
Chris Lu
42969c9c62
Merge branch 'master' into add_remote_storage
2021-08-03 21:20:05 -07:00
Chris Lu
8cf0c515bf
shell: volume.fix.replication retries even when there is no error #2235
...
fix https://github.com/chrislusf/seaweedfs/issues/2235
2021-08-03 09:32:55 -07:00
Chris Lu
2ae9705442
adjust text
2021-08-01 22:55:19 -07:00
Chris Lu
3739717092
Revert "adds a test"
...
This reverts commit f690643b47
.
2021-08-01 22:54:45 -07:00
Chris Lu
f690643b47
adds a test
2021-08-01 22:53:50 -07:00
Chris Lu
6de786185d
volume.balance: balance read only volumes first
2021-08-01 22:32:50 -07:00
byunghwa.yun
9e839cb5cc
Add force option in volume.deleteEmpty command
2021-08-02 11:37:20 +09:00
Chris Lu
81b255df8b
Merge branch 'master' into add_remote_storage
2021-08-01 15:44:21 -07:00
Chris Lu
2ca1839d77
shell: add volume.deleteEmpty
command
2021-08-01 15:33:45 -07:00
Chris Lu
9df7d16791
read <- remote_storage
2021-07-31 22:39:38 -07:00
Chris Lu
f5a69a0e44
Merge branch 'master' into add_remote_storage
2021-07-31 09:39:48 -07:00
byunghwa.yun
5d4438a72f
Fix typo
2021-07-31 22:23:52 +09:00
Chris Lu
a3290faf17
shell command to calculate size / number of volumes in a collection
...
fix https://github.com/chrislusf/seaweedfs/issues/2224
2021-07-30 15:49:46 -07:00
Chris Lu
c6f992b2a3
remove dead code
2021-07-30 15:18:01 -07:00
Chris Lu
899963ac20
remote storage location changed to struct
2021-07-29 02:08:55 -07:00
Chris Lu
4deac06da0
log fix
2021-07-28 22:25:04 -07:00
divanikus
5c6270a93a
shell: ability to use wildcards for collections, all collections if ommited
2021-07-27 23:53:01 +03:00
Chris Lu
1752eeb538
remote.mount saves the mapping
2021-07-27 03:26:35 -07:00
Chris Lu
4b94b03d90
directory to remote storage mapping
2021-07-27 01:16:28 -07:00
Chris Lu
99b599aa8a
remote.mount
2021-07-26 22:53:44 -07:00
Chris Lu
35f70c51b0
refactor
2021-07-25 03:37:37 -07:00
Chris Lu
72eec84167
shell: add fs.mkdir
2021-07-25 02:37:39 -07:00
Chris Lu
092932af8f
shell: faster bucket delete
...
avoid waiting for large object metadata
2021-07-23 11:25:46 -07:00
Chris Lu
7359193e97
go fmt
2021-07-21 14:38:12 -07:00
Chris Lu
67537ff05b
Merge pull request #2208 from combineads/add_rm_options
2021-07-21 11:08:43 -07:00
byunghwa.yun
41d559f54e
Add fs rm options
2021-07-21 21:10:36 +09:00
Chris Lu
70effac0d3
configure and store remote configurations
2021-07-21 02:24:34 -07:00
Chris Lu
3138805b33
Merge pull request #2200 from combineads/add_fs_rm
...
weed shell: fs.rm remove file or directory
2021-07-21 00:35:54 -07:00
Chris Lu
7ab389e7ec
optimization: improve random range query for large files
2021-07-19 23:07:22 -07:00
byunghwa.yun
b23b307e08
Add fs rm
2021-07-20 08:49:28 +09:00
Chris Lu
99155e4880
add -retry
option for volume.fix.replication
...
fix https://github.com/chrislusf/seaweedfs/issues/2191
2021-07-16 12:13:46 -07:00
Chris Lu
2f209675ab
Added -retry
option for volumeServer.evacuate
...
related to https://github.com/chrislusf/seaweedfs/issues/2191
2021-07-16 12:08:21 -07:00
Chris Lu
fb7a1be1c4
refactor
2021-07-16 11:47:37 -07:00
Chris Lu
4be5b4ff5c
[volume.check.disk] skip errors and check next volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2129
2021-07-16 02:03:32 -07:00
Chris Lu
5a838dbe53
fix compilation
2021-07-13 11:40:21 -07:00
Chris Lu
6103649ffb
shell: volume.check.disk adds retries in case the volumes are just moved
...
related to https://github.com/chrislusf/seaweedfs/issues/2194
2021-07-13 11:19:56 -07:00
Chris Lu
01adc567aa
shell: volume.fsck deletes a volume is the volume has only orphaned data.
...
fix https://github.com/chrislusf/seaweedfs/issues/2190
2021-07-12 11:22:00 -07:00
bingoohuang
9e48bff8ee
fix typo in fs.configure help message text
2021-07-12 13:51:21 +08:00
Chris Lu
d013d6d968
shell: volume.fsck "reallyDeleteFromVolume" should send padded file ids to delete
...
fix https://github.com/chrislusf/seaweedfs/issues/2188
2021-07-10 23:16:06 -07:00
Chris Lu
3d624d1e16
rename
2021-07-09 03:19:21 -07:00
Chris Lu
da7bd62822
/etc files are stored inside metadata store
2021-07-09 02:33:14 -07:00
Chris Lu
08377fecb8
fixes
2021-06-27 23:32:57 -07:00