Chris Lu
30b30b8fe0
volume.tier.move: passing non-empty disk type
2021-02-22 01:59:03 -08:00
Chris Lu
5da63e045e
avoid moving to another server with the same volume id
2021-02-22 01:44:18 -08:00
Chris Lu
6a4546d2c0
shell: add volume.tier.move
2021-02-22 01:30:07 -08:00
Chris Lu
1c233ad986
refactoring
2021-02-22 00:28:42 -08:00
Chris Lu
151c281f36
2.27
2021-02-21 19:29:27 -08:00
Chris Lu
258e93bc86
Merge pull request #1823 from bingoohuang/master
...
feature: support command line arguments in a configuration file
2021-02-20 22:35:18 -08:00
Chris Lu
03c643aa6b
fix test
...
fileSize need to be divided by 8
2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa
disk file avoid file.Stat()
2021-02-20 20:06:06 -08:00
bingoo
1af6c96cd0
fix log help
2021-02-21 10:46:01 +08:00
Chris Lu
a2383b3b12
Revert "Revert "avoid file.Stat()""
...
This reverts commit 7ef4c24f18
.
2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18
Revert "avoid file.Stat()"
...
This reverts commit 98c93ca465
.
2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094
skip seek() when index file writes
2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465
avoid file.Stat()
...
avoid one Syscall, but did not help on performance though
2021-02-20 09:44:17 -08:00
bingoohuang
ee082ae402
revert comment temporarily
2021-02-20 16:59:38 +08:00
bingoohuang
ee7cdf3668
revert ErrXyz to ErrorXyz temporarily.
2021-02-20 16:58:08 +08:00
bingoohuang
50df484d86
support command line arguments in a configuration file so we can weed master -config=master.conf
, weed volume -config=volume.conf
and etc.
2021-02-20 16:52:57 +08:00
bingoohuang
f69356f589
Merge pull request #3 from chrislusf/master
2021-02-20 16:45:02 +08:00
Chris Lu
565f7a6e72
Update data_node.go
2021-02-19 14:22:36 -08:00
Chris Lu
a37473ae60
add back volume ids
...
address https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-782339576
2021-02-19 14:22:12 -08:00
Chris Lu
a1210d1e8d
Revert "volume: avoid file.stat(), file.seek() if possible during writes"
...
This reverts commit c78409a598
.
2021-02-19 03:56:27 -08:00
Chris Lu
b961cd6208
add WIP message
2021-02-19 03:39:19 -08:00
Chris Lu
f465d63b5d
grpc should fail when heart beating to master
...
fix https://github.com/chrislusf/seaweedfs/issues/1820
2021-02-19 03:37:29 -08:00
Chris Lu
c576ad04ac
fix volume server display for volumes
2021-02-19 01:38:56 -08:00
bingoohuang
7ffe736d20
renaming to ErrXyz
2021-02-19 15:59:55 +08:00
Chris Lu
c78409a598
volume: avoid file.stat(), file.seek() if possible during writes
2021-02-18 23:22:54 -08:00
Chris Lu
29575dadc5
remove extra files
2021-02-18 19:14:58 -08:00
Chris Lu
73958e357d
add descriptive error if no free volumes
2021-02-18 19:10:20 -08:00
Chris Lu
776f497469
filer: fs.configure should try to read from entry.content also
...
related to https://github.com/chrislusf/seaweedfs/issues/1792
2021-02-18 17:07:02 -08:00
Chris Lu
e1992c83de
webdav adds disktype
2021-02-18 15:45:44 -08:00
Chris Lu
c2ad6f1047
webdav add replication setting
...
fix https://github.com/chrislusf/seaweedfs/issues/1817
2021-02-18 12:15:09 -08:00
Chris Lu
816ef0ed30
add more help message
...
fix https://github.com/chrislusf/seaweedfs/issues/1816
2021-02-18 10:34:49 -08:00
bingoohuang
352ac2f271
Merge remote-tracking branch 'origin/master'
2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34
use backticks instead of double quotes to avoid escaped additionally in regex
2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009
go fmt
2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c
refactoring to get master function, instead of passing master values directly
...
this will enable retrying later
2021-02-17 20:55:55 -08:00
Chris Lu
dd9f3a0104
add sleep between upload retries
2021-02-17 20:55:11 -08:00
Chris Lu
aa17311063
s3: set filename when uploading file chunks
2021-02-17 20:54:53 -08:00
Chris Lu
3f8b0da677
filer: do not print password on error
...
fix https://github.com/chrislusf/seaweedfs/issues/1809
2021-02-17 02:13:52 -08:00
Chris Lu
cd866664a8
skip JWT if fileId is empty
...
related to https://github.com/chrislusf/seaweedfs/issues/1808
2021-02-16 15:39:12 -08:00
Chris Lu
632c94d438
Merge branch 'extend_to_disk_type'
2021-02-16 11:06:13 -08:00
Chris Lu
0611233f16
adjust printing
2021-02-16 10:55:30 -08:00
Chris Lu
68775d29e3
fix tests
2021-02-16 10:51:03 -08:00
Chris Lu
7403cd43c5
only use "" for hdd to avoid two values for the same thing
2021-02-16 10:50:42 -08:00
Chris Lu
b314d78e97
fix print
2021-02-16 10:48:28 -08:00
Chris Lu
53ca7e66ef
avoid dead lock
2021-02-16 10:48:16 -08:00
Chris Lu
38efc6f572
simplify
2021-02-16 10:47:32 -08:00
Chris Lu
ad70107eb4
ec balance among nodes with HDD
2021-02-16 06:22:50 -08:00
Chris Lu
3097b9a9b7
fix existence checking
2021-02-16 05:59:43 -08:00
Chris Lu
ebf320ec22
avoid divided by zero
2021-02-16 05:59:24 -08:00
Chris Lu
a0c6db361c
avoid nil
2021-02-16 05:33:38 -08:00
Chris Lu
36f95e50a9
avoid possible nil disk info
2021-02-16 05:13:48 -08:00
Chris Lu
43101ccea0
move to the empty nodes first
2021-02-16 04:27:16 -08:00
Chris Lu
38bbef7ec1
avoid nil
2021-02-16 04:16:46 -08:00
Chris Lu
cb9cc29518
volume.list display; fix updating maxVolumeCount for disk
2021-02-16 03:55:24 -08:00
Chris Lu
3fe628f04e
use hdd instead of empty string
2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab
this can compile now!!!
2021-02-16 02:47:02 -08:00
Chris Lu
b9b5b932c5
filer: postgres2 mysql2 avoid repeatedly creating tables
2021-02-15 16:19:24 -08:00
Chris Lu
71f0c19515
2.26
2021-02-15 13:38:38 -08:00
Chris Lu
8b169aa47c
s3: Critical - S3 Delete Multiple Objects, delete the bucket
...
fix https://github.com/chrislusf/seaweedfs/issues/1806
2021-02-15 13:38:29 -08:00
Chris Lu
50202d1391
Merge pull request #1805 from LazyDBA247-Anyvision/master
...
better postgresql connection pool management
2021-02-14 22:26:29 -08:00
Chris Lu
8726ef34c1
2.25
2021-02-14 22:23:43 -08:00
LazyDBA247-Anyvision
7f458d5e78
better postgres connection pool management
...
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime )
to enable refresh of stale connections.
2021-02-15 07:45:09 +02:00
Chris Lu
0bc3a1f9e8
disk type only supports hdd and ssd, not ready for random tags yet
2021-02-14 11:38:43 -08:00
LazyDBA247-Anyvision
51b4963e2e
postgres2 & memsql2
...
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
2021-02-14 13:14:36 +02:00
Chris Lu
876dbe3d26
collect disk types
2021-02-13 23:25:16 -08:00
Chris Lu
861f8fab9b
refactoring
2021-02-13 22:34:12 -08:00
Chris Lu
881f05f5a0
adjust help message
2021-02-13 16:13:52 -08:00
Chris Lu
ef76365ec2
adjust help message
2021-02-13 15:47:08 -08:00
Chris Lu
7ce647f27e
support customizable disk type
2021-02-13 15:42:42 -08:00
Chris Lu
4bd8a692d8
disk type can be generic tags
2021-02-13 13:50:14 -08:00
Chris Lu
712b3e9e53
mount: release need to avoid nil reader
...
fix https://github.com/chrislusf/seaweedfs/issues/1803
2021-02-13 13:39:37 -08:00
Chris Lu
4ce56bac08
fix format error
2021-02-12 06:44:26 -08:00
Chris Lu
0604afcda7
mount: remove unnecessary flush on release
2021-02-12 06:01:04 -08:00
Chris Lu
7d9dc3c6a2
use fasthttp lib to read
2021-02-12 05:59:52 -08:00
Chris Lu
487e435679
adjust http max idle connections per host
...
related to https://github.com/chrislusf/seaweedfs/issues/1802
2021-02-12 03:47:15 -08:00
Chris Lu
0f426ce34d
Merge pull request #1801 from kmlebedev/recoveringRabbitMQ
...
Do reconnect to RabbitMQ
2021-02-11 01:05:03 -08:00
Konstantin Lebedev
94eac4f00e
Do reconnect to RabbitMQ
...
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
2021-02-11 13:59:36 +05:00
Chris Lu
885ca34748
volume: fail fast if idx files are missing
...
fix https://github.com/chrislusf/seaweedfs/issues/1796
2021-02-11 00:44:40 -08:00
Chris Lu
a0e84c4fbc
go fmt
2021-02-10 23:41:05 -08:00
Chris Lu
c959aaa19b
mount: load directory based on correct filer path
...
fix https://github.com/chrislusf/seaweedfs/issues/1799
2021-02-10 23:14:18 -08:00
Konstantin Lebedev
0cfed8c3cb
restart replication if RabbitMQ connection closed
...
https://github.com/google/go-cloud/issues/2958
2021-02-10 15:29:05 +05:00
Chris Lu
770393a48c
volume: add capability to change disk type when moving a volume
2021-02-09 23:58:08 -08:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
Chris Lu
a833021132
fix refactoring left over
2021-02-07 23:03:03 -08:00
Chris Lu
5e4b5109dd
2.24
2021-02-07 22:29:43 -08:00
Chris Lu
15c60cbb26
close the grpc connection after 10 hours
...
related to https://github.com/chrislusf/seaweedfs/issues/1782
2021-02-07 03:50:01 -08:00
bingoohuang
7256902fb0
fix typo offset.ToAcutalOffset to offset.ToActualOffset
2021-02-07 12:11:51 +08:00
bingoohuang
94ea3bd3a5
renaming NeedleMapType to NeedleMapKind
2021-02-07 09:00:03 +08:00
Chris Lu
c3af72d950
fier store: fix elastic search regression
...
fix https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-04 01:30:14 -08:00
Konstantin Lebedev
2fb4c60540
fix log warn
2021-02-03 21:34:32 +05:00
Konstantin Lebedev
24ec17219b
RabbitMQ delay retry with Dead Letter Exchange
...
https://github.com/chrislusf/seaweedfs/issues/1773
https://github.com/google/go-cloud/issues/2952
2021-02-03 21:32:30 +05:00
Chris Lu
a331bbb3ae
filer: should return 204 on DELETE to nonexistent file
...
related to
https://github.com/chrislusf/seaweedfs/issues/1776
https://github.com/chrislusf/seaweedfs/issues/1160
2021-02-03 00:40:31 -08:00
Chris Lu
5353e38469
S3: should return 204 on DELETE to nonexistent file
...
fix https://github.com/chrislusf/seaweedfs/issues/1776
2021-02-03 00:35:44 -08:00
Chris Lu
2396ac234c
filer: fix elastic search pagination
...
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-02 11:34:12 -08:00
Chris Lu
609daaf387
s3: DeleteMultipleObjectsHandler clean up leftover empty folders
...
fix https://github.com/chrislusf/seaweedfs/issues/1772
2021-02-01 10:49:17 -08:00
Chris Lu
d67ccb66c0
webdav: can start together with "weed server" or "weed filer"
2021-01-31 22:16:52 -08:00
Chris Lu
76430790b9
webdav: cache to version specific folder
2021-01-31 22:02:03 -08:00
Chris Lu
318a3d2efc
2.23
2021-01-31 20:18:06 -08:00
Chris Lu
4be51c0701
filer: leveldb and hbase may miss files when listing large directories more than 1024
...
fix https://github.com/chrislusf/seaweedfs/issues/1768
2021-01-31 20:11:44 -08:00
Chris Lu
1102ae32c4
fix concurrent map reads
2021-01-31 18:26:26 -08:00
Chris Lu
314dc1c957
filer: etcd store fix listing
...
fix https://github.com/chrislusf/seaweedfs/issues/1767
2021-01-30 13:38:44 -08:00
Chris Lu
d475c89fcc
go fmt
2021-01-28 15:23:46 -08:00
Chris Lu
19295600f9
mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes
2021-01-28 15:23:16 -08:00
Chris Lu
990fa69bfe
add back AdjustedUrl() related code
2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2
fmt
2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac
delete from the deepest directory first when checking empty folders
2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925
s3: batch purge empty folders
2021-01-28 13:20:06 -08:00
Chris Lu
cf252fc0cd
mount: report error when Flush()
...
The error in Release() is not working.
See https://github.com/jaderhs/libfuse/blob/master/FAQ
related to https://github.com/chrislusf/seaweedfs/issues/1765
2021-01-28 04:46:37 -08:00
Chris Lu
9a06c35da4
replicate: incremental sink only contains new and updated files
...
address da08402ba2
2021-01-28 02:39:22 -08:00
Chris Lu
da08402ba2
replicate: use creation time for local incremental file sink
...
related to https://github.com/chrislusf/seaweedfs/pull/1762
2021-01-28 02:17:41 -08:00
Chris Lu
822f1ade9d
Merge pull request #1762 from kmlebedev/backupsink
...
replication to create time date directory
2021-01-28 02:03:33 -08:00
Konstantin Lebedev
be1062b7fc
rename in scaffold
2021-01-28 14:59:20 +05:00
Konstantin Lebedev
02fdc0a333
rename backup to local_incremental and use mtime
2021-01-28 14:56:13 +05:00
Chris Lu
5f72c388bc
Merge pull request #1764 from danlsgiga/handle-umask
...
Use provided umask on mount
2021-01-28 01:09:20 -08:00
Daniel Santos
d38613c49a
match upstream
2021-01-27 20:56:41 -07:00
Chris Lu
b81956bcb5
mount: invalidate kernel cache when mounted to a filer path
...
fix https://github.com/chrislusf/seaweedfs/issues/1752#issuecomment-768178422
2021-01-27 10:28:37 -08:00
Konstantin Lebedev
6b54ff9912
replication to create time date directory
2021-01-27 15:01:33 +05:00
Chris Lu
f20ec82a28
update help message
...
see https://github.com/chrislusf/seaweedfs/issues/1758#issuecomment-768123224
2021-01-27 00:48:31 -08:00
Chris Lu
1045ab8b07
avoid error message
2021-01-27 00:21:59 -08:00
Chris Lu
0488c84bea
adjust help message
2021-01-27 00:17:52 -08:00
Konstantin Lebedev
3634811408
replication to todays date directory
2021-01-27 12:45:58 +05:00
Chris Lu
5e07afb0f0
Merge pull request #1759 from kmlebedev/sink.local
...
replication to local disk storage
2021-01-26 22:31:54 -08:00
Chris Lu
b3f66199db
shell: volume.fix.replication adds collectionPattern with wildcard characters
...
fix https://github.com/chrislusf/seaweedfs/issues/1758
2021-01-26 22:30:39 -08:00
Daniel Santos
0c64d60fdb
Fix log message with correct mode
2021-01-26 21:45:52 -07:00
Daniel Santos
0aaaa9b470
Fix var assignment
2021-01-26 21:06:08 -07:00
Daniel Santos
5a761bb7a4
Enforce umask correctly
2021-01-26 20:42:10 -07:00
Chris Lu
ad2a20c8a5
notification add ack and nack
2021-01-26 11:08:44 -08:00
Konstantin Lebedev
612b7975a1
replication to local disk storage
2021-01-26 22:50:25 +05:00
Chris Lu
3a1d3d3413
mount: properly invalidate kernel node cache entry
...
fix https://github.com/chrislusf/seaweedfs/issues/1752
2021-01-26 02:50:53 -08:00
Chris Lu
cc1f3907ff
filer.sync: default filerProxy to true
2021-01-25 10:14:42 -08:00
Chris Lu
62ebb917e0
mount: avoid the need to specify different cache directory for different mount
...
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
2021-01-25 09:10:20 -08:00
Chris Lu
17f7c1c43f
Merge branch 'mount_from_outside_cluster'
2021-01-24 19:03:02 -08:00
Chris Lu
00707ec00f
mount: outsideContainerClusterMode proxy through filer
...
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
2021-01-24 19:01:58 -08:00
Chris Lu
9bcb28a3ea
avoid possible nil reader
...
fix https://github.com/chrislusf/seaweedfs/issues/1754
2021-01-24 17:19:35 -08:00
Chris Lu
2c5eac5705
2.22
2021-01-24 14:06:09 -08:00
Chris Lu
6ca10725b8
Revert "mount: when outside cluster network, use filer as proxy to access volume servers"
...
This reverts commit 096e088d7b
.
2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b
mount: when outside cluster network, use filer as proxy to access volume servers
2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89
s3: fix for listing objects if more than 1000 in the folder
...
many existing tools have max key set to 1000
2021-01-24 00:27:52 -08:00
Chris Lu
80b8692688
filer.sync: replicate outside of either cluster, only need to see filers
2021-01-24 00:01:44 -08:00
Chris Lu
f8dbb03bdd
filer: append operation returns final file size
2021-01-23 04:19:49 -08:00
Chris Lu
759482e2ac
filer: proxy all http methods to volume servers
2021-01-23 03:52:38 -08:00
Chris Lu
937cfacc01
filer: add "proxyToFileId" to reverse proxy to a volume server
2021-01-23 03:43:48 -08:00
Chris Lu
20ef3bb8d4
mount: use direct_io to avoid OS page cache
...
fix https://github.com/chrislusf/seaweedfs/issues/1752
2021-01-22 22:39:46 -08:00
Chris Lu
711c3f3939
filer: fix http status 206 setting
...
fix https://github.com/chrislusf/seaweedfs/issues/1753
2021-01-22 00:26:30 -08:00
Chris Lu
024c186a48
filer.meta.tail: output json format for easier parsing
2021-01-21 22:39:25 -08:00
Chris Lu
84f05787f8
mount: limit background requests and congestion threshold
2021-01-20 23:30:14 -08:00
Chris Lu
f98817cfe6
filer: support appending to a file
2021-01-20 13:40:32 -08:00
Chris Lu
1b8e3da295
refactoring
2021-01-20 12:59:54 -08:00
Chris Lu
1a3f859c58
filer store: postgres configuration adjust text
2021-01-19 18:12:07 -08:00
Chris Lu
b434f7e4e0
filer store: postgres adjust default config, add optioanl schema
2021-01-19 18:10:36 -08:00