chrislu
70e5a1b632
volume close should wait for committing compaction
2022-04-26 23:34:05 -07:00
Chris Lu
ffe028f8d0
Merge pull request #2974 from kmlebedev/wait_volume_closed_compression
...
waite volume being closed during compression idx
2022-04-26 23:29:22 -07:00
chrislu
37ab8909b0
use two flags: v.isCompacting and v.isCommitCompacting
2022-04-26 23:28:34 -07:00
chrislu
94f824e1ce
volume: sync to disk before copying volume files
...
address https://github.com/chrislusf/seaweedfs/issues/2976
2022-04-26 13:03:43 -07:00
Konstantin Lebedev
7315d1d039
waite volume being closed during compression idx
2022-04-26 13:40:42 +05:00
chrislu
8e9ad7db5a
fix bug deleting volume or unmount volume if a volume server has multiple directories
2022-04-19 22:58:57 -07:00
Konstantin Lebedev
1e35b4929f
shell vacuum volume by collection and volume id
2022-04-18 18:40:58 +05:00
Konstantin Lebedev
9438738693
avoid invalid memory address or nil pointer dereference
2022-04-18 12:10:22 +05:00
justin
3551ca2fcf
enhancement: replace sort.Slice with slices.SortFunc to reduce reflection
2022-04-18 10:35:43 +08:00
justin
d51a724101
fix: encode small chunk return error maybe have some bug.
2022-03-28 13:11:24 +08:00
Konstantin Lebedev
f53cff045f
checks disk file exist
2022-03-10 18:58:56 +05:00
Chris Lu
97a4b66df7
Merge pull request #2704 from guo-sj/fix_bugs_in_return_value
...
fix return value in storage/volume_vacuum.go:444
2022-02-24 00:49:29 -08:00
guosj
3e7aa1caf5
fix return value in storage/volume_vacuum.go:444
2022-02-24 15:54:36 +08:00
chrislu
c29bc9a367
fix error handling
2022-02-23 15:34:25 -08:00
Chris Lu
da58c748bc
Merge pull request #2698 from guo-sj/fix_bugs_in_return_value
2022-02-23 00:41:30 -08:00
Chris Lu
bd092d8318
Merge pull request #2697 from guo-sj/fix_bugs_in_return_value
2022-02-23 00:30:52 -08:00
guosj
d68c27f82d
fix another return value bug
2022-02-23 16:21:25 +08:00
guosj
8f9aa0cddd
fix bugs in return value
2022-02-23 16:17:48 +08:00
chrislu
a129bda7d9
sync data first before stopping
2022-02-16 09:11:34 -08:00
Eng Zer Jun
b92df1654c
test: use T.TempDir
to create temporary test directory
...
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-02-14 10:38:13 +08:00
Konstantin Lebedev
ef541972f8
updated needle with fsync
2022-02-08 00:10:53 +05:00
Konstantin Lebedev
fb97e234c9
skips compact if store is stopping
2022-02-07 20:16:15 +05:00
chrislu
433fde4b18
move error to a separate file
...
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Chris Lu
a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics
...
error metrics for filer and store
2022-02-04 22:35:13 -08:00
chrislu
76e297d64f
sync call to write file, avoid vif loading error
...
fix https://github.com/chrislusf/seaweedfs/issues/2633
2022-02-04 11:14:04 -08:00
Konstantin Lebedev
3f4e17aa24
error metrics for filer and store
2022-02-04 14:07:14 +05:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
chrislu
50ddd8c8e2
remove debug messages
...
fix https://github.com/chrislusf/seaweedfs/issues/2514
2021-12-16 00:58:15 -08:00
chrislu
488afa5002
volume: load volume can optionally be skipped, if ec volume exists
...
fix https://github.com/chrislusf/seaweedfs/issues/2489
2021-12-05 02:28:52 -08:00
chrislu
59e58c4b23
volume: fix loading old volume format
...
fix https://github.com/chrislusf/seaweedfs/issues/2487
2021-12-05 01:06:01 -08:00
Chris Lu
0c8dea9de8
go fmt
2021-11-02 23:39:16 -07:00
Chris Lu
c857cc7286
cloud tier: remove tagging since not all s3 vendors support this
2021-10-29 12:39:19 -07:00
Chris Lu
d04cdcf40d
s3 header add user agent
2021-10-29 12:28:24 -07:00
Chris Lu
3be3c17f59
volume vacuum: avoid timeout with streaming progress report
...
fix https://github.com/chrislusf/seaweedfs/issues/2396
2021-10-24 01:55:34 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
...
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
f0b928ff5e
go fmt
2021-10-11 23:23:46 -07:00
Chris Lu
3afa451cdc
volume: find a non-empty offset when binary searching by timestamp
2021-10-11 22:00:41 -07:00
Chris Lu
b530f12327
volume: find a non-empty offset when binary searching by timestamp
...
fix https://github.com/chrislusf/seaweedfs/issues/2364
2021-10-11 16:11:50 -07:00
Chris Lu
ce1efeb0eb
adds more error message
2021-10-09 04:54:14 -07:00
Chris Lu
d688e10ed1
do not try to compress if not sure about the file content
2021-10-07 13:29:00 -07:00
Chris Lu
332f5ad3a8
revert temporary changes that disabled compression
...
fix https://github.com/chrislusf/seaweedfs/issues/2362
2021-10-07 13:24:16 -07:00
Chris Lu
f0d1e7bd05
skip ec volumes when loading normal volumes
2021-10-05 02:31:44 -07:00
Chris Lu
f58ea6a2ee
add source name to error message
2021-10-01 02:19:30 -07:00
Chris Lu
2e9372dcf7
volume stream read skips deleted content
2021-09-27 03:07:44 -07:00
Chris Lu
5956a8b05a
adjust comment
2021-09-27 01:58:03 -07:00
Chris Lu
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
Chris Lu
8c6ff55226
add volume not found error type, to reduce error log
2021-09-11 14:26:41 -07:00
Chris Lu
1b942dfa3c
add error message
2021-09-02 12:32:46 -07:00
Chris Lu
7ce97b59d8
go fmt
2021-09-01 02:45:42 -07:00
Chris Lu
f811fd0903
fix build
2021-08-30 02:03:08 -07:00
Chris Lu
5adfdd7982
change to a new bloom filter library
2021-08-29 21:26:38 -07:00
Chris Lu
05a648bb96
refactor: separating out remote.proto
2021-08-26 15:18:34 -07:00
Chris Lu
c08ac536ed
cloud drive: add support for Wasabi
...
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
2021-08-25 17:34:29 -07:00
Chris Lu
00c4e06caa
cloud drive: s3 configurable force path style
2021-08-23 03:30:41 -07:00
Chris Lu
f0cc130849
do not force path style for better compatibility
2021-08-23 03:09:41 -07:00
Konstantin Lebedev
865b06b7fe
The IEEE assembler optimizations has been submitted and will be part of the Go 1.6 standard library.
...
https://github.com/klauspost/crc32
2021-08-17 13:06:48 +05:00
Chris Lu
78e8ddf910
Only when tailing volume, the zero-ed cookie should skip checking.
...
This only happens when checkCookie == false and fsync == false.
2021-08-13 02:09:35 -07:00
Chris Lu
a8617c1a39
tail volume: fix zero cookie problem from batch deletion
2021-08-13 01:54:35 -07:00
Chris Lu
734c980040
volume: support concurrent download data size limit
2021-08-08 23:25:16 -07:00
Chris Lu
9cc84a910f
volume: deletion can skip volume size checking
...
fix https://github.com/chrislusf/seaweedfs/issues/2225
2021-08-01 00:32:51 -07:00
Chris Lu
49c66e88a0
volume: change all writes to fsync during graceful stopping
...
fix https://github.com/chrislusf/seaweedfs/issues/2193
2021-07-13 01:29:57 -07:00
Chris Lu
b624090398
go fmt
2021-07-01 01:21:14 -07:00
Chris Lu
24e11d1e90
look back when adding to sorted values
...
look back when adding to sorted values, before adding it to overflow
2021-06-28 22:46:49 -07:00
Chris Lu
fc8dd58aea
volume: large_volume version has bug when using in memory index
...
fix https://github.com/chrislusf/seaweedfs/issues/2162
2021-06-28 15:48:07 -07:00
Chris Lu
56eb522b13
fix stats when a collection is deleted
2021-06-15 21:11:31 -07:00
Chris Lu
6c82326575
use bytes.Buffer to reduce memory allocation and gc
2021-06-06 13:42:36 -07:00
Konstantin Lebedev
99ef280c7c
avoid data loss after restarting a container with a volum server
2021-06-02 17:07:19 +05:00
Chris Lu
1aa7e99a89
skip file not found error when deleting
2021-05-15 09:37:39 -07:00
qieqieplus
ac26080bd2
fix concurrent vacuum & delete panic
2021-05-05 17:54:50 +08:00
bingoohuang
cf552417a7
minFreeSpace refactored
2021-04-27 10:37:24 +08:00
bingoohuang
31f1cdeac2
minFreeSpace argument allows size like 10GiB
2021-04-26 18:48:34 +08:00
Chris Lu
ca998328c2
do not add new volumes when below minFreeSpacePercent
...
fix https://github.com/chrislusf/seaweedfs/issues/2017
2021-04-21 23:11:11 -07:00
Chris Lu
a8114da02d
avoid thundering herd effect
...
transient errors may cause thundering herd effect to all trying to recover from remove ec shards
2021-04-21 10:17:12 -07:00
Chris Lu
ae74d8f02a
fix error message
...
related to https://github.com/chrislusf/seaweedfs/issues/2012
2021-04-21 01:40:16 -07:00
Chris Lu
283d703d50
adjust text
2021-04-15 11:29:58 -07:00
Chris Lu
8e404a1433
go fmt
2021-04-02 02:22:26 -07:00
Chris Lu
b465095db1
shell: add volume.check.disk to fix inconsistency for replicated volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1923
2021-03-22 00:03:16 -07:00
Chris Lu
102a951377
refactor, split into 2 files
2021-03-21 13:05:53 -07:00
Chris Lu
f6e2566ee5
better logs for cookie mismatch
2021-03-20 12:02:57 -07:00
Chris Lu
35d939a1d2
go fmt
2021-03-14 13:21:02 -07:00
Chris Lu
cb423312a4
prevent nil volume message
2021-03-13 11:05:29 -08:00
Chris Lu
972327f966
prevent nil volume nm
2021-03-13 11:04:51 -08:00
Chris Lu
737bde5ab7
refactoring, remove duplicated code
2021-03-09 23:23:01 -08:00
Chris Lu
828f6e9f4d
volume: auto add missing vif files
...
fix https://github.com/chrislusf/seaweedfs/issues/1878
2021-03-09 12:09:32 -08:00
qieqieplus
5b16820924
fix: collection stats won't update if all volumes expired at same iteration
2021-03-09 18:49:45 +08:00
Chris Lu
726edab054
avoid nil when closing an index
...
fix https://github.com/chrislusf/seaweedfs/issues/1870
2021-03-07 11:03:09 -08:00
Chris Lu
bdfed16d42
avoid nil exception
...
fix https://github.com/chrislusf/seaweedfs/issues/1869
2021-03-07 01:49:06 -08:00
Chris Lu
e9eddfb8a2
remove wrong writes
2021-03-06 14:26:27 -08:00
Chris Lu
38fc200e56
CRCWriter consistent with CRC
2021-03-06 14:26:27 -08:00
Chris Lu
400de380f4
volume server: support tcp direct put/get/delete
2021-03-05 02:29:38 -08:00
Chris Lu
c46d16b2f3
skip network error when remote volumes start
2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17
add logs for remote file loading
2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f
add vif file versions in case loading superblock fails
2021-03-01 00:48:30 -08:00
Chris Lu
d680676d45
skip already loaded backends
2021-03-01 00:47:03 -08:00
Chris Lu
540441fd38
go fmt
2021-02-28 20:34:14 -08:00
Chris Lu
2cca07b44c
cloud tier to non-AWS s3 gateways
2021-02-28 18:59:09 -08:00
Chris Lu
30b30b8fe0
volume.tier.move: passing non-empty disk type
2021-02-22 01:59:03 -08:00
Chris Lu
03c643aa6b
fix test
...
fileSize need to be divided by 8
2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa
disk file avoid file.Stat()
2021-02-20 20:06:06 -08:00
Chris Lu
a2383b3b12
Revert "Revert "avoid file.Stat()""
...
This reverts commit 7ef4c24f18
.
2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18
Revert "avoid file.Stat()"
...
This reverts commit 98c93ca465
.
2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094
skip seek() when index file writes
2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465
avoid file.Stat()
...
avoid one Syscall, but did not help on performance though
2021-02-20 09:44:17 -08:00
Chris Lu
a1210d1e8d
Revert "volume: avoid file.stat(), file.seek() if possible during writes"
...
This reverts commit c78409a598
.
2021-02-19 03:56:27 -08:00
Chris Lu
c78409a598
volume: avoid file.stat(), file.seek() if possible during writes
2021-02-18 23:22:54 -08:00
Chris Lu
73958e357d
add descriptive error if no free volumes
2021-02-18 19:10:20 -08:00
bingoohuang
352ac2f271
Merge remote-tracking branch 'origin/master'
2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34
use backticks instead of double quotes to avoid escaped additionally in regex
2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009
go fmt
2021-02-17 20:57:08 -08:00
Chris Lu
7403cd43c5
only use "" for hdd to avoid two values for the same thing
2021-02-16 10:50:42 -08:00
Chris Lu
ebf320ec22
avoid divided by zero
2021-02-16 05:59:24 -08:00
Chris Lu
3fe628f04e
use hdd instead of empty string
2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab
this can compile now!!!
2021-02-16 02:47:02 -08:00
Chris Lu
7ce647f27e
support customizable disk type
2021-02-13 15:42:42 -08:00
Chris Lu
4bd8a692d8
disk type can be generic tags
2021-02-13 13:50:14 -08:00
Chris Lu
885ca34748
volume: fail fast if idx files are missing
...
fix https://github.com/chrislusf/seaweedfs/issues/1796
2021-02-11 00:44:40 -08:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
bingoohuang
9e06ef66b9
fix renaming for 5bytes
2021-02-09 09:44:28 +08:00
Chris Lu
a833021132
fix refactoring left over
2021-02-07 23:03:03 -08:00
bingoohuang
7256902fb0
fix typo offset.ToAcutalOffset to offset.ToActualOffset
2021-02-07 12:11:51 +08:00
bingoohuang
94ea3bd3a5
renaming NeedleMapType to NeedleMapKind
2021-02-07 09:00:03 +08:00
Chris Lu
cfb9342a15
avoid concurrent map updates to viper
2021-01-12 02:28:13 -08:00
Chris Lu
039ad101b1
manifest file also need to detect compression
...
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
2021-01-03 01:42:21 -08:00
Chris Lu
4f31c1bb94
go fmt
2020-12-22 02:34:08 -08:00
Chris Lu
9abf016af1
Merge branch 'master' into support_ssd_volume
2020-12-17 22:30:19 -08:00
Chris Lu
738c4fd203
fix logging when error
...
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
2020-12-17 19:37:46 -08:00
Chris Lu
daa8157fc2
Merge branch 'master' into support_ssd_volume
2020-12-17 13:05:20 -08:00
Chris Lu
e2076201d7
volume: avoid reprocessing the same volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1682
2020-12-17 13:03:39 -08:00
Chris Lu
986cbdf7d9
Revert "Merge pull request #1683 from qieqieplus/master"
...
This reverts commit 8cb67952db
, reversing
changes made to 200e56215a
.
2020-12-17 12:46:42 -08:00
Chris Lu
f56e6d231e
Revert "Merge pull request #1683 from qieqieplus/master"
...
This reverts commit 8cb67952db
, reversing
changes made to 200e56215a
.
2020-12-17 12:46:20 -08:00
Chris Lu
1bf22c0b5b
go fmt
2020-12-16 09:14:05 -08:00
Chris Lu
1d88865869
passing disk type along
2020-12-16 09:10:14 -08:00
Chris Lu
23014b6810
Merge branch 'master' into support_ssd_volume
2020-12-16 08:26:51 -08:00
chenqieqie
45f902a9b7
smaller critical section
2020-12-16 20:23:51 +08:00
chenqieqie
4e58a4f24e
fix race condition when loading volumes concurrently
2020-12-16 18:49:10 +08:00
Chris Lu
2e8dba571b
adjust volume server UI
2020-12-14 00:51:57 -08:00
Chris Lu
933f2f4cfd
find location with matching disk type
2020-12-13 23:39:00 -08:00
Chris Lu
94525aa0fd
allocate volume by disk type
2020-12-13 23:08:21 -08:00
Chris Lu
f6a419c26c
disk type configurable for each folder
2020-12-13 22:49:56 -08:00
Chris Lu
0d2ec832e2
rename from volumeType to diskType
2020-12-13 11:59:32 -08:00
Chris Lu
d156c74ec0
volume server set volume type and heartbeat to the master
2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
ae655033ac
adjust logging
2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d
use append time instead of filer's own modification time
...
fix https://github.com/chrislusf/seaweedfs/issues/1669
2020-12-11 16:55:18 -08:00
Chris Lu
d171d9f988
volume: ensure the volume dat and idx files are always closed
...
fix https://github.com/chrislusf/seaweedfs/issues/1646
2020-12-04 21:50:26 -08:00
Chris Lu
cc839f935d
better unit detection
2020-12-03 19:23:59 -08:00
Chris Lu
b219ccfe68
Update volume_ttl_test.go
2020-12-03 12:55:44 -08:00
Chris Lu
a83b8e7b01
ttl fix: 3y was converted into 96m
2020-12-03 10:53:29 -08:00
Chris Lu
f3bb645018
file open error
2020-12-01 23:37:49 -08:00
Chris Lu
dc0bc48257
return file open error
2020-12-01 23:36:49 -08:00