gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
8104 commits 28 branches 278 tags 135 MiB
Commit graph

713 commits

Author SHA1 Message Date
Chris Lu c857cc7286 cloud tier: remove tagging since not all s3 vendors support this 2021-10-29 12:39:19 -07:00
Chris Lu d04cdcf40d s3 header add user agent 2021-10-29 12:28:24 -07:00
Chris Lu 3be3c17f59 volume vacuum: avoid timeout with streaming progress report 
fix https://github.com/chrislusf/seaweedfs/issues/2396
 2021-10-24 01:55:34 -07:00
Eng Zer Jun a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu f0b928ff5e go fmt 2021-10-11 23:23:46 -07:00
Chris Lu 3afa451cdc volume: find a non-empty offset when binary searching by timestamp 2021-10-11 22:00:41 -07:00
Chris Lu b530f12327 volume: find a non-empty offset when binary searching by timestamp 
fix https://github.com/chrislusf/seaweedfs/issues/2364
 2021-10-11 16:11:50 -07:00
Chris Lu ce1efeb0eb adds more error message 2021-10-09 04:54:14 -07:00
Chris Lu d688e10ed1 do not try to compress if not sure about the file content 2021-10-07 13:29:00 -07:00
Chris Lu 332f5ad3a8 revert temporary changes that disabled compression 
fix https://github.com/chrislusf/seaweedfs/issues/2362
 2021-10-07 13:24:16 -07:00
Chris Lu f0d1e7bd05 skip ec volumes when loading normal volumes 2021-10-05 02:31:44 -07:00
Chris Lu f58ea6a2ee add source name to error message 2021-10-01 02:19:30 -07:00
Chris Lu 2e9372dcf7 volume stream read skips deleted content 2021-09-27 03:07:44 -07:00
Chris Lu 5956a8b05a adjust comment 2021-09-27 01:58:03 -07:00
Chris Lu e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu 8c6ff55226 add volume not found error type, to reduce error log 2021-09-11 14:26:41 -07:00
Chris Lu 1b942dfa3c add error message 2021-09-02 12:32:46 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu f811fd0903 fix build 2021-08-30 02:03:08 -07:00
Chris Lu 5adfdd7982 change to a new bloom filter library 2021-08-29 21:26:38 -07:00
Chris Lu 05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
Chris Lu c08ac536ed cloud drive: add support for Wasabi 
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
 2021-08-25 17:34:29 -07:00
Chris Lu 00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu f0cc130849 do not force path style for better compatibility 2021-08-23 03:09:41 -07:00
Konstantin Lebedev 865b06b7fe The IEEE assembler optimizations has been submitted and will be part of the Go 1.6 standard library. 
https://github.com/klauspost/crc32
 2021-08-17 13:06:48 +05:00
Chris Lu 78e8ddf910 Only when tailing volume, the zero-ed cookie should skip checking. 
This only happens when checkCookie == false and fsync == false.
 2021-08-13 02:09:35 -07:00
Chris Lu a8617c1a39 tail volume: fix zero cookie problem from batch deletion 2021-08-13 01:54:35 -07:00
Chris Lu 734c980040 volume: support concurrent download data size limit 2021-08-08 23:25:16 -07:00
Chris Lu 9cc84a910f volume: deletion can skip volume size checking 
fix https://github.com/chrislusf/seaweedfs/issues/2225
 2021-08-01 00:32:51 -07:00
Chris Lu 49c66e88a0 volume: change all writes to fsync during graceful stopping 
fix https://github.com/chrislusf/seaweedfs/issues/2193
 2021-07-13 01:29:57 -07:00
Chris Lu b624090398 go fmt 2021-07-01 01:21:14 -07:00
Chris Lu 24e11d1e90 look back when adding to sorted values 
look back when adding to sorted values, before adding it to overflow
 2021-06-28 22:46:49 -07:00
Chris Lu fc8dd58aea volume: large_volume version has bug when using in memory index 
fix https://github.com/chrislusf/seaweedfs/issues/2162
 2021-06-28 15:48:07 -07:00
Chris Lu 56eb522b13 fix stats when a collection is deleted 2021-06-15 21:11:31 -07:00
Chris Lu 6c82326575 use bytes.Buffer to reduce memory allocation and gc 2021-06-06 13:42:36 -07:00
Konstantin Lebedev 99ef280c7c avoid data loss after restarting a container with a volum server 2021-06-02 17:07:19 +05:00
Chris Lu 1aa7e99a89 skip file not found error when deleting 2021-05-15 09:37:39 -07:00
qieqieplus ac26080bd2 fix concurrent vacuum & delete panic 2021-05-05 17:54:50 +08:00
bingoohuang cf552417a7 minFreeSpace refactored 2021-04-27 10:37:24 +08:00
bingoohuang 31f1cdeac2 minFreeSpace argument allows size like 10GiB 2021-04-26 18:48:34 +08:00
Chris Lu ca998328c2 do not add new volumes when below minFreeSpacePercent 
fix https://github.com/chrislusf/seaweedfs/issues/2017
 2021-04-21 23:11:11 -07:00
Chris Lu a8114da02d avoid thundering herd effect 
transient errors may cause thundering herd effect to all trying to recover from remove ec shards
 2021-04-21 10:17:12 -07:00
Chris Lu ae74d8f02a fix error message 
related to https://github.com/chrislusf/seaweedfs/issues/2012
 2021-04-21 01:40:16 -07:00
Chris Lu 283d703d50 adjust text 2021-04-15 11:29:58 -07:00
Chris Lu 8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu 102a951377 refactor, split into 2 files 2021-03-21 13:05:53 -07:00
Chris Lu f6e2566ee5 better logs for cookie mismatch 2021-03-20 12:02:57 -07:00
Chris Lu 35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu cb423312a4 prevent nil volume message 2021-03-13 11:05:29 -08:00
Chris Lu 972327f966 prevent nil volume nm 2021-03-13 11:04:51 -08:00
Chris Lu 737bde5ab7 refactoring, remove duplicated code 2021-03-09 23:23:01 -08:00
Chris Lu 828f6e9f4d volume: auto add missing vif files 
fix https://github.com/chrislusf/seaweedfs/issues/1878
 2021-03-09 12:09:32 -08:00
qieqieplus 5b16820924 fix: collection stats won't update if all volumes expired at same iteration 2021-03-09 18:49:45 +08:00
Chris Lu 726edab054 avoid nil when closing an index 
fix https://github.com/chrislusf/seaweedfs/issues/1870
 2021-03-07 11:03:09 -08:00
Chris Lu bdfed16d42 avoid nil exception 
fix https://github.com/chrislusf/seaweedfs/issues/1869
 2021-03-07 01:49:06 -08:00
Chris Lu e9eddfb8a2 remove wrong writes 2021-03-06 14:26:27 -08:00
Chris Lu 38fc200e56 CRCWriter consistent with CRC 2021-03-06 14:26:27 -08:00
Chris Lu 400de380f4 volume server: support tcp direct put/get/delete 2021-03-05 02:29:38 -08:00
Chris Lu c46d16b2f3 skip network error when remote volumes start 2021-03-01 01:20:06 -08:00
Chris Lu 6e43e8ce17 add logs for remote file loading 2021-03-01 00:48:59 -08:00
Chris Lu 015d16f43f add vif file versions in case loading superblock fails 2021-03-01 00:48:30 -08:00
Chris Lu d680676d45 skip already loaded backends 2021-03-01 00:47:03 -08:00
Chris Lu 540441fd38 go fmt 2021-02-28 20:34:14 -08:00
Chris Lu 2cca07b44c cloud tier to non-AWS s3 gateways 2021-02-28 18:59:09 -08:00
Chris Lu 30b30b8fe0 volume.tier.move: passing non-empty disk type 2021-02-22 01:59:03 -08:00
Chris Lu 03c643aa6b fix test 
fileSize need to be divided by 8
 2021-02-20 21:19:21 -08:00
Chris Lu 7635f6b9fa disk file avoid file.Stat() 2021-02-20 20:06:06 -08:00
Chris Lu a2383b3b12 Revert "Revert "avoid file.Stat()"" 
This reverts commit 7ef4c24f18.
 2021-02-20 12:42:09 -08:00
Chris Lu 7ef4c24f18 Revert "avoid file.Stat()" 
This reverts commit 98c93ca465.
 2021-02-20 12:39:33 -08:00
Chris Lu 1ad3200094 skip seek() when index file writes 2021-02-20 12:39:25 -08:00
Chris Lu 98c93ca465 avoid file.Stat() 
avoid one Syscall, but did not help on performance though
 2021-02-20 09:44:17 -08:00
Chris Lu a1210d1e8d Revert "volume: avoid file.stat(), file.seek() if possible during writes" 
This reverts commit c78409a598.
 2021-02-19 03:56:27 -08:00
Chris Lu c78409a598 volume: avoid file.stat(), file.seek() if possible during writes 2021-02-18 23:22:54 -08:00
Chris Lu 73958e357d add descriptive error if no free volumes 2021-02-18 19:10:20 -08:00
bingoohuang 352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu 3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu 7403cd43c5 only use "" for hdd to avoid two values for the same thing 2021-02-16 10:50:42 -08:00
Chris Lu ebf320ec22 avoid divided by zero 2021-02-16 05:59:24 -08:00
Chris Lu 3fe628f04e use hdd instead of empty string 2021-02-16 03:03:00 -08:00
Chris Lu f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu 7ce647f27e support customizable disk type 2021-02-13 15:42:42 -08:00
Chris Lu 4bd8a692d8 disk type can be generic tags 2021-02-13 13:50:14 -08:00
Chris Lu 885ca34748 volume: fail fast if idx files are missing 
fix https://github.com/chrislusf/seaweedfs/issues/1796
 2021-02-11 00:44:40 -08:00
Chris Lu 821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
bingoohuang 9e06ef66b9 fix renaming for 5bytes 2021-02-09 09:44:28 +08:00
Chris Lu a833021132 fix refactoring left over 2021-02-07 23:03:03 -08:00
bingoohuang 7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
bingoohuang 94ea3bd3a5 renaming NeedleMapType to NeedleMapKind 2021-02-07 09:00:03 +08:00
Chris Lu cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu 039ad101b1 manifest file also need to detect compression 
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
 2021-01-03 01:42:21 -08:00
Chris Lu 4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu 9abf016af1 Merge branch 'master' into support_ssd_volume 2020-12-17 22:30:19 -08:00
Chris Lu 738c4fd203 fix logging when error 
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
 2020-12-17 19:37:46 -08:00
Chris Lu daa8157fc2 Merge branch 'master' into support_ssd_volume 2020-12-17 13:05:20 -08:00
Chris Lu e2076201d7 volume: avoid reprocessing the same volume 
fix https://github.com/chrislusf/seaweedfs/issues/1682
 2020-12-17 13:03:39 -08:00
Chris Lu 986cbdf7d9 Revert "Merge pull request #1683 from qieqieplus/master" 
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
 2020-12-17 12:46:42 -08:00
Chris Lu f56e6d231e Revert "Merge pull request #1683 from qieqieplus/master" 
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
 2020-12-17 12:46:20 -08:00
Chris Lu 1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00