gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6878 commits 28 branches 278 tags 135 MiB
Commit graph

668 commits

Author SHA1 Message Date
Chris Lu daa8157fc2 Merge branch 'master' into support_ssd_volume 2020-12-17 13:05:20 -08:00
Chris Lu e2076201d7 volume: avoid reprocessing the same volume 
fix https://github.com/chrislusf/seaweedfs/issues/1682
 2020-12-17 13:03:39 -08:00
Chris Lu 986cbdf7d9 Revert "Merge pull request #1683 from qieqieplus/master" 
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
 2020-12-17 12:46:42 -08:00
Chris Lu f56e6d231e Revert "Merge pull request #1683 from qieqieplus/master" 
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
 2020-12-17 12:46:20 -08:00
Chris Lu 1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu 1d88865869 passing disk type along 2020-12-16 09:10:14 -08:00
Chris Lu 23014b6810 Merge branch 'master' into support_ssd_volume 2020-12-16 08:26:51 -08:00
chenqieqie 45f902a9b7 smaller critical section 2020-12-16 20:23:51 +08:00
chenqieqie 4e58a4f24e fix race condition when loading volumes concurrently 2020-12-16 18:49:10 +08:00
Chris Lu 2e8dba571b adjust volume server UI 2020-12-14 00:51:57 -08:00
Chris Lu 933f2f4cfd find location with matching disk type 2020-12-13 23:39:00 -08:00
Chris Lu 94525aa0fd allocate volume by disk type 2020-12-13 23:08:21 -08:00
Chris Lu f6a419c26c disk type configurable for each folder 2020-12-13 22:49:56 -08:00
Chris Lu 0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu ae655033ac adjust logging 2020-12-11 16:57:53 -08:00
Chris Lu d986c7196d use append time instead of filer's own modification time 
fix https://github.com/chrislusf/seaweedfs/issues/1669
 2020-12-11 16:55:18 -08:00
Chris Lu d171d9f988 volume: ensure the volume dat and idx files are always closed 
fix https://github.com/chrislusf/seaweedfs/issues/1646
 2020-12-04 21:50:26 -08:00
Chris Lu cc839f935d better unit detection 2020-12-03 19:23:59 -08:00
Chris Lu b219ccfe68 Update volume_ttl_test.go 2020-12-03 12:55:44 -08:00
Chris Lu a83b8e7b01 ttl fix: 3y was converted into 96m 2020-12-03 10:53:29 -08:00
Chris Lu f3bb645018 file open error 2020-12-01 23:37:49 -08:00
Chris Lu dc0bc48257 return file open error 2020-12-01 23:36:49 -08:00
Chris Lu 96c48bc8a8 fix test 2020-11-28 03:03:43 -08:00
Chris Lu 2c913dde04 volume: detect and drop volumes with disk IO error 
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
 2020-11-28 00:09:29 -08:00
Chris Lu 9ac4935f22 read from volume index file directly instead of open a separate file 
fix https://github.com/chrislusf/seaweedfs/issues/1640

read from volume index file directly instead of open a separate file,
to ensure reading latest index entries.
 2020-11-27 16:18:48 -08:00
Chris Lu 6d30b21b10 volume: add "-dir.idx" option for separate index storage 
fix https://github.com/chrislusf/seaweedfs/issues/1265
 2020-11-27 03:17:10 -08:00
Chris Lu 3c229eb677 refactor 2020-11-26 17:21:55 -08:00
Chris Lu c0fb4a3766 add comments 2020-11-26 15:22:42 -08:00
Chris Lu 6783ae5f8f rename 2020-11-26 15:19:43 -08:00
Chris Lu 97f706f35b just refactoring 2020-11-26 15:08:05 -08:00
Chris Lu b11449f955 pass in dir.idx parameter to DiskLocation 2020-11-26 14:59:03 -08:00
Chris Lu 0da7ecfd29 go fmt 2020-11-26 11:22:30 -08:00
Chris Lu 92f906b6fc remove zstd 
fix https://github.com/chrislusf/seaweedfs/issues/1629
 2020-11-21 13:06:45 -08:00
Chris Lu 1dd3a6ac36 avoid verbose logs 2020-11-09 15:56:11 -08:00
Chris Lu 0dafcf1f5a volume: detect max volume count changes based on disk usage 
fix https://github.com/chrislusf/seaweedfs/issues/1594
 2020-11-03 14:43:17 -08:00
Chris Lu de86945aeb go fmt 2020-10-31 16:45:38 -07:00
Chris Lu 9708df47d1 ensure offset in the index file is positive 2020-10-31 15:50:01 -07:00
Konstantin Lebedev 46303c36bf When the volume server is stopped, the master server immediately sees the deletion of volumes 2020-10-31 02:48:25 +05:00
Chris Lu 6560ac6466 volume loading: trim out unreachable idx file content 
fix https://github.com/chrislusf/seaweedfs/issues/1583
 2020-10-29 22:25:23 -07:00
Konstantin Lebedev 1ad1b8c4f6 collection Volume ReadOnly Count with detailed status 2020-10-28 17:38:26 +05:00
Chris Lu 744c5594da volume: automatically trim out unreachable entries 2020-10-28 01:14:39 -07:00
Chris Lu 3e925faddd trim out name extension 2020-10-28 01:14:05 -07:00
shibinbin a67eb1afa1 fix: restart volumeserver sometime occurred out of memory error 2020-10-28 12:03:06 +08:00
Chris Lu 53c3aad875 volume: add a note file to avoid incomplete volume files 
fix https://github.com/chrislusf/seaweedfs/issues/1567
 2020-10-27 15:56:49 -07:00
Chris Lu 06c15ab35c volume: add special handling for .dat larger than 32GB 2020-10-27 13:11:56 -07:00
Konstantin Lebedev 2fb1fce8a0 avoid old values when missing read only flags 2020-10-25 16:21:40 +05:00
Chris Lu 9104cfa744 reduce locks 2020-10-24 19:40:35 -07:00
Chris Lu e9d40b80b2 less verbose logs 2020-10-24 09:42:54 -07:00
Chris Lu 575d7952a1 add available resource stats 
fix https://github.com/chrislusf/seaweedfs/issues/1555
 2020-10-22 09:13:47 -07:00
Chris Lu 4ff2ceee33 UI fix on rendering EC volumes 
addressing UI problem with https://github.com/chrislusf/seaweedfs/issues/1551
 2020-10-21 22:05:58 -07:00
Konstantin Lebedev dc2e13092d add number of read only volumes metric 2020-10-15 15:48:40 +05:00
Chris Lu bd8b9b0c2e some changes to prepare fix after failed reads 2020-10-08 18:33:06 -07:00
Chris Lu d522df0ecd adjust log level 2020-10-04 01:35:32 -07:00
Chris Lu a1c01d716b volume: avoid deadlock when deleting volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1501
 2020-10-01 07:10:03 -07:00
Chris Lu 103fafe00b weed export: print out [start,stop) content range 2020-09-14 22:57:23 -07:00
Chris Lu f2723c1bc8 do not idx file format 
revert c9ab8d05fa
 2020-09-12 12:42:36 -07:00
Chris Lu 446e476a11 go fmt 2020-09-12 04:08:03 -07:00
Chris Lu ea26a98753 volume: validate volume correctness if last entry is a deletion 2020-09-12 04:07:04 -07:00
Chris Lu 9d4bdfcfdf fix volume integrity checking 2020-09-11 11:34:10 -07:00
Chris Lu 401ccf1509 fix test 2020-09-10 15:13:23 -07:00
Chris Lu c9ab8d05fa fixes for reading deleted fid 2020-09-10 14:42:52 -07:00
James Hartig 8e54e34576 volume: Don't unmount before deleting volume in copy 
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
 2020-09-01 22:00:07 -04:00
Chris Lu 38e06d783d volume: check disk space before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1440
 2020-08-31 18:10:53 -07:00
Chris Lu dbb10e0f0b testing with more than 30days 2020-08-31 17:59:29 -07:00
Chris Lu 99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu ab759f0ec2 erasure coding: fix EC error if multiple disks are configured in one volume server 2020-08-26 09:16:58 -07:00
Chris Lu d1cf39f180 fix logging 2020-08-26 08:55:15 -07:00
James Hartig 3ccfa4c6ad Added VolumeMarkWritable and VolumeStatus grpc methods 
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
 2020-08-19 11:42:56 -04:00
Chris Lu 3b4b1d4a77 fix tests 2020-08-19 01:37:56 -07:00
Chris Lu ed4b43b419 adjust logs 2020-08-18 23:42:09 -07:00
Chris Lu fe01191b5b support read option readDeleted=true 2020-08-18 19:22:16 -07:00
Chris Lu 6ccd7f0a4d refactoring 2020-08-18 18:01:37 -07:00
Chris Lu c026eb0592 refactoring 2020-08-18 17:39:29 -07:00
Chris Lu 7e91ae592c pass in option to read deleted entries 
not working yet
 2020-08-18 17:37:26 -07:00
Chris Lu ee11d98650 refactoring 2020-08-18 17:35:19 -07:00
Chris Lu 6a92f0bc7a refactoring to typed Size 
Go is amazing with refactoring!
 2020-08-18 17:04:28 -07:00
Chris Lu 627b081b67 adjust logs 2020-08-16 16:32:22 -07:00
Chris Lu c03bb180eb fix error reporting 2020-08-14 00:44:02 -07:00
Chris Lu ab6e5c0dc4 adjust error message 2020-08-08 10:18:32 -07:00
Chris Lu 20e2ac1add filer: store md5 metadata for files uploaded by filer 
fix https://github.com/chrislusf/seaweedfs/issues/1412
 2020-08-06 10:04:17 -07:00
Chris Lu 4ecfa9879d volume: report Content-MD5 in response header 2020-08-06 05:22:53 -07:00
Chris Lu cbd80253e3 better needle id format 2020-08-04 22:09:07 -07:00
Chris Lu f89186a3cf better logging 2020-08-01 11:19:57 -07:00
Chris Lu 943fa4ea0c fix compilation error 2020-07-22 16:11:17 -07:00
Chris Lu 49446af516 simplify 2020-07-22 13:43:54 -07:00
James Hartig 0be3bb4a87 idx: Switch WalkIndexFile to accept generic io.ReaderAt 
This is necessary to use it with in-memory idx files.
 2020-07-22 15:01:16 -04:00
Chris Lu 97d97f3528 go code can read and write chunk manifest 2020-07-19 17:59:43 -07:00
Chris Lu f43146b237 resolve directories if containing home directory 2020-07-16 22:50:14 -07:00
Chris Lu ddec7b2bb9 go fmt 2020-07-14 07:34:16 -07:00
Chris Lu f2e8ad6241 refactoring 2020-07-11 06:35:54 -07:00
李远军 6608cb5f43 Revert "remove fixJpgOrientation" 
This reverts commit de5ca9b2
 2020-07-10 10:08:36 +08:00
Chris Lu 4bed1663e2 adjust log level 2020-07-07 23:17:31 -07:00
Chris Lu 4d1484628a refactoring 2020-07-03 16:41:30 -07:00
Chris Lu faa5c2e89a refactoring 2020-07-03 16:34:31 -07:00
Chris Lu 20d9cfae86 volume: following md5 base64 encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1374
 2020-06-28 07:25:50 -07:00
Chris Lu 212c64241b checking ecx file 2020-06-25 16:36:06 -07:00
Chris Lu 9fd7cdadf1 fix 2020-06-25 10:45:34 -07:00
Chris Lu 3b638d3994 add more ec encoding logging 2020-06-25 09:43:38 -07:00
Chris Lu 32a1f22730 more logs 2020-06-24 22:07:53 -07:00
Chris Lu 2ff37ccdbd testing parsing the upload 
The mime type is always the value passed in.

Compress or not depends on the content detection, file name extension, and compression ratio.

If the content is already compressed, need to know the content size.
 2020-06-24 11:39:12 -07:00
Chris Lu c21f4ebfee
Merge pull request #1371 from Kimbsen/content_md5_validation 
Optional md5 validation of uploads
 2020-06-24 11:33:44 -07:00
Kimbsen 4eeab2a379 remove unused imports 2020-06-24 13:37:00 +02:00
Kimbsen ffddecebef do md5 validation AFTER decompression 2020-06-24 13:35:13 +02:00
Chris Lu fe60db404a prepare for zstd 2020-06-23 09:12:02 -07:00
Kimbsen 6b1e93ba0b check request header not part header for content-encoding 2020-06-23 17:04:24 +02:00
Kimbsen 38626cb584 don't forget the last few bytes 2020-06-23 16:53:49 +02:00
Kimbsen 8d1f6cd567 added optional md5 verification of uploaded data 2020-06-23 16:24:11 +02:00
Chris Lu de5ca9b258 remove fixJpgOrientation 2020-06-20 12:50:40 -07:00
Chris Lu 3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu 16fe132a20 detect mime type before replicating to other volume servers 2020-06-19 22:11:36 -07:00
Chris Lu ae1994cbc1 erasure coding: fix cases where there are no .ecj files 2020-06-18 09:52:35 -07:00
Chris Lu 7ccc31853c hard limit volume file size 2020-06-16 00:27:48 -07:00
Chris Lu bdbe5ddbe9 volume server: add a fix for dead lock 
https://github.com/chrislusf/seaweedfs/issues/1353
 2020-06-08 08:12:59 -07:00
Chris Lu 45a9d852d3 go fmt 2020-06-05 15:27:10 -07:00
Evgenii Kozlov 2410ffbd79 Name fix 2020-06-05 19:00:38 +03:00
Evgenii Kozlov 0e0db70f55 Set volumes ReadOnly if low free disk space 2020-06-05 18:18:15 +03:00
Chris Lu ef2b3a0801 leveldb recover error handling 2020-05-26 00:03:44 -07:00
Chris Lu e5a0787653 filer: leveldb store add auto recovery, report correct error 
fix https://github.com/chrislusf/seaweedfs/issues/1326
 2020-05-22 10:54:42 -07:00
Chris Lu 7ff2b0121d specify constant data type to uint64 instead of default int 
fix https://github.com/chrislusf/seaweedfs/issues/1324
 2020-05-19 07:55:23 -07:00
Chris Lu 11b33c0760 fix compilation problem 2020-05-18 10:53:31 -07:00
zhangsong b161baa3f0 fix volume vacuum test unit 2020-05-07 19:58:06 +08:00
zhangsong 3703c7e176 write request keep compatible with before 2020-05-06 21:35:32 +08:00
zhangsong f9e8702bb4 use async write to persistent file to disk - part1 2020-05-04 17:39:44 +08:00
Chris Lu 6190fd665d printout error 2020-04-27 12:41:31 -07:00
Chris Lu bafa95045b volume: deletion checks all disk locations 
fix https://github.com/chrislusf/seaweedfs/issues/1283
 2020-04-21 14:49:58 -07:00
Chris Lu 7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu c8ca234773 refactoring 2020-04-11 14:27:25 -07:00
Chris Lu df9d538044 rename function 2020-04-11 14:19:44 -07:00
Chris Lu 9fa065f600 typo in logs 2020-04-09 23:43:09 -07:00
Chris Lu e63a79ade8 better handle lock in case of exception 2020-03-24 18:41:25 -07:00
Chris Lu 3137777d83 volume: automatically detect max volume count 2020-03-22 16:21:42 -07:00
Chris Lu cbfe31a9a8 idx file sync before compaction 2020-03-20 23:38:46 -07:00
Chris Lu 81797a059a volume: sync volume file right before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1237
 2020-03-19 23:54:52 -07:00
Chris Lu 709f231e23 tiered storage: add s3 endpoint for private s3 implementation 
fix https://github.com/chrislusf/seaweedfs/issues/1238
 2020-03-19 21:13:56 -07:00
Chris Lu b964bbab3d fix compilation 2020-03-17 10:01:24 -07:00
Chris Lu c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu 22400c6633 consistent gzip logic 
local store gzip same as replicated writes
 2020-03-15 04:15:40 -07:00
Chris Lu 560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu d439d83772 volume: follow compactionBytePerSecond 
related to https://github.com/chrislusf/seaweedfs/issues/1108
 2020-03-11 10:32:17 -07:00
Chris Lu d3e1ef64b3 ensure error checking on windows 2020-03-10 13:33:26 -07:00
Chris Lu 88a110e67e release file handle 2020-03-10 13:32:22 -07:00
Chris Lu 43e6261579 volume: fix memory leak 2020-03-10 13:31:23 -07:00
Chris Lu 0871d2cff0 volume: fix memory leak during compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1222
 2020-03-09 22:29:02 -07:00
Chris Lu 89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu 9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu 1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00