gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
2747 commits 28 branches 278 tags 135 MiB
Commit graph

343 commits

Author SHA1 Message Date
Chris Lu 2f7710a75d debug sporadic test error on travis 2019-06-04 02:09:06 -07:00
Chris Lu 11cffb3168 fix ec.rebuild bugs 2019-06-03 11:50:54 -07:00
Chris Lu 7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu 133b772fb5 destroy ec volume if it is empty 2019-06-01 01:51:28 -07:00
Chris Lu ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu f919d0235c ec encode volumes quiet for a period of time 2019-05-31 23:41:17 -07:00
Chris Lu de1a35acdd conditionally fresh the shard locations 2019-05-31 02:22:12 -07:00
Chris Lu 47f1901843 ask for the ec volume version 2019-05-31 00:58:51 -07:00
Chris Lu 689930f092 forget shards that are broken 2019-05-31 00:19:13 -07:00
Chris Lu aaab2c25f5 resolve concurrent modification problem 
n and err can be modified concurrently
 2019-05-30 13:59:07 -07:00
Chris Lu 40ca2f2903 add collection.delete 2019-05-30 09:47:54 -07:00
Chris Lu 5c6c757619 clean up test files 2019-05-29 00:34:30 -07:00
Chris Lu 5dd67f9acf reading by recover from other shards 2019-05-28 23:48:39 -07:00
Chris Lu 3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu 302d9fbc6d adjust error message for file not found 2019-05-28 14:27:21 -07:00
Chris Lu 4f76342cbc WIP 
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
 2019-05-28 00:51:01 -07:00
Chris Lu 2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu 713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu 3a8c1055a2 refactoring ecx to ecVolume 2019-05-27 22:00:36 -07:00
Chris Lu 217cde0a3b refactoring 2019-05-27 21:40:51 -07:00
Chris Lu 03b9291e5d volum server passes grpc option and master location to store 2019-05-27 21:22:23 -07:00
Chris Lu b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu a4f3d82c57 convert needle id to ec intervals to read from 2019-05-27 01:29:46 -07:00
Chris Lu a463759edf adjust error message 2019-05-26 01:14:42 -07:00
Chris Lu af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu 6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu 228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu 8a96445f40 register ec shards to each data node 2019-05-23 22:51:18 -07:00
Chris Lu 4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu 7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Chris Lu 17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu 693d6c0f1b clean up tests 2019-05-19 21:16:01 -07:00
Chris Lu 6386a3174b able to validate by randomly selected ec files 2019-05-19 14:24:33 -07:00
Chris Lu 7c2c60c376 add locating data inside the ec files 2019-05-19 03:01:58 -07:00
Chris Lu 87f63b9c08 generate ec01~ec14, generate ecx file with sorted needle values 2019-05-18 22:46:24 -07:00
Chris Lu 12dc6608f0 Merge branch 'master' into erasure_coding 2019-05-18 11:16:07 -07:00
Chris Lu 76cf57a0e1 volume: optimzation for leveldb 2019-05-17 17:34:03 -07:00
Chris Lu 8156958ee9 move function to make travis happy 2019-05-15 10:02:44 -07:00
Chris Lu 0a36f628c6 testing RS coding 2019-05-15 01:02:00 -07:00
Chris Lu cf58fc0e63 refactor: extract out the write throttler 2019-05-06 13:56:08 -07:00
Chris Lu 46a675ecc2 align memory to avoid nil with atomic operations 2019-05-04 21:33:05 -07:00
Chris Lu 177f3add81 adjust memory layout to avoid possible nil pointer 2019-05-04 08:47:11 -07:00
Chris Lu 4b27f892e6 remove 128 bit needle id build option 
fix https://github.com/chrislusf/seaweedfs/issues/949
 2019-05-03 22:39:00 -07:00
Chris Lu b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Wine93 32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu 9beea63960 memory alignment 
fix https://github.com/chrislusf/seaweedfs/issues/939
 2019-04-24 09:23:19 -07:00
Chris Lu 53e2e73de2 fix metrics 
fix https://github.com/chrislusf/seaweedfs/issues/935
 2019-04-22 12:32:10 -07:00
Chris Lu 440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu 0302b9496c volume: also check cookie for possible duplicates 2019-04-21 13:31:45 -07:00
Chris Lu 132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu 316bd27f75 add remove volumes with version info 2019-04-20 23:53:37 -07:00
Chris Lu 6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
stlpmo-jn 2200ea9cb9 add function ParseFileIdFromString 2019-04-20 18:39:06 +08:00
Chris Lu 3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu 0be2d51c96 read volume lastAppendAtNs when loading a volume 2019-04-19 01:56:38 -07:00
Chris Lu ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu 33c92b819a refactoring 2019-04-18 21:17:43 -07:00
Chris Lu fa176fe80f volume tailing chunks large files 2019-04-18 19:22:13 -07:00
Chris Lu b142f9f1d5 go fmt 2019-04-18 00:19:18 -07:00
Chris Lu b09e8dbb37 add VolumeStreamFollow, but not used yet 2019-04-18 00:18:29 -07:00
Chris Lu 13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu f567777d64 volume: fix needle checking for deleted needles 2019-04-15 21:58:43 -07:00
Chris Lu bd76ad0ff1 Revert "Revert "volume: close request body if any parsing error"" 
This reverts commit 359b7c1650.
 2019-04-15 21:43:28 -07:00
Chris Lu 359b7c1650 Revert "volume: close request body if any parsing error" 
This reverts commit 5996656141.
 2019-04-15 21:35:12 -07:00
Chris Lu 5996656141 volume: close request body if any parsing error 2019-04-15 21:11:45 -07:00
Chris Lu 3e8a3a8fec fix race detector found problems 2019-04-14 23:00:37 -07:00
Chris Lu 0e33272f56 small refactoring 2019-04-10 23:39:53 -07:00
stlpmo-jn c1a0403da2 repair the error replications of the volume 2019-04-10 19:41:55 +08:00
Chris Lu 3555628ad5 weed volume: fix bug, remove boltdb, btree 2019-04-09 10:08:59 -07:00
Chris Lu 8a5ce16e96 weed volume: remove boltdb, btree options, add options for leveldb medium, large 2019-04-09 09:42:06 -07:00
Chris Lu 9924fa3b1a weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower lookup
 2019-04-09 00:24:32 -07:00
Chris Lu 784c5bb73a add build option support 5-byte offset 2019-04-08 22:01:29 -07:00
Chris Lu 7a4b234ea1 divide offset into higher and lower sections 2019-04-08 21:44:06 -07:00
Chris Lu 000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu 94302935b2 weed volume: only store compressed data if compression is effective 2019-04-06 00:10:52 -07:00
Chris Lu 7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu 19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu 70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu 95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu cea5c53bf7 fix length checking 
fix https://github.com/chrislusf/seaweedfs/issues/890
 2019-03-19 20:56:27 -07:00
Chris Lu da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu 916b809c08 add error checking 2019-03-19 05:34:43 -07:00
Chris Lu ece9d13312 volume info collect compact revision number 2019-03-18 09:32:21 -07:00
Chris Lu 44647a46c0 needle scanner read appendAtNs 2019-03-17 23:28:43 -07:00
Chris Lu 8b26d15740 refactoring: simplify function parameter 2019-03-17 22:32:01 -07:00
Chris Lu 2fcc88116e go fmt 2019-03-17 20:27:36 -07:00
Chris Lu aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu 95ef4513c8 print out memory size 2019-03-13 23:14:40 -07:00
Chris Lu 346541a101 print out per entry memory usage 2019-03-13 23:07:24 -07:00
chenwanli fd27ed7755 Fix https://github.com/chrislusf/seaweedfs/issues/861 2019-02-26 17:12:39 +08:00
Chris Lu 77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu 09471b46f9 fix related tests 2019-01-26 00:10:29 -06:00
Chris Lu 5668ed798d memory needle map mark size to be TombstoneFileSize 
fix https://github.com/chrislusf/seaweedfs/issues/850
 2019-01-26 00:05:44 -06:00
bingoohuang cf0fc797ac use tempFile.Truncate to reset file size to speedup test 2019-01-23 16:24:51 +08:00
bingoohuang ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang 6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
bingoohuang 27093bc2e5 fix needle Append return offset to avoid uint32 overflow 2019-01-16 16:48:03 +08:00
Chris Lu badd8fdf8f shortcut for empty file 2019-01-08 09:03:28 -08:00
Chris Lu 6b5d6bb5a6 adjust error names 2019-01-05 19:52:38 -08:00
Chris Lu 9383c91eb1 wait to read again if the volume is compacting 2019-01-05 19:52:17 -08:00
Chris Lu d5ebfd5b0c fix s3cmd put 
1. fix volume put data size
 2019-01-02 12:57:33 -08:00
Chris Lu 1478d7ea21 reduce file seek when writing 2018-12-31 15:08:32 -08:00
Chris Lu 0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu c043fd17cb detect more gzippable content 2018-12-22 15:05:31 -08:00
Chris Lu 852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu be946c9e54 filer store original data size instead of data size after gzip 2018-12-22 13:11:07 -08:00
Chris Lu 141d302492 fix binarySearchCompactSection 2018-12-22 11:52:58 -08:00
Chris Lu 5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu 36d13355bb compact map fix bug when iterating 2018-12-22 11:05:29 -08:00
Chris Lu f64dc7465b remove index file 
fix https://github.com/chrislusf/seaweedfs/issues/793
 2018-12-17 20:33:32 -08:00
Chris Lu 42cb9b76c4 reduce needle map memory usage by 25% 2018-12-15 05:55:56 -08:00
Chris Lu 168f461761 fix MaxUint8 2018-12-10 20:57:51 -08:00
Chris Lu 8f78a069c6 fix possible name overflow when len(name)>255 2018-12-10 20:57:01 -08:00
Chris Lu 4a490fe35a go fmt 2018-12-09 01:27:25 -08:00
Chris Lu 5741134ca0 change overflow from map to slice 2018-12-09 01:27:11 -08:00
Chris Lu d9189c857c remove unused file 2018-12-09 00:30:53 -08:00
Chris Lu df4b9df142 add memory usage report 2018-12-09 00:15:23 -08:00
Chris Lu ec5a305624 refactor 2018-12-09 00:12:37 -08:00
Chris Lu 9d3be33e5c optimize a little bit 2018-12-08 21:45:14 -08:00
Chris Lu 86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu 2697d6a4fb fix deleting bucket 2018-11-08 22:15:21 -08:00
Chris Lu a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu f9dbaf6292 delete file by name instead of referencing the dataFile pointer 2018-11-05 08:53:38 -08:00
alex8224 8d3f9c0b75 fix invalid point when delete collection 2018-11-05 19:53:55 +08:00
Chris Lu f050b22d6c close file in ScanVolumeFile() 
fix https://github.com/chrislusf/seaweedfs/issues/761
 2018-11-04 00:28:24 -07:00
Chris Lu 496a7398d1 adjust logging 2018-10-18 20:47:30 -07:00
Chris Lu eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu 66a353dcb5 remove volume server /admin/volume/delete 2018-10-15 01:26:49 -07:00
Chris Lu fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu 333709657c adjust log level 2018-10-15 00:40:12 -07:00
Chris Lu b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu a362261972 avoid possible racing issue if leveldb or boltdb is used. 
remove the to-be-generated files during commit

fix https://github.com/chrislusf/seaweedfs/issues/738
 2018-09-26 18:45:51 -07:00
Chris Lu 164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu 9b3bf0e46c fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:48:58 -07:00
Chris Lu bc8d34143d Revert "fix "weed backup" rerunning" 
This reverts commit 1937be2bb2.
 2018-09-09 02:38:30 -07:00
Chris Lu 1937be2bb2 fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:31:12 -07:00
Chris Lu 76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu ac793a3c5a clean remove all left over files when deleting a collection 2018-08-23 23:33:16 -07:00
Chris Lu d361a1ddf5 fix id String() instead 2018-08-23 22:44:30 -07:00
Chris Lu 694f93de80 parse needle id correctly 
fix https://github.com/chrislusf/seaweedfs/issues/713
 2018-08-23 20:48:43 -07:00
Chris Lu 1f030af06f adjusting startup logging 2018-08-22 13:59:04 -07:00
Chris Lu 75d63db60d randomize raft server startup 
also some go fmt
 2018-08-12 14:27:14 -07:00
Chris Lu 495a776671 support 128 bit NeedleId 2018-07-31 23:25:26 -07:00
Chris Lu 10bb76fe6b go fmt 2018-07-27 02:10:42 -07:00