gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
5595 commits 28 branches 278 tags 135 MiB
Commit graph

623 commits

Author SHA1 Message Date
Chris Lu 0e33272f56 small refactoring 2019-04-10 23:39:53 -07:00
stlpmo-jn c1a0403da2 repair the error replications of the volume 2019-04-10 19:41:55 +08:00
Chris Lu 3555628ad5 weed volume: fix bug, remove boltdb, btree 2019-04-09 10:08:59 -07:00
Chris Lu 8a5ce16e96 weed volume: remove boltdb, btree options, add options for leveldb medium, large 2019-04-09 09:42:06 -07:00
Chris Lu 9924fa3b1a weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower lookup
 2019-04-09 00:24:32 -07:00
Chris Lu 784c5bb73a add build option support 5-byte offset 2019-04-08 22:01:29 -07:00
Chris Lu 7a4b234ea1 divide offset into higher and lower sections 2019-04-08 21:44:06 -07:00
Chris Lu 000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu 94302935b2 weed volume: only store compressed data if compression is effective 2019-04-06 00:10:52 -07:00
Chris Lu 7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu 19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu 70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu 95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu cea5c53bf7 fix length checking 
fix https://github.com/chrislusf/seaweedfs/issues/890
 2019-03-19 20:56:27 -07:00
Chris Lu da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu 916b809c08 add error checking 2019-03-19 05:34:43 -07:00
Chris Lu ece9d13312 volume info collect compact revision number 2019-03-18 09:32:21 -07:00
Chris Lu 44647a46c0 needle scanner read appendAtNs 2019-03-17 23:28:43 -07:00
Chris Lu 8b26d15740 refactoring: simplify function parameter 2019-03-17 22:32:01 -07:00
Chris Lu 2fcc88116e go fmt 2019-03-17 20:27:36 -07:00
Chris Lu aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu 95ef4513c8 print out memory size 2019-03-13 23:14:40 -07:00
Chris Lu 346541a101 print out per entry memory usage 2019-03-13 23:07:24 -07:00
chenwanli fd27ed7755 Fix https://github.com/chrislusf/seaweedfs/issues/861 2019-02-26 17:12:39 +08:00
Chris Lu 77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu 09471b46f9 fix related tests 2019-01-26 00:10:29 -06:00
Chris Lu 5668ed798d memory needle map mark size to be TombstoneFileSize 
fix https://github.com/chrislusf/seaweedfs/issues/850
 2019-01-26 00:05:44 -06:00
bingoohuang cf0fc797ac use tempFile.Truncate to reset file size to speedup test 2019-01-23 16:24:51 +08:00
bingoohuang ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang 6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
bingoohuang 27093bc2e5 fix needle Append return offset to avoid uint32 overflow 2019-01-16 16:48:03 +08:00
Chris Lu badd8fdf8f shortcut for empty file 2019-01-08 09:03:28 -08:00
Chris Lu 6b5d6bb5a6 adjust error names 2019-01-05 19:52:38 -08:00
Chris Lu 9383c91eb1 wait to read again if the volume is compacting 2019-01-05 19:52:17 -08:00
Chris Lu d5ebfd5b0c fix s3cmd put 
1. fix volume put data size
 2019-01-02 12:57:33 -08:00
Chris Lu 1478d7ea21 reduce file seek when writing 2018-12-31 15:08:32 -08:00
Chris Lu 0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu c043fd17cb detect more gzippable content 2018-12-22 15:05:31 -08:00
Chris Lu 852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu be946c9e54 filer store original data size instead of data size after gzip 2018-12-22 13:11:07 -08:00
Chris Lu 141d302492 fix binarySearchCompactSection 2018-12-22 11:52:58 -08:00
Chris Lu 5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu 36d13355bb compact map fix bug when iterating 2018-12-22 11:05:29 -08:00
Chris Lu f64dc7465b remove index file 
fix https://github.com/chrislusf/seaweedfs/issues/793
 2018-12-17 20:33:32 -08:00
Chris Lu 42cb9b76c4 reduce needle map memory usage by 25% 2018-12-15 05:55:56 -08:00
Chris Lu 168f461761 fix MaxUint8 2018-12-10 20:57:51 -08:00
Chris Lu 8f78a069c6 fix possible name overflow when len(name)>255 2018-12-10 20:57:01 -08:00
Chris Lu 4a490fe35a go fmt 2018-12-09 01:27:25 -08:00
Chris Lu 5741134ca0 change overflow from map to slice 2018-12-09 01:27:11 -08:00
Chris Lu d9189c857c remove unused file 2018-12-09 00:30:53 -08:00
Chris Lu df4b9df142 add memory usage report 2018-12-09 00:15:23 -08:00
Chris Lu ec5a305624 refactor 2018-12-09 00:12:37 -08:00
Chris Lu 9d3be33e5c optimize a little bit 2018-12-08 21:45:14 -08:00
Chris Lu 86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu 2697d6a4fb fix deleting bucket 2018-11-08 22:15:21 -08:00
Chris Lu a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu f9dbaf6292 delete file by name instead of referencing the dataFile pointer 2018-11-05 08:53:38 -08:00
alex8224 8d3f9c0b75 fix invalid point when delete collection 2018-11-05 19:53:55 +08:00
Chris Lu f050b22d6c close file in ScanVolumeFile() 
fix https://github.com/chrislusf/seaweedfs/issues/761
 2018-11-04 00:28:24 -07:00
Chris Lu 496a7398d1 adjust logging 2018-10-18 20:47:30 -07:00
Chris Lu eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu 66a353dcb5 remove volume server /admin/volume/delete 2018-10-15 01:26:49 -07:00
Chris Lu fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu 333709657c adjust log level 2018-10-15 00:40:12 -07:00
Chris Lu b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu a362261972 avoid possible racing issue if leveldb or boltdb is used. 
remove the to-be-generated files during commit

fix https://github.com/chrislusf/seaweedfs/issues/738
 2018-09-26 18:45:51 -07:00
Chris Lu 164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu 9b3bf0e46c fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:48:58 -07:00
Chris Lu bc8d34143d Revert "fix "weed backup" rerunning" 
This reverts commit 1937be2bb2.
 2018-09-09 02:38:30 -07:00
Chris Lu 1937be2bb2 fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:31:12 -07:00
Chris Lu 76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu ac793a3c5a clean remove all left over files when deleting a collection 2018-08-23 23:33:16 -07:00
Chris Lu d361a1ddf5 fix id String() instead 2018-08-23 22:44:30 -07:00
Chris Lu 694f93de80 parse needle id correctly 
fix https://github.com/chrislusf/seaweedfs/issues/713
 2018-08-23 20:48:43 -07:00
Chris Lu 1f030af06f adjusting startup logging 2018-08-22 13:59:04 -07:00
Chris Lu 75d63db60d randomize raft server startup 
also some go fmt
 2018-08-12 14:27:14 -07:00
Chris Lu 495a776671 support 128 bit NeedleId 2018-07-31 23:25:26 -07:00
Chris Lu 10bb76fe6b go fmt 2018-07-27 02:10:42 -07:00
Chris Lu 4d322df95e avoid reparsing the multipart form 2018-07-27 02:10:10 -07:00
Chris Lu 57ded4fd80 speed up leveldb bolddb loading 2018-07-24 22:17:56 -07:00
Chris Lu ea8ecf35f4 set needle AppendAtNs value during writes 2018-07-24 02:44:33 -07:00
Chris Lu 77fadab560 go fmt 2018-07-24 01:38:08 -07:00
Chris Lu 4f317c7e3d working version3 2018-07-24 01:36:04 -07:00
Chris Lu 852af28f91 needle priting format 2018-07-21 17:41:21 -07:00
Chris Lu 7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu c98df05ed0 support PUT 2018-07-21 15:58:48 -07:00
Chris Lu feb8eeb830 refactor: separate multipart parsing file 2018-07-21 12:01:35 -07:00
Chris Lu 3edfe1d28f extend export command to show tombstone + change output format to CSV 
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
 2018-07-14 20:51:17 -07:00
Chris Lu a51aa4c586 avoid extra log messages 
fix https://github.com/chrislusf/seaweedfs/issues/689
 2018-07-13 22:07:32 -07:00
Chris Lu 8a12fd9ec4 add error checking when loading dat file 2018-07-09 23:31:25 -07:00
Chris Lu d0982cafa7 refactoring 2018-07-09 00:22:50 -07:00
Chris Lu 5b1fd374be fix all tests 2018-07-08 02:39:04 -07:00
Chris Lu d4d7ced922 refactoring: add type for needle id, offset 
later the type size can possibly be adjusted
 2018-07-08 02:28:04 -07:00
Chris Lu 5bfb72d058 faster loading boltdb or leveldb needle map metrics by bloomfilter 
avoid btree
 2018-07-07 00:51:17 -07:00
Chris Lu aba1fe01b3 use constant NeedleIndexSize instead of the number 16 2018-07-07 00:50:14 -07:00
Chris Lu defcd4edc5 adjust log level 2018-07-07 00:48:58 -07:00
Chris Lu 9bcaa65574 avoid divided by 0 
fix https://github.com/chrislusf/seaweedfs/issues/650
 2018-06-25 01:20:15 -07:00
Chris Lu 233a9a9b6c fix log error 2018-06-24 16:12:41 -07:00
Chris Lu 49f4216280 fix compilation error 2018-06-24 15:37:54 -07:00
Chris Lu 7f6500ae07 super block reserves a couple of bits for future extension 2018-06-24 15:23:03 -07:00
Chris Lu f339e9c284 add support for extra super block info 2018-06-24 15:19:57 -07:00
Chris Lu 9ba335a7c4 prepare for flexible super block 2018-06-24 11:37:08 -07:00
Chris Lu 3a6f9c9bcf increase test size 2018-06-23 18:30:13 -07:00
Chris Lu 7f7e4e9885 fix error for deleted files during compaction 
deletion during commit may cause trouble when make up the difference during commitCompact()
 2018-06-23 18:24:59 -07:00
Chris Lu 3699996985 add compaction test 2018-06-23 16:48:19 -07:00
Chris Lu 0467195f07 fix file closing on windows 2018-06-13 00:36:51 -07:00
Chris Lu eaf000eff1 fix error on go tip 2018-06-11 23:39:42 -07:00
Chris Lu 43e3f5724c use fixed list of masters in both filer and volume servers 2018-06-01 00:39:39 -07:00
Chris Lu 458ada173e go fmt 2018-05-27 11:52:26 -07:00
Chris Lu 1675243f29 maybe speed up a little when loading index 2018-05-23 02:38:11 -07:00
Chris Lu 69b9d8c3c2 fix boltdb variable usage 2018-05-22 10:18:09 -07:00
Chris Lu 942c2cbd7b mv pb to master_pb 2018-05-09 23:11:54 -07:00
Chris Lu 57c5405657 add todo 2018-05-09 04:18:31 -07:00
Chris Lu 6db1c8bafb Revert "adjust the needle mapper" 
This reverts commit e2c7be3a2e.
 2018-05-09 03:57:07 -07:00
Chris Lu e2c7be3a2e adjust the needle mapper 
as discussed on
https://github.com/chrislusf/seaweedfs/pull/642#issuecomment-387606960
 2018-05-09 03:50:14 -07:00
WenLong LI b9068c1304
use volumes variable 
fix leveldb mount bug
 2018-05-03 12:13:53 +08:00
liwenlong05 02a1597642 fix leveldb mount bug 2018-05-03 11:20:38 +08:00
mtolman e4b1e5c3c8
Fix for issue #635 
Fix for the following issue on master: https://github.com/chrislusf/seaweedfs/issues/635
 2018-04-17 13:26:48 -06:00
Yang Pengcheng a2112e1be0 fix ldb dir deleting 2017-08-30 22:55:03 +08:00
Chris Lu 58344980e4 preallocate disk space during compaction also, add cleanup for failed compaction 2017-08-29 23:59:53 -07:00
Chris Lu 5047bdb4a2 skip bytes cache 2017-06-03 11:44:24 -07:00
Chris Lu 82c3ccc8dd add btree for volume index 2017-05-26 22:51:25 -07:00
Chris Lu bd97cbc523 skip busy reporting back to master 
possible fix for https://github.com/chrislusf/seaweedfs/issues/499
 2017-05-23 21:12:03 -07:00
Chris Lu 043b7a7c65 add locking to access l.volumes 2017-01-20 10:18:43 -08:00
brstgt 0656838fe5 Don't return actual file size from CheckVolumeDataIntegrity, it will be 0 if last needle is a tombstone, so it's not reliable anyway 2017-01-20 16:37:45 +01:00
brstgt 4fb5bb09b6 Remove obsolete property Volume.dataFileSize 2017-01-20 16:31:11 +01:00
brstgt e074a54a20 Delete volumes online without restarting volume server 2017-01-20 13:02:37 +01:00
brstgt 492f93416d Mount and unmount volumes online without restarting volume server 2017-01-20 12:49:20 +01:00
brstgt 18b3afc97a Log volumeId if dataFileSize != actual data file size - improve log types 2017-01-20 12:22:25 +01:00
brstgt 0c898df430 Log volumeId if dataFileSize != actual data file size 2017-01-20 11:52:23 +01:00
Chris Lu 5b0f5f456e volume server get notified of leader change 2017-01-18 09:34:27 -08:00
Chris Lu 59022b6fe0 fix byte counter on loading index file 
fix https://github.com/chrislusf/seaweedfs/issues/441
 2017-01-18 08:40:39 -08:00
Chris Lu 3065506b38 volume servers always connect to the master leader 2017-01-13 10:20:40 -08:00
Chris Lu e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00
Chris Lu 4beaaa0650 simplify the gzip optimization logic 
fix https://github.com/chrislusf/seaweedfs/issues/436
 2017-01-09 19:31:58 -08:00
Chris Lu ed44f12f6d support Fallocate on linux 2017-01-08 11:01:46 -08:00
Chris Lu 7b6837cbc2 move back the section 2017-01-08 10:35:47 -08:00
sparklxb da9b672d1b support additional header name-value pairs 2017-01-08 22:34:42 +08:00
sparklxb 86a7c56275 support additional header name-value pairs 2017-01-08 09:16:40 +08:00
Chris Lu 13e7069eb9 keep track of total data file size 2017-01-06 10:22:20 -08:00
lixianbin d96d0a87cf fix bug: upload big .gz file more than maxMB 2017-01-04 11:23:40 +08:00
Chris Lu dcaf1796fe add option to enable caching 2016-11-16 07:09:57 -08:00
Chris Lu f54f530ada adjust verbose logging 2016-11-13 14:07:51 -08:00
霍晓栋 b9f385bd66 refactor concurrent loading code logic 2016-11-13 13:24:52 +08:00
霍晓栋 096ffa9744 concurrent loading volume 2016-11-11 11:53:22 +08:00
Chris Lu 36f9633223 add locks for location.volumes 
fix https://github.com/chrislusf/seaweedfs/issues/392
 2016-11-06 20:55:22 -08:00
霍晓栋 7d73bbb073 comment UT case 2016-10-07 16:40:51 +08:00
霍晓栋 7d382ba5fe comment UT case 2016-10-07 16:34:22 +08:00
霍晓栋 ce1f7ab662 makediff func with UT case 2016-10-07 16:22:24 +08:00
霍晓栋 ed848425c7 supplemental data between compacting and commit compacting 2016-09-29 13:57:23 +08:00
Chris Lu dffad65f2f fix compilation 2016-09-26 22:30:44 -07:00
Chris Lu 7e29218327 add a template for makeupDiff 2016-09-26 22:26:41 -07:00
Chris Lu 1bc041b46d add a new way to manually compact corrupted volume 
fix https://github.com/chrislusf/seaweedfs/issues/371
 2016-09-22 20:31:20 -07:00
Chris Lu 01cbd5cb58 lock 
fix https://github.com/chrislusf/seaweedfs/issues/367
 2016-09-08 09:50:31 -07:00
霍晓栋 b9b3651a98 deleted index entry could not point to deleted needle 2016-07-25 14:56:58 +08:00
霍晓栋 09bd3d015d deleted index entry could not point to deleted needle 2016-07-25 14:54:40 +08:00
霍晓栋 78678f4bcb deleted needle does not need checksum verification 2016-07-25 11:40:35 +08:00
Chris Lu d981eb282f refactor volume_read_write.go out of volume.go 2016-07-03 00:10:27 -07:00
Chris Lu 576573711a refactoring 2016-07-02 23:58:23 -07:00
Chris Lu 3d8df0f709 refactor volume_loading.go out of volume.go 2016-07-02 23:56:49 -07:00
Chris Lu 582d5d526c refactoring 2016-07-02 23:50:58 -07:00
Chris Lu 3713119ab9 adjust verification logic. 
1. log file name for easier debugging
2. remove unused version in readIndexEntryAtOffset()
3. if indexSize==0, the dataFile usually still has a super block of
size 8. So skipping this case.
 2016-07-02 23:45:37 -07:00
霍晓栋 16f6984d02 refactor data integrity checking code v2 2016-07-03 14:11:25 +08:00
霍晓栋 dda13def2a refactor data integrity checking code 2016-07-03 12:53:03 +08:00
霍晓栋 2e74fb60c8 add data integrity checking 2016-07-02 21:34:30 +08:00
Chris Lu 101e784577 add locking on possible concurrent map access 
fix https://github.com/chrislusf/seaweedfs/issues/328
 2016-06-23 09:10:25 -07:00
霍晓栋 1ef81ac518 filer could detect master nodes healthy status and choose a working one when encountering SPOF 2016-06-07 11:38:40 +08:00
Chris Lu 5ce6bbf076 directory structure change to work with glide 
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
 2016-06-02 18:09:14 -07:00
Chris Lu 5071f528f6 testing compilation with remove package 2013-02-10 03:49:51 -08:00
Chris Lu 55f2627fcf testing new directory structure 2013-02-10 03:31:40 -08:00
Chris Lu ab6fb13ad7 avoid the "src" folder 2013-02-10 03:25:35 -08:00