gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
2747 commits 28 branches 278 tags 135 MiB
Commit graph

343 commits

Author SHA1 Message Date
Tom Maxwell 4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock cc756ddd79 Rename End_of_file variable, implement platform indepenent memory_map.go 2019-09-04 14:24:17 +01:00
j.laycock de5a7e60a2 remove temp hack 2019-09-03 18:20:09 +01:00
j.laycock cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
j.laycock 1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
j.laycock 9a459d984b Do not vacuum memory mapped files 2019-09-03 15:05:43 +01:00
j.laycock 0123c7a898 Change to max 2GB for now 2019-09-02 17:46:22 +01:00
joeslay e91139348e
Merge pull request #2 from chrislusf/master 
merge seaweed master
 2019-09-02 11:39:40 +01:00
j.laycock 595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
j.laycock 1e62a2b233 Fix memory not being unmapped 2019-08-30 16:55:33 +01:00
j.laycock 840ccdc35d Refactor to pass memory maps by reference instead of value, fix memory maps not being created properly or written to properly 2019-08-30 12:30:00 +01:00
j.laycock dc50701e7d Overload createFile function, expose File in memory_map struct, delete from memory_map map on delete 2019-08-30 12:30:00 +01:00
j.laycock 523f3a12b3 use Os.File/uintptr instead of windows.Handle in memory map struct 2019-08-30 12:30:00 +01:00
j.laycock 0e5d3b1a70 Move memory_map_windows to new folder, intercept most of the read and write calls to volume dat files 2019-08-30 12:30:00 +01:00
Chris Lu 58168a8c52 volume: truncate dat files when creating 2019-08-26 13:01:57 -07:00
joeslay 36bde84000
Merge pull request #1 from chrislusf/master 
merge changes from chrislusf/seaweed master
 2019-08-23 17:03:37 +01:00
j.laycock 1e89e719b3 Add windows memory map functions to storage package 2019-08-23 16:19:00 +01:00
xushuxun 9af9f36b89 volume: read dat files' last modified time no matter dat files are catWrite or readonly 2019-08-21 11:26:01 +08:00
Chris Lu f9c1cd7370 avoid dead lock from dataFileAccessLock 
fix https://github.com/chrislusf/seaweedfs/issues/1043
 2019-08-15 09:24:54 -07:00
Chris Lu d829df4f59 volume: protect against nil needle map 
fix @mastak reported nil problem in https://github.com/chrislusf/seaweedfs/issues/1037
 2019-08-14 01:08:01 -07:00
Chris Lu e40634e6b4 volume: fail the volume deletion if compaction is in progress 
fix https://github.com/chrislusf/seaweedfs/issues/1035
 2019-08-12 00:53:50 -07:00
Chris Lu f3b99cbfe5 fix wrong alignment fix 2019-07-21 13:50:42 -07:00
Chris Lu 644c5c27b4 refactoring 2019-07-21 13:50:24 -07:00
Chris Lu 33b3366043 volume: avoid compacting wait group 2019-07-21 13:49:59 -07:00
Chris Lu fd16adbde1 stricter checking before writes 2019-07-21 11:21:30 -07:00
Chris Lu 5a135fef87 purge old code 2019-07-21 10:45:36 -07:00
Chris Lu c54d9221b9 write requests also checks cookie if overwrites 
protect against edge cases, avoid https://github.com/chrislusf/seaweedfs/issues/1014
 2019-07-17 23:57:34 -07:00
Chris Lu 898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu c33f423955 stop early if compaction fails 
fix https://github.com/chrislusf/seaweedfs/issues/1015
 2019-07-17 23:22:01 -07:00
Chris Lu a7b1b23c58 fix wrong volume count 
fix https://github.com/chrislusf/seaweedfs/issues/1013
 2019-07-17 00:03:15 -07:00
Chris Lu a3d1296ed9 go fmt 2019-06-27 12:18:59 -07:00
Chris Lu ec75b2d761 volume: fix bug with 8000GB version if using in memory index 
fix https://github.com/chrislusf/seaweedfs/issues/994
 2019-06-27 12:18:45 -07:00
Chris Lu 6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu c2960c7a86 defaulting file ttl to volume ttl 
fix https://github.com/chrislusf/seaweedfs/issues/992
 2019-06-26 11:18:23 -07:00
Chris Lu a14bd31f5b go fmt 2019-06-22 20:05:25 -07:00
Chris Lu f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu e63317fb08 ec deletion code complete, not tested yet 2019-06-20 00:55:30 -07:00
Chris Lu 4cea8aefd0 add grpc VolumeEcBlobDelete 2019-06-20 00:17:11 -07:00
Chris Lu 856da7aae2 ec volume support deletes 2019-06-19 22:57:14 -07:00
Chris Lu 115558e5f5 adjust counters 2019-06-17 21:02:50 -07:00
Chris Lu 935639b908 metrics: disk size for <collection, normal/EC> 2019-06-16 21:56:41 -07:00
Chris Lu 0fdb1e705d collect volume disk usage metrics 2019-06-16 02:44:20 -07:00
Chris Lu 289fd7eb39 count number of volumes and ec shards 2019-06-16 02:24:15 -07:00
Chris Lu 57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu 450f4733ce report empty volume or ec shards 2019-06-05 13:32:33 -07:00
Chris Lu ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu 2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu 0e52862586 avoid the test that depends on ordering in a different folder 2019-06-04 02:22:32 -07:00
Chris Lu 2f7710a75d debug sporadic test error on travis 2019-06-04 02:09:06 -07:00
Chris Lu 11cffb3168 fix ec.rebuild bugs 2019-06-03 11:50:54 -07:00
Chris Lu 7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu 133b772fb5 destroy ec volume if it is empty 2019-06-01 01:51:28 -07:00
Chris Lu ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu f919d0235c ec encode volumes quiet for a period of time 2019-05-31 23:41:17 -07:00
Chris Lu de1a35acdd conditionally fresh the shard locations 2019-05-31 02:22:12 -07:00
Chris Lu 47f1901843 ask for the ec volume version 2019-05-31 00:58:51 -07:00
Chris Lu 689930f092 forget shards that are broken 2019-05-31 00:19:13 -07:00
Chris Lu aaab2c25f5 resolve concurrent modification problem 
n and err can be modified concurrently
 2019-05-30 13:59:07 -07:00
Chris Lu 40ca2f2903 add collection.delete 2019-05-30 09:47:54 -07:00
Chris Lu 5c6c757619 clean up test files 2019-05-29 00:34:30 -07:00
Chris Lu 5dd67f9acf reading by recover from other shards 2019-05-28 23:48:39 -07:00
Chris Lu 3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu 302d9fbc6d adjust error message for file not found 2019-05-28 14:27:21 -07:00
Chris Lu 4f76342cbc WIP 
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
 2019-05-28 00:51:01 -07:00
Chris Lu 2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu 713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu 3a8c1055a2 refactoring ecx to ecVolume 2019-05-27 22:00:36 -07:00
Chris Lu 217cde0a3b refactoring 2019-05-27 21:40:51 -07:00
Chris Lu 03b9291e5d volum server passes grpc option and master location to store 2019-05-27 21:22:23 -07:00
Chris Lu b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu a4f3d82c57 convert needle id to ec intervals to read from 2019-05-27 01:29:46 -07:00
Chris Lu a463759edf adjust error message 2019-05-26 01:14:42 -07:00
Chris Lu af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu 6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu 228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu 8a96445f40 register ec shards to each data node 2019-05-23 22:51:18 -07:00
Chris Lu 4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu 7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Chris Lu 17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu 693d6c0f1b clean up tests 2019-05-19 21:16:01 -07:00
Chris Lu 6386a3174b able to validate by randomly selected ec files 2019-05-19 14:24:33 -07:00
Chris Lu 7c2c60c376 add locating data inside the ec files 2019-05-19 03:01:58 -07:00
Chris Lu 87f63b9c08 generate ec01~ec14, generate ecx file with sorted needle values 2019-05-18 22:46:24 -07:00
Chris Lu 12dc6608f0 Merge branch 'master' into erasure_coding 2019-05-18 11:16:07 -07:00
Chris Lu 76cf57a0e1 volume: optimzation for leveldb 2019-05-17 17:34:03 -07:00
Chris Lu 8156958ee9 move function to make travis happy 2019-05-15 10:02:44 -07:00
Chris Lu 0a36f628c6 testing RS coding 2019-05-15 01:02:00 -07:00
Chris Lu cf58fc0e63 refactor: extract out the write throttler 2019-05-06 13:56:08 -07:00
Chris Lu 46a675ecc2 align memory to avoid nil with atomic operations 2019-05-04 21:33:05 -07:00
Chris Lu 177f3add81 adjust memory layout to avoid possible nil pointer 2019-05-04 08:47:11 -07:00
Chris Lu 4b27f892e6 remove 128 bit needle id build option 
fix https://github.com/chrislusf/seaweedfs/issues/949
 2019-05-03 22:39:00 -07:00
Chris Lu b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Wine93 32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu 9beea63960 memory alignment 
fix https://github.com/chrislusf/seaweedfs/issues/939
 2019-04-24 09:23:19 -07:00
Chris Lu 53e2e73de2 fix metrics 
fix https://github.com/chrislusf/seaweedfs/issues/935
 2019-04-22 12:32:10 -07:00