Commit graph

453 commits

Author SHA1 Message Date
Chris Lu d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu b6e6ca8595 adjust logs 2020-01-24 20:06:58 -08:00
Chris Lu c8b2dac6c1 volume: avoid sharing volume dat file handle
possibly help on https://github.com/chrislusf/seaweedfs/issues/1184
2020-01-21 21:18:01 -08:00
Chris Lu f8a20ef35e add the old way to compact as a comment 2020-01-08 09:45:42 -08:00
Chris Lu 288baf37fd saving .vif files correctly 2020-01-08 09:45:26 -08:00
Chris Lu acf7ca7b93 volume: fix compaction 2020-01-08 09:45:03 -08:00
Chris Lu 943f4986ef fix possible nil 2020-01-08 08:49:18 -08:00
Chris Lu 9995d3bcb5 remove println 2020-01-07 21:49:28 -08:00
Chris Lu 3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu 672868b460 always create .vif file 2019-12-28 21:52:06 -08:00
Chris Lu b6f7033478 go fmt 2019-12-28 21:37:29 -08:00
Chris Lu 9134092a8d endure creating .vif files 2019-12-28 21:36:15 -08:00
Chris Lu 0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu 37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu c06f7eb48a load volume info from .vif file, use superblock as a backup 2019-12-28 12:28:58 -08:00
Chris Lu f4a74e03d1 refactoring: separating .vif from tier file loading 2019-12-28 11:35:27 -08:00
Chris Lu 2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu e8b357fd53 rename from .tier to .vif 2019-12-28 11:17:39 -08:00
Chris Lu 48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
Chris Lu 1346437d71
Merge pull request #1171 from stlpmo-jn/redundant_type_conversion
remove the redundant type conversion
2019-12-25 17:02:05 -08:00
Chris Lu 35393b4a02 cleanly destroy remote files 2019-12-25 16:17:58 -08:00
Chris Lu eecad6062a go fmt 2019-12-25 09:56:42 -08:00
Chris Lu d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
stlpmo f3de4b6c18 remove the redundant type conversion 2019-12-25 10:13:45 +08:00
Chris Lu 9ff72f616a go fmt 2019-12-24 14:56:16 -08:00
Chris Lu efd2f50ede compaction changed to .idx based deletion 2019-12-24 14:55:50 -08:00
Chris Lu d8b39fe92a testing 2019-12-24 11:29:26 -08:00
Chris Lu f42b5bd0f5 rename .sdb to .sdx 2019-12-24 10:19:12 -08:00
Chris Lu abffe857a1 change btree map to in memory level db 2019-12-24 10:18:56 -08:00
Chris Lu 72a561ab7c refactoring 2019-12-24 08:13:01 -08:00
Chris Lu 409a3fe41f
Merge pull request #1168 from stlpmo-jn/optimize_DeleteCollectionFromDiskLocation
decouple the volume.Destroy() from the operation of unmountVolume()
2019-12-24 07:08:04 -08:00
stlpmo 38e4b79125 decouple the volume.Destroy() from the operation of unmountVolume() 2019-12-24 17:20:34 +08:00
stlpmo f300c35266 resolve issue : when the host disk full, the volume server will core dump 2019-12-24 16:36:15 +08:00
Chris Lu a18f62bbe7 only copy required shards 2019-12-23 18:06:13 -08:00
Chris Lu 8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu 09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu ab966d7192 refactoring variable name 2019-12-19 00:44:46 -08:00
Chris Lu f61de28c69 volume: add deletion capability for previously readonly volumes 2019-12-19 00:42:46 -08:00
Chris Lu deb03ecfa3 remove duplicated checking
already checked in store.go
2019-12-18 23:59:53 -08:00
Chris Lu ef3ae3cd41 clean up sorted index file 2019-12-18 20:59:15 -08:00
Chris Lu 39810a9951 tiered volume is read only 2019-12-18 01:26:22 -08:00
Chris Lu 58f88e530c volume: use sorted index map for readonly volumes 2019-12-18 01:21:21 -08:00
Chris Lu 356bd1b629 adjust text 2019-12-10 22:09:29 -08:00
Chris Lu 10bd3c6b4b refactoring 2019-12-08 19:44:16 -08:00
Chris Lu 2b8e20f122 remove unused functions 2019-12-08 18:33:25 -08:00
Chris Lu f38f90b7ea cleanly remove .tier volume file 2019-12-06 22:19:47 -08:00
Chris Lu 6827cabfbb volume: dataFileAccessLock change to RW Lock 2019-12-06 06:59:57 -08:00
Chris Lu e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu 7ae8b1cc86 show volume info in volume.list in weed shell 2019-12-02 23:38:56 -08:00
Chris Lu 1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu 70648d35ad go fmt 2019-12-02 20:49:58 -08:00
Chris Lu 6383b45bd0 add lock variable 2019-12-02 20:49:50 -08:00
Chris Lu caae543a9f fix test 2019-12-02 15:54:24 -08:00
Chris Lu ec8de250e2 tiered storage: can copy to s3, read from s3
master not aware tiered volume yet, file assigning is not working yet
2019-12-02 15:08:31 -08:00
Chris Lu 0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu 61bc1d6ffd tweaking 2019-11-28 18:47:51 -08:00
Chris Lu f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu ee2072dff9 volume: avoid inifinite loop reading ec volume info if error happens 2019-11-27 23:07:53 -08:00
Chris Lu 0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Chris Lu e4c63ff2f8 go fmt 2019-11-22 14:51:09 -08:00
Chris Lu 9711a6ffaa WIP 2019-11-18 19:24:37 -08:00
chenwanli 295f3beed5 rm ldb and bdb directory 2019-11-19 09:35:06 +08:00
Chris Lu c6e8225a20 adjust memory mapped file size
related to https://github.com/chrislusf/seaweedfs/pull/1087
2019-11-12 21:05:48 -08:00
Lei Liu 46755ea1e1 fix master maintenance logic
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-12 14:46:10 +08:00
Chris Lu 85f8649320 refactor memory mapped file into backend storage 2019-11-09 00:10:59 -08:00
Chris Lu c5c1d83d91 Merge branch 'master' into refactoring_dat_backend 2019-11-08 22:54:41 -08:00
Chris Lu c34ffed43f go fmt 2019-11-08 22:47:50 -08:00
Chris Lu 84c503c6a7 adjust ec reading log level 2019-11-08 22:41:02 -08:00
zhangsong 1dd101f782 use read lock to avoid io hang during heartbeat 2019-11-08 20:27:55 +08:00
Chris Lu db30a46050 skip memory mapped file when compacting 2019-10-29 23:34:38 -07:00
Chris Lu 5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Chris Lu 57e441d67b fix compaction logic 2019-10-29 23:18:01 -07:00
Chris Lu b7156291a8 adjust variable names 2019-10-29 23:16:43 -07:00
Chris Lu 4b5ba4927f adjust NewDiskFile() paramaters 2019-10-29 22:37:36 -07:00
Chris Lu cd7ce720c7 fix test 2019-10-29 00:37:15 -07:00
Chris Lu 19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu 7a51a9a582 refactor: simplifying to ReadAt() and WriteAt() 2019-10-25 00:11:25 -07:00
Chris Lu 46ed2ca902 refactoring 2019-10-24 23:41:32 -07:00
Chris Lu f937933b3d refactoring 2019-10-23 22:25:53 -07:00
Chris Lu 3c865ee39b fix missing needle header read 2019-10-22 00:57:47 -07:00
Chris Lu fc412e428b refactor ScanVolumeFileFrom() 2019-10-22 00:50:30 -07:00
Chris Lu c9a183eb69 refactor memory map related code 2019-10-22 00:49:42 -07:00
Chris Lu fec07c829d go fmt 2019-10-21 23:03:48 -07:00
Chris Lu faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
j.laycock 2e2fe00dbd Comma, no space? 2019-10-18 11:31:25 +01:00
j.laycock 248f3be6e3 using a space instead of a comma to hopefully fix the build! 2019-10-18 11:23:02 +01:00
j.laycock 2c455841ea Make volumeCreate more consistent between all 3 implementations. 2019-10-18 11:01:45 +01:00
j.laycock b0ddad6889 Fix volume_create breaking the build 2019-10-18 10:32:07 +01:00
joeslay d53aee179b
Merge pull request #10 from chrislusf/master
merge seaweed master
2019-10-14 16:03:40 +01:00
Chris Lu 87e5a02a99 fix logic error 2019-10-09 00:07:18 -07:00
Chris Lu 09874f0d16 volume: return error if superblock is not initialized
fix https://github.com/chrislusf/seaweedfs/issues/1079
2019-10-09 00:02:50 -07:00
j.laycock eb27c2b037 Make releaseMemory private and return byte array instead, fix other platform compilation issues, reduce in-memory chunk size. 2019-10-01 12:21:44 +01:00
j.laycock d5f5acb734 limit locking physical memory to 80% of max physical memory 2019-09-20 12:44:29 +01:00
j.laycock 476140fd6b minor change to setProcessWorkingSetSize function 2019-09-19 11:59:00 +01:00
j.laycock 6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
joeslay d8c34b032f
Merge pull request #6 from chrislusf/master
merge seaweed master
2019-09-12 11:44:09 +01:00
Chris Lu 5e9c65469e volume: skip readonly checking when Destroy()
fix https://github.com/chrislusf/seaweedfs/issues/1063
2019-09-11 09:45:09 -07:00
j.laycock c1a928887a Add some comments/documentation 2019-09-11 10:42:37 +01:00
joeslay 9ea9a5e97a
Merge pull request #5 from chrislusf/master
merge seaweed master
2019-09-11 10:25:27 +01:00