gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3403 commits 28 branches 278 tags 135 MiB
Commit graph

619 commits

Author SHA1 Message Date
Yoni Nakache cd4c9a365b
DiskStats: adding Total & Percent Usage 
making relevant data visible and readable.
 2020-02-23 22:33:47 +02:00
Chris Lu f55f49970f move volume server stats to grpc 2020-02-21 21:45:03 -08:00
Chris Lu 9b6296e77a volume: add grpc file read operation 
This is added more for performance benchmarking.
 2020-02-14 00:37:32 -08:00
Chris Lu 2a6db0fd43 volume: return 204 for unchanged file uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1196
 2020-02-11 09:45:02 -08:00
Chris Lu 382ff218d3 filer: set file size, streaming chunk file uploading 
fix https://github.com/chrislusf/seaweedfs/issues/1193
 2020-02-03 17:04:06 -08:00
Chris Lu 40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu d4cde5df49 return empty response if not found 2020-01-24 18:07:34 -08:00
Chris Lu 3cdb27fafc adjust logging 2020-01-24 17:55:39 -08:00
Chris Lu e6f9f5da3a adjust logging 2020-01-24 17:26:03 -08:00
Chris Lu d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu 9b01a99d9a adjust logging 2020-01-21 22:45:50 -08:00
Chris Lu 1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu 4e731f1c8b volume: copy volumes also include .vif file 2020-01-04 11:28:29 -08:00
Chris Lu df636e4c14 remove .vif files 2020-01-03 12:46:42 -08:00
Chris Lu 3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu 86d4b18eb5 filer: optional recursive deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1176
 2019-12-31 11:52:54 -08:00
Chris Lu 509f314350 keep .vif file for ec to normal volume conversion 2019-12-28 21:52:55 -08:00
Chris Lu 0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu 8fbc0a9163 fix edge cases 2019-12-28 12:59:31 -08:00
Chris Lu 37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu 2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu b3c83f9404 fix spelling 2019-12-28 11:16:10 -08:00
Chris Lu 48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
Chris Lu d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
Chris Lu 7ba6be2312 volume ui add readonly status 2019-12-24 14:55:26 -08:00
stlpmo f87ccbddb4 remove the http 304 response body 2019-12-24 17:56:50 +08:00
Chris Lu 8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu 09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu 36ddca9d1f filer: return not found 404 status if not found 
fix https://github.com/chrislusf/seaweedfs/issues/1160
 2019-12-18 21:04:40 -08:00
Chris Lu 58f88e530c volume: use sorted index map for readonly volumes 2019-12-18 01:21:21 -08:00
Chris Lu 014487b1a6 passing entry.Extended 2019-12-17 21:39:48 -08:00
Chris Lu e707211dad passing entry.exended 2019-12-17 21:22:16 -08:00
Chris Lu ad9d5ff5d0 adjust UI 2019-12-15 20:57:17 -08:00
Chris Lu 392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu bbb6ebc3c0 filer: DeleteFolderChildren for deleting large folders 2019-12-13 00:23:05 -08:00
Chris Lu 0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu d0b423bbc0 filer: increase directory listing pagination size 2019-12-12 09:11:10 -08:00
Chris Lu 1f400489c6 filer: UI ensure a correct foler link 2019-12-11 23:09:01 -08:00
Chris Lu f51a6d8ba5 increase filer pagination limit 2019-12-11 22:21:37 -08:00
Chris Lu 1ddfbff2d3 deletion add option skipChunkDeletion 2019-12-11 14:58:22 -08:00
Chris Lu f06b65a6fd filer: EXPERT add shouldDeleteChunks parameter for fast deletion 2019-12-11 07:50:53 -08:00
Yoni Nakache 9e45702334
Filer: Enable delete entries without flag volume 
same as happens on the s3 API on delete bucket,
to reduce the IO requests on the system, you can just clear recursively folders without flagging them in volume server (they will be removed on delete bucket/drop collection)
this will remove the user ability to retrieve data under these folders
 2019-12-11 11:30:19 +02:00
Chris Lu 10bd3c6b4b refactoring 2019-12-08 19:44:16 -08:00
Chris Lu e2a8cdfc48 webdav: fix writing file 
fix https://github.com/chrislusf/seaweedfs/issues/1114
 2019-12-07 07:56:51 -08:00
Chris Lu 0eda75fa2c add logging 2019-12-07 07:56:05 -08:00
Chris Lu 81f424a630 go fmt 2019-12-06 06:57:32 -08:00
Chris Lu 916423e96a master: submit support "dataCenter" parameter 2019-12-05 15:51:24 -08:00
Chris Lu e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu 1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu 6383b45bd0 add lock variable 2019-12-02 20:49:50 -08:00
Chris Lu ec8de250e2 tiered storage: can copy to s3, read from s3 
master not aware tiered volume yet, file assigning is not working yet
 2019-12-02 15:08:31 -08:00
Chris Lu 0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu ea9a7e61d0 only send metrics address during on the first response 2019-11-28 20:48:08 -08:00
Chris Lu f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu 586798ecc0 go fmt 2019-11-27 12:34:57 -08:00
Chris Lu 0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Lei Liu 2700ac6d1f filer: Add attr.mode in query string 
Signed-off-by: Lei Liu <liul.stone@gmail.com>
 2019-11-27 13:57:29 +08:00
Chris Lu fb3c9ed80d shell: fs.meta.save missed directory data with more than 1024 entries 
fix https://github.com/chrislusf/seaweedfs/issues/1130
 2019-11-22 01:39:50 -08:00
Chris Lu de0c983808 refactoring scaffolding for master.toml 2019-11-14 22:20:38 -08:00
Lei Liu 46755ea1e1 fix master maintenance logic 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-12 14:46:10 +08:00
Chris Lu ee90236a97 Merge branch 'master' into refactoring_dat_backend 2019-11-11 21:30:14 -08:00
Chris Lu d7852ebb3b
Merge pull request #1102 from stlpmo-jn/add_ETCD_sequencer 
Add etcd sequencer
 2019-11-11 12:43:24 -08:00
stlpmo 62d393d6c9 ut pass 2019-11-11 10:52:21 +08:00
stlpmo 802a0eb3fe move from cmd-line to scaffold 2019-11-11 09:15:17 +08:00
zhangsong 61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00
Chris Lu c5c1d83d91 Merge branch 'master' into refactoring_dat_backend 2019-11-08 22:54:41 -08:00
Chris Lu 6cc9e0d788 volume: fix ec shard reading 
fix https://github.com/chrislusf/seaweedfs/issues/1106
 2019-11-08 22:41:02 -08:00
stlpmo 1c8bed3810 delete the var etcdClient and comments 
refactor the code
add sequencer cmd-line
delete nerver used codes
 2019-11-05 15:45:25 +08:00
Lei Liu 1294999d8b return error when garbageThreshold is not a valid float number 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 20:43:20 +08:00
Lei Liu 1bcef02a6c fix dir/lookup and col/delete api 
1, Fix Layouts first letter capitalized
2, Return http 204 when delete a collection

Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 16:55:45 +08:00
Lei Liu f2f90436ef fix leader master /dir/lookup api 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-30 16:38:40 +08:00
Chris Lu 5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Lei Liu c2884cace2 misc updated 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-29 21:28:28 +08:00
Chris Lu 19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu 71eb8efd20 fix error logs 2019-10-25 09:44:58 -07:00
Chris Lu 55a2221eba master: add logging for volume server disconnection 
add logging for https://github.com/chrislusf/seaweedfs/issues/1093
 2019-10-25 09:35:19 -07:00
Chris Lu 46ed2ca902 refactoring 2019-10-24 23:41:32 -07:00
divinerapier f1ee6b7a54
fix abused 404 status code 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-10-24 16:51:26 +08:00
Chris Lu c358f96aaa volume: redirection follow public url 
fix https://github.com/chrislusf/seaweedfs/issues/1089
 2019-10-23 22:11:42 -07:00
Chris Lu 23240efa63 filer: add tikv support 2019-10-23 00:31:31 -07:00
Chris Lu c71fddee2b go fmt 2019-10-22 00:50:50 -07:00
Chris Lu fc412e428b refactor ScanVolumeFileFrom() 2019-10-22 00:50:30 -07:00
Chris Lu c9a183eb69 refactor memory map related code 2019-10-22 00:49:42 -07:00
Chris Lu faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
Chris Lu d16c450682 fix unreachable code 2019-10-21 22:36:41 -07:00
joeslay d53aee179b
Merge pull request #10 from chrislusf/master 
merge seaweed master
 2019-10-14 16:03:40 +01:00
Chris Lu 939e4b57a8 go fmt 2019-10-09 00:03:18 -07:00
Chris Lu f8d4b7d1c0 support basic json filtering and selection 2019-10-06 22:35:08 -07:00
Chris Lu cf47f657af scaffold for volume server query feature 2019-10-06 22:35:08 -07:00
joeslay 9d03aa6a38
Merge pull request #8 from chrislusf/master 
merge seaweed master
 2019-10-01 17:55:31 +01:00
Lei Liu 3cc084269c master api: return http 404 when volumeId not exist 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-09-29 14:23:39 +08:00
j.laycock 6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
joeslay d8c34b032f
Merge pull request #6 from chrislusf/master 
merge seaweed master
 2019-09-12 11:44:09 +01:00
Chris Lu ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Tom Maxwell 4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
joeslay 6ee65356e3
Merge pull request #3 from chrislusf/master 
merge seaweed master
 2019-09-04 14:58:04 +01:00
j.laycock cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
Chris Lu ee8bcb0c12 remove println 2019-09-03 09:22:56 -07:00
j.laycock 1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
joeslay e91139348e
Merge pull request #2 from chrislusf/master 
merge seaweed master
 2019-09-02 11:39:40 +01:00
j.laycock 595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
Chris Lu 60c9215a00 HCFS: avoid verbose file not found exception stack trace 2019-09-01 21:40:26 -07:00
Chris Lu 0dc43ecdca skip writing content if not modified 
related to https://github.com/chrislusf/seaweedfs/issues/1036
 2019-08-09 14:38:54 -07:00
Feng Ye fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu ca2b81a944 fix test error 2019-07-31 02:09:13 -07:00
Chris Lu 19360ab10a master: redirect clients to the new leader 2019-07-31 01:54:42 -07:00
Chris Lu 8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu 898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu 0264a7f50a set the max file key when ever possible 2019-07-17 23:23:01 -07:00
Chris Lu aff911c00d skip all azuresb 2019-07-17 01:24:20 -07:00
Chris Lu 0ea98dc6a2 disable azure due to api changes 2019-07-17 00:39:36 -07:00
Chris Lu 9e47c4749a shell: add fs.mv 
fix https://github.com/chrislusf/seaweedfs/issues/954
 2019-07-11 09:29:04 -07:00
Chris Lu 1d1b355f9e filer: set file name when uploading to a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1005
 2019-07-11 04:20:03 -07:00
Chris Lu 476a7c269c detect a filer path is a file or a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1004
 2019-07-10 12:11:18 -07:00
Chris Lu 15ecf4e3bf filer: default to leveldb2 2019-06-30 00:44:57 -07:00
Chris Lu 6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu 359563e95a refactoring 2019-06-25 09:49:27 -07:00
Chris Lu c777102da9 go fmt 2019-06-23 15:30:16 -07:00
Chris Lu d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu f16375621f big refactoring 2019-06-23 03:08:27 -07:00
Chris Lu 6c01fb6d2d go fmt 2019-06-23 01:57:51 -07:00
Chris Lu 8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu d1cd8f8c5b add metrics, refactoring 2019-06-22 22:53:52 -07:00
Chris Lu 3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu 613a2e8060 delete garbage only when successful 2019-06-20 23:46:00 -07:00
Chris Lu c74dc2b306 go fmt 2019-06-20 00:55:52 -07:00
Chris Lu e63317fb08 ec deletion code complete, not tested yet 2019-06-20 00:55:30 -07:00
Chris Lu 4cea8aefd0 add grpc VolumeEcBlobDelete 2019-06-20 00:17:11 -07:00
Chris Lu 856da7aae2 ec volume support deletes 2019-06-19 22:57:14 -07:00
Chris Lu d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu ff6a6dd11e refactoring 2019-06-15 21:46:55 -07:00
Chris Lu 8b43679ae3 refactoring 2019-06-15 12:21:44 -07:00
Chris Lu 5336008dcd refactoring 2019-06-14 13:06:01 -07:00
Chris Lu d67189921f go fmt 2019-06-14 00:55:12 -07:00
Chris Lu 5f6c9825f8 volume server: adds basic metrics 2019-06-14 00:54:56 -07:00
Chris Lu a11525fe4e filer: adds basic metrics pushing to Prometheus gateway 2019-06-13 02:01:54 -07:00
Chris Lu 57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu e66bddd84c customizable sleep intervals 2019-06-06 00:39:08 -07:00
Chris Lu 50aa769554 jwt for read access control 2019-06-06 00:29:02 -07:00
Chris Lu d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu 450f4733ce report empty volume or ec shards 2019-06-05 13:32:33 -07:00
Chris Lu 7f6fb9ffd0 every 17 minutes instead of seconds 
TODO: make this configurable
 2019-06-05 13:32:08 -07:00
Chris Lu ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu 2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00