gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
2833 commits 28 branches 278 tags 135 MiB
Commit graph

1460 commits

Author SHA1 Message Date
Chris Lu e6ab75972f use constant 2019-11-10 12:05:58 -08:00
Chris Lu 471902cd51 change parameter name 2019-11-10 12:05:45 -08:00
zhangsong 61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00
Chris Lu 85f8649320 refactor memory mapped file into backend storage 2019-11-09 00:10:59 -08:00
Chris Lu c5c1d83d91 Merge branch 'master' into refactoring_dat_backend 2019-11-08 22:54:41 -08:00
Chris Lu c34ffed43f go fmt 2019-11-08 22:47:50 -08:00
Chris Lu 84c503c6a7 adjust ec reading log level 2019-11-08 22:41:02 -08:00
Chris Lu 6cc9e0d788 volume: fix ec shard reading 
fix https://github.com/chrislusf/seaweedfs/issues/1106
 2019-11-08 22:41:02 -08:00
zhangsong 1dd101f782 use read lock to avoid io hang during heartbeat 2019-11-08 20:27:55 +08:00
stlpmo 1c8bed3810 delete the var etcdClient and comments 
refactor the code
add sequencer cmd-line
delete nerver used codes
 2019-11-05 15:45:25 +08:00
stlpmo 364f7200ad Create etcd_sequencer.go 
the 1st version
 2019-11-04 16:36:06 +08:00
Lei Liu 1294999d8b return error when garbageThreshold is not a valid float number 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 20:43:20 +08:00
Lei Liu 1bcef02a6c fix dir/lookup and col/delete api 
1, Fix Layouts first letter capitalized
2, Return http 204 when delete a collection

Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 16:55:45 +08:00
Lei Liu f2f90436ef fix leader master /dir/lookup api 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-30 16:38:40 +08:00
Chris Lu db30a46050 skip memory mapped file when compacting 2019-10-29 23:34:38 -07:00
Chris Lu 5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Chris Lu 57e441d67b fix compaction logic 2019-10-29 23:18:01 -07:00
Chris Lu b7156291a8 adjust variable names 2019-10-29 23:16:43 -07:00
Chris Lu 4b5ba4927f adjust NewDiskFile() paramaters 2019-10-29 22:37:36 -07:00
Lei Liu c2884cace2 misc updated 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-29 21:28:28 +08:00
Chris Lu cd7ce720c7 fix test 2019-10-29 00:37:15 -07:00
Chris Lu 19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu eb2172f63f FUSE: add os specific mount options 
related to https://github.com/chrislusf/seaweedfs/issues/1094
 2019-10-27 09:12:10 -07:00
Chris Lu 71eb8efd20 fix error logs 2019-10-25 09:44:58 -07:00
Chris Lu 55a2221eba master: add logging for volume server disconnection 
add logging for https://github.com/chrislusf/seaweedfs/issues/1093
 2019-10-25 09:35:19 -07:00
Chris Lu 69de05f6cb go fmt 2019-10-25 07:45:12 -07:00
Chris Lu 288c45a690 fix iitial filer url 2019-10-25 07:44:37 -07:00
Chris Lu 05fe7a2366
Merge pull request #1092 from binbinshi/master 
fix: weed shell filer config is constant
 2019-10-25 07:08:59 -07:00
shibinbin fe1f469978 fix : change default filer.port to 8888 2019-10-25 17:39:29 +08:00
shibinbin 3464f21b5c fix: weed shell add filer runtime attributes 2019-10-25 17:34:03 +08:00
Chris Lu 7a51a9a582 refactor: simplifying to ReadAt() and WriteAt() 2019-10-25 00:11:25 -07:00
Chris Lu 46ed2ca902 refactoring 2019-10-24 23:41:32 -07:00
Chris Lu 9573c0f4b3 filer: filer.copy cleanup in case of failed uploads 2019-10-24 10:16:01 -07:00
Chris Lu 1b71608b3c
Merge pull request #1091 from divinerapier/fix/abused-404 
fix abused 404 status code
 2019-10-24 09:35:54 -07:00
Chris Lu 5d78871487 filer: parallize weed filer.copy for single file 2019-10-24 07:26:23 -07:00
divinerapier f1ee6b7a54
fix abused 404 status code 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-10-24 16:51:26 +08:00
Chris Lu 253dd20bc6 go fmt 2019-10-23 22:25:59 -07:00
Chris Lu f937933b3d refactoring 2019-10-23 22:25:53 -07:00
Chris Lu c358f96aaa volume: redirection follow public url 
fix https://github.com/chrislusf/seaweedfs/issues/1089
 2019-10-23 22:11:42 -07:00
Chris Lu c0776aa40f trying to compile 2019-10-23 01:27:08 -07:00
Chris Lu 23240efa63 filer: add tikv support 2019-10-23 00:31:31 -07:00
Chris Lu 3c865ee39b fix missing needle header read 2019-10-22 00:57:47 -07:00
Chris Lu c71fddee2b go fmt 2019-10-22 00:50:50 -07:00
Chris Lu fc412e428b refactor ScanVolumeFileFrom() 2019-10-22 00:50:30 -07:00
Chris Lu c9a183eb69 refactor memory map related code 2019-10-22 00:49:42 -07:00
Chris Lu fec07c829d go fmt 2019-10-21 23:03:48 -07:00
Chris Lu faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
Chris Lu d16c450682 fix unreachable code 2019-10-21 22:36:41 -07:00
Chris Lu c262526d8a
Merge pull request #1087 from joeslay/master 
Storing files in in-memory collections on windows
 2019-10-21 22:30:35 -07:00
j.laycock 2e2fe00dbd Comma, no space? 2019-10-18 11:31:25 +01:00
j.laycock 248f3be6e3 using a space instead of a comma to hopefully fix the build! 2019-10-18 11:23:02 +01:00
j.laycock 2c455841ea Make volumeCreate more consistent between all 3 implementations. 2019-10-18 11:01:45 +01:00
j.laycock b0ddad6889 Fix volume_create breaking the build 2019-10-18 10:32:07 +01:00
joeslay d53aee179b
Merge pull request #10 from chrislusf/master 
merge seaweed master
 2019-10-14 16:03:40 +01:00
zhangsong 65f1095a54 fix benchmark cmd usage info 2019-10-14 20:23:32 +08:00
Chris Lu 50e885da45 1.44 2019-10-11 23:56:58 -07:00
Chris Lu 7a707617e5 follow the redirected master 
fix https://github.com/chrislusf/seaweedfs/issues/1070
 2019-10-09 21:00:40 -07:00
Chris Lu 87e5a02a99 fix logic error 2019-10-09 00:07:18 -07:00
Chris Lu 939e4b57a8 go fmt 2019-10-09 00:03:18 -07:00
Chris Lu 09874f0d16 volume: return error if superblock is not initialized 
fix https://github.com/chrislusf/seaweedfs/issues/1079
 2019-10-09 00:02:50 -07:00
Chris Lu f8d4b7d1c0 support basic json filtering and selection 2019-10-06 22:35:08 -07:00
Chris Lu e26670c67a add some basic sql types 
copied some code from vitness
 2019-10-06 22:35:08 -07:00
Chris Lu cf47f657af scaffold for volume server query feature 2019-10-06 22:35:08 -07:00
joeslay 9d03aa6a38
Merge pull request #8 from chrislusf/master 
merge seaweed master
 2019-10-01 17:55:31 +01:00
j.laycock eb27c2b037 Make releaseMemory private and return byte array instead, fix other platform compilation issues, reduce in-memory chunk size. 2019-10-01 12:21:44 +01:00
Lei Liu 2cb348a2c2 s3 api: fix listbucket common_prefixes issue 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-09-29 21:15:40 +08:00
Lei Liu 3cc084269c master api: return http 404 when volumeId not exist 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-09-29 14:23:39 +08:00
j.laycock d5f5acb734 limit locking physical memory to 80% of max physical memory 2019-09-20 12:44:29 +01:00
joeslay 48b093b3d2
Merge pull request #7 from chrislusf/master 
merge seaweed master
 2019-09-19 12:00:05 +01:00
j.laycock 476140fd6b minor change to setProcessWorkingSetSize function 2019-09-19 11:59:00 +01:00
Chris Lu 972e881d48 add more logs 
add more logs
 2019-09-16 09:48:30 -07:00
Chris Lu 20ae559437 add logs 
add logs for https://github.com/chrislusf/seaweedfs/issues/1064
 2019-09-16 08:19:10 -07:00
Chris Lu ad3efbb197 tweaking data types 2019-09-14 01:21:51 -07:00
Chris Lu b0e4771135
Merge pull request #1060 from divinerapier/master 
fix: non-thread-safe rand will panic
 2019-09-14 01:06:14 -07:00
divinerapier bb31462b52
fix: thread unsafe 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-09-13 20:06:02 +08:00
j.laycock 6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
joeslay d8c34b032f
Merge pull request #6 from chrislusf/master 
merge seaweed master
 2019-09-12 11:44:09 +01:00
Chris Lu ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Chris Lu 5e9c65469e volume: skip readonly checking when Destroy() 
fix https://github.com/chrislusf/seaweedfs/issues/1063
 2019-09-11 09:45:09 -07:00
j.laycock c1a928887a Add some comments/documentation 2019-09-11 10:42:37 +01:00
joeslay 9ea9a5e97a
Merge pull request #5 from chrislusf/master 
merge seaweed master
 2019-09-11 10:25:27 +01:00
j.laycock 15a4c91c28 Remove Finalizer 2019-09-10 17:05:54 +01:00
j.laycock 9c9dff7386 Virtual lock memory pages and Set Process Working set size to hint to windows harder not to write pages to disk as much, add finalize function, minor renaming 2019-09-10 16:18:51 +01:00
Chris Lu 02710f9624 refactoring: remove unused functions 2019-09-09 10:03:41 -07:00
Chris Lu 0f861d23a2 avoid nil needle map 
fix https://github.com/chrislusf/seaweedfs/issues/1061
 2019-09-09 09:48:08 -07:00
joeslay d757ed2fe3
Merge pull request #4 from chrislusf/master 
merge seaweed master
 2019-09-06 10:49:27 +01:00
Chris Lu 803345e6af remove glide 2019-09-06 00:15:59 -07:00
j.laycock 63acc8c972 rename mem_buffer to mBuffer 2019-09-05 13:46:29 +01:00
j.laycock 44ae041e80 rename more memorymapped variables to memoryMapMaxSizeMB 2019-09-05 13:42:23 +01:00
j.laycock 5885ab67b3 rename volume property MemoryMap to MemoryMapMaxSizeMB 2019-09-05 11:56:48 +01:00
j.laycock 5a6dfb05fc Only use truncate flag when creating non memory map files, rename and use memory map size for reserving max file size 2019-09-05 11:46:12 +01:00
j.laycock f6146d6ad2 use 0 instead of false for max memory flag 2019-09-05 11:43:18 +01:00
j.laycock 4257582db5 Allocate in 16MB chunks, make creation of memory maps always aligned to 16MB chunks 2019-09-05 11:15:01 +01:00
j.laycock 84f2dc3b53 Remove test code 2019-09-05 11:14:02 +01:00
Chris Lu 20d90dea5a filer: avoid hard-coded upload timeout 2019-09-05 01:26:59 -07:00
Tom Maxwell 4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
joeslay 6ee65356e3
Merge pull request #3 from chrislusf/master 
merge seaweed master
 2019-09-04 14:58:04 +01:00
j.laycock cc756ddd79 Rename End_of_file variable, implement platform indepenent memory_map.go 2019-09-04 14:24:17 +01:00
j.laycock 9aa0859697 Remove inmemory from vaccum stuff 2019-09-03 18:20:45 +01:00
j.laycock de5a7e60a2 remove temp hack 2019-09-03 18:20:09 +01:00
j.laycock cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
Chris Lu ee8bcb0c12 remove println 2019-09-03 09:22:56 -07:00
j.laycock 1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
j.laycock 9a459d984b Do not vacuum memory mapped files 2019-09-03 15:05:43 +01:00
Chris Lu 89a0cb2e11 1.43 2019-09-03 01:31:58 -07:00
j.laycock 0123c7a898 Change to max 2GB for now 2019-09-02 17:46:22 +01:00
joeslay e91139348e
Merge pull request #2 from chrislusf/master 
merge seaweed master
 2019-09-02 11:39:40 +01:00
j.laycock 595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
Chris Lu 60c9215a00 HCFS: avoid verbose file not found exception stack trace 2019-09-01 21:40:26 -07:00
j.laycock 1e62a2b233 Fix memory not being unmapped 2019-08-30 16:55:33 +01:00
j.laycock 840ccdc35d Refactor to pass memory maps by reference instead of value, fix memory maps not being created properly or written to properly 2019-08-30 12:30:00 +01:00
j.laycock dc50701e7d Overload createFile function, expose File in memory_map struct, delete from memory_map map on delete 2019-08-30 12:30:00 +01:00
j.laycock 523f3a12b3 use Os.File/uintptr instead of windows.Handle in memory map struct 2019-08-30 12:30:00 +01:00
j.laycock 0e5d3b1a70 Move memory_map_windows to new folder, intercept most of the read and write calls to volume dat files 2019-08-30 12:30:00 +01:00
Chris Lu 58168a8c52 volume: truncate dat files when creating 2019-08-26 13:01:57 -07:00
joeslay 36bde84000
Merge pull request #1 from chrislusf/master 
merge changes from chrislusf/seaweed master
 2019-08-23 17:03:37 +01:00
j.laycock 1e89e719b3 Add windows memory map functions to storage package 2019-08-23 16:19:00 +01:00
Chris Lu 35d471a818
Revert "use std encoding lib" 2019-08-23 07:59:09 -07:00
Chris Lu fa160e00f3
Merge pull request #1051 from xushuxun/master 
volume: read dat files' last modified time correctly
 2019-08-20 22:25:37 -07:00
xushuxun 9af9f36b89 volume: read dat files' last modified time no matter dat files are catWrite or readonly 2019-08-21 11:26:01 +08:00
divinerapier e798e4e9c5
use std encoding lib 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-08-19 18:07:54 +08:00
xushuxun 5904d78bd4 weed backup: add ttl and replication parameter 2019-08-16 11:05:22 +08:00
Chris Lu f9c1cd7370 avoid dead lock from dataFileAccessLock 
fix https://github.com/chrislusf/seaweedfs/issues/1043
 2019-08-15 09:24:54 -07:00
xushuxun 46ae35cd7e wdclient: clear client cache when failed to connect to master leader 2019-08-15 15:13:16 +08:00
Chris Lu d829df4f59 volume: protect against nil needle map 
fix @mastak reported nil problem in https://github.com/chrislusf/seaweedfs/issues/1037
 2019-08-14 01:08:01 -07:00
Chris Lu e40634e6b4 volume: fail the volume deletion if compaction is in progress 
fix https://github.com/chrislusf/seaweedfs/issues/1035
 2019-08-12 00:53:50 -07:00
Chris Lu 6f75df8660 server: fix starting default master list 2019-08-10 03:10:03 -07:00
Chris Lu 0dc43ecdca skip writing content if not modified 
related to https://github.com/chrislusf/seaweedfs/issues/1036
 2019-08-09 14:38:54 -07:00
Chris Lu b9f6236f8a filer: fix typo 
fix https://github.com/chrislusf/seaweedfs/issues/1031
 2019-08-06 00:42:19 -07:00
Chris Lu c79274f7eb notes on supported meta data store 2019-08-05 16:45:21 -07:00
Feng Ye fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu ca2b81a944 fix test error 2019-07-31 02:09:13 -07:00
Chris Lu 19360ab10a master: redirect clients to the new leader 2019-07-31 01:54:42 -07:00
Chris Lu 8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu 2c6cf72e73 refactoring 2019-07-28 01:55:05 -07:00
Chris Lu fb90d63589 shell: Added minor changes to fs.meta.save 
* added an option to set the output file name
* added an option to see verbose output, default to non-verbose mode

related to: https://github.com/chrislusf/seaweedfs/issues/1019
 2019-07-26 09:35:22 -07:00
divinerapier 9e3b77c3ad avoid double warpping 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-07-24 16:41:45 +08:00
Chris Lu 344ce90315 remove weedfuse 2019-07-24 00:42:12 -07:00
Chris Lu 5956dfd08d mount: add umask option 
related to https://github.com/chrislusf/seaweedfs/issues/978
 2019-07-24 00:03:05 -07:00
Chris Lu 3f851feb59 1.42 2019-07-21 21:52:48 -07:00
Chris Lu 862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu 79762385bd master: ensure only one exclusive vacuum process 
fix https://github.com/chrislusf/seaweedfs/issues/1011
 2019-07-21 21:49:10 -07:00
Chris Lu f3b99cbfe5 fix wrong alignment fix 2019-07-21 13:50:42 -07:00
Chris Lu 644c5c27b4 refactoring 2019-07-21 13:50:24 -07:00
Chris Lu 33b3366043 volume: avoid compacting wait group 2019-07-21 13:49:59 -07:00
Chris Lu 4b15c8f0c4 volume: lock writables changes 2019-07-21 13:49:09 -07:00
Chris Lu fd16adbde1 stricter checking before writes 2019-07-21 11:21:30 -07:00
Chris Lu 5a135fef87 purge old code 2019-07-21 10:45:36 -07:00
Chris Lu c54d9221b9 write requests also checks cookie if overwrites 
protect against edge cases, avoid https://github.com/chrislusf/seaweedfs/issues/1014
 2019-07-17 23:57:34 -07:00