gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
8226 commits 28 branches 278 tags 135 MiB
Commit graph

248 commits

Author SHA1 Message Date
cheng.li01 dad1161c70 fix dn.volumes Iterate when write issue 2020-07-08 19:57:19 +08:00
Chris Lu e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Evgenii Kozlov 0e0db70f55 Set volumes ReadOnly if low free disk space 2020-06-05 18:18:15 +03:00
bingoohuang 1f8782a1ed try showing the first 100 volume ids and an extra ... 2020-05-29 16:15:33 +08:00
bingoohuang 1a642b9876 add Volume Ids column only for max 100 volumes for convenience in the master ui. 2020-05-29 15:37:58 +08:00
Chris Lu e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
James Hartig eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu e39e78ea8d remove println 2020-03-22 18:37:12 -07:00
Chris Lu 35208711e5 logging 2020-03-22 18:32:49 -07:00
Chris Lu c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu 560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu 7edbee6f57 volume: proxy writes to remote volume server, with replication or not 
the panic is triggered by uploading a file to a volume server not holding the designated replica.
2020-03-15 10:20:14.365488 I | http: panic serving 127.0.0.1:57124: runtime error: invalid memory address or nil pointer dereference
goroutine 119 [running]:
net/http.(*conn).serve.func1(0xc0001a8000)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1772 +0x139
panic(0x2316fe0, 0x3662900)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:973 +0x396
github.com/chrislusf/seaweedfs/weed/topology.getWritableRemoteReplications(0xc00009c000, 0x2, 0x7ffeefbffbd2, 0xe, 0x0, 0xa, 0x0, 0x0, 0xbb4bf1f7)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:157 +0x53
github.com/chrislusf/seaweedfs/weed/topology.ReplicatedWrite(0x7ffeefbffbd2, 0xe, 0xc00009c000, 0xc000000002, 0xc000472750, 0xc0001b2200, 0x0, 0x1, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:29 +0xc7
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).PostHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers_write.go:52 +0x56f
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).privateStoreHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers.go:37 +0x21f
net/http.HandlerFunc.ServeHTTP(0xc0004420e0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0001fc800, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2387 +0x1a5
net/http.serverHandler.ServeHTTP(0xc0001781c0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0001a8000, 0x2934420, 0xc000212400)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2933 +0x35c
Eg:
server A (datacenter 1) and server B (datacenter 2) hold replica (100) for volume 1.
If you upload a file with a key 1,xxxxx to server C (datacenter 3) will trigger the panic on server C.
The server C should either proxy upload file to the correct volume server or should return an HTTP error code and not panic.
 2020-03-15 02:50:42 -07:00
Chris Lu d022b6bc0e fix compilation 2020-03-14 16:32:16 -07:00
Chris Lu 0576a27f44 protect against possible nil 
which is unlikely to happen though
 2020-03-14 16:06:06 -07:00
Chris Lu 3cc9e85895 volume: vacuum pass preallocate variable 2020-03-13 16:17:44 -07:00
Chris Lu 4ce23e8809 fix compilation error 2020-03-13 15:51:38 -07:00
Chris Lu c90eb0da1f volume: handling readonly volumes after compaction 
ensure readonly volumes are not added as writable
 2020-03-13 15:41:27 -07:00
Chris Lu 14ae33d642 adjust logging 2020-03-09 10:36:03 -07:00
Chris Lu 2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu 1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu 564629444b format 2020-03-07 06:12:20 -08:00
Chris Lu e3b8bf5588
Merge pull request #1219 from song-zhang/master 
schedule new volume by free volume number of nodes
 2020-03-07 05:54:06 -08:00
zhangsong 40f70481cd schedule new volume by free volume number of nodes 2020-03-07 21:41:42 +08:00
Chris Lu 13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu 892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu 45156cc2fe add test cases for replication=011 2020-02-20 01:21:11 -08:00
Chris Lu dc786a63ac master: add configurable volume growth toml setting 2020-02-03 18:15:16 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu d1ab16b6e3 treat it as a single node cluster if empty raft server name 
possible fix for https://github.com/chrislusf/seaweedfs/issues/1118
 2020-01-10 00:37:44 -08:00
Chris Lu 09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu 4cba32d1d0 go fmt 2019-12-11 22:22:09 -08:00
zhangsong 5eb86aad47 improve data consistency when replication write 2019-12-08 22:21:35 +08:00
Chris Lu e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
zhangsong e83c36e26f fix the bug of volume never be vacuumed 2019-12-02 13:25:32 +08:00
divinerapier 5656d43264
can not break out of for-select block 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-11-20 08:25:29 +08:00
zhangsong 61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00
Lei Liu 1bcef02a6c fix dir/lookup and col/delete api 
1, Fix Layouts first letter capitalized
2, Return http 204 when delete a collection

Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 16:55:45 +08:00
Lei Liu f2f90436ef fix leader master /dir/lookup api 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-30 16:38:40 +08:00
Lei Liu c2884cace2 misc updated 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-29 21:28:28 +08:00
Chris Lu faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
j.laycock 6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
Tom Maxwell 4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
j.laycock 595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
Chris Lu 862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu 79762385bd master: ensure only one exclusive vacuum process 
fix https://github.com/chrislusf/seaweedfs/issues/1011
 2019-07-21 21:49:10 -07:00
Chris Lu 4b15c8f0c4 volume: lock writables changes 2019-07-21 13:49:09 -07:00
Chris Lu 898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu 57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu 6b08db65b0 fix shard count reporting 2019-06-05 01:58:37 -07:00
Chris Lu eaa76f11b7 free volume slots factor in ec shard counts 2019-06-04 23:41:56 -07:00
Chris Lu ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu 2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu f53024d79d delete ec collection 2019-05-30 10:40:22 -07:00
Chris Lu 1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu e941d0c2f4 list ec collections also 2019-05-30 09:17:58 -07:00
Chris Lu 3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu 2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu ac14cd5ad0 wdclient need to have ec shard info on initial dump 2019-05-26 01:01:04 -07:00
Chris Lu b58e25e588 broadcast vid->location map to master clients for ec chards 2019-05-26 00:49:15 -07:00
Chris Lu af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu 6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu 228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu 8838176d82 register ec shards to topology 2019-05-23 23:34:29 -07:00
Chris Lu 8a96445f40 register ec shards to each data node 2019-05-23 22:51:18 -07:00
Chris Lu 4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu e913ee380a add comments 2019-05-23 00:04:24 -07:00
Chris Lu 7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Wine93 32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu 440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu ebe94be1a1 maintain layout correctness when changing volumes 2019-04-21 13:32:36 -07:00
Chris Lu 132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu 316bd27f75 add remove volumes with version info 2019-04-20 23:53:37 -07:00
Chris Lu 6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
Chris Lu e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu 2f76681d62 refactor: adjust grpc API name 2019-04-10 21:41:17 -07:00
Chris Lu 0bd7ced7c2 fix test 2019-04-04 20:11:59 -07:00
Chris Lu 9cc73f4a9a fix test 2019-04-04 19:34:15 -07:00
Chris Lu 766396d249 weed master: atomic volume counting 
possible fix for https://github.com/chrislusf/seaweedfs/issues/913
 2019-04-04 19:27:00 -07:00
Chris Lu 95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu 657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
chenwanli 900aad4935 Fix https://github.com/chrislusf/seaweedfs/issues/864 2019-02-28 11:37:44 +08:00
chenwanli 0bc3cdd26a Fix https://github.com/chrislusf/seaweedfs/issues/825 2019-02-25 10:47:41 +08:00
Chris Lu e108688990 avoid grpc 5 seconds timeout 
some operations may take longer than 5 seconds.

only keep the timeout for raft operations
 2019-02-20 01:01:01 -08:00
Chris Lu 77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu 98a03b38e5 avoid util package depends on security package 2019-02-16 12:23:35 -08:00
Chris Lu 31aa60ce05 Merge branch 'master' into add_jwt 2019-02-16 08:55:38 -08:00
Chris Lu ad257ae179 simplify isLeader() logic 2019-02-16 08:55:23 -08:00
Chris Lu 7103c1ab7e go fmt 2019-02-15 00:09:48 -08:00
Sergey aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu 7a493bbefa better way to detect a master isLeader 2019-01-28 10:36:16 -08:00
bingoohuang ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang 219838db87
fail fast when failed to write local volume 2019-01-15 15:47:43 +08:00
Chris Lu 2d23d86fd3 no timeout for volume vacuum 
revert changes on volume vacuum timemout from https://github.com/chrislusf/seaweedfs/pull/829
 2019-01-10 09:07:40 -08:00
chenwanli 0a3e83a36a Set timeout for master and volume non-streaming rpc 2019-01-10 19:41:03 +08:00
Chris Lu d67f7ddfaf adjust log 2019-01-02 12:58:06 -08:00
Chris Lu 03cfb4267f adjust vaccum logging 2018-12-31 00:06:52 -08:00
Chris Lu 871dee4674 fix test 2018-11-23 12:37:09 -08:00
Chris Lu 444dfded84 add fs.FSStatfser for SeaweedFS weed mount 2018-11-23 00:24:51 -08:00
Chris Lu d64546c791 adjust logs 2018-11-22 14:31:10 -08:00
Chris Lu 4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu 9e98bba206 go fmt 2018-10-24 00:00:01 -07:00
Chris Lu 82a5d39778 add back volume vacuum cleanup 2018-10-18 23:02:58 -07:00
Chris Lu 201c24c110 add locking to volume layout vacuum 2018-10-18 20:34:43 -07:00
Chris Lu fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu e8d4be579d send initial list of volume location 2018-07-28 01:17:35 -07:00
Chris Lu a12c7b86b0 broadcast messages of new and deleted volumes 2018-07-27 23:09:55 -07:00
Chris Lu 2fd48df9d9 fix test 2018-07-11 12:52:48 -07:00
Chris Lu f7afa80250 also remove volumeLayout if collection is empty 2018-07-11 02:01:33 -07:00
Chris Lu 0d83c1b91e refactoring 2018-06-25 00:01:53 -07:00
Chris Lu 018a9a20be add test for volume registeration at master 2018-06-24 23:55:27 -07:00
Chris Lu cb19a7da99 add a note 2018-06-24 23:55:06 -07:00
Chris Lu afc1532b24 file rename 2018-06-24 23:19:56 -07:00
Chris Lu 4433b4c006 treat 0 sized file as normal 
fix https://github.com/chrislusf/seaweedfs/issues/409
 2018-06-12 21:11:26 -07:00
Chris Lu 8b0718ac92 go vet 2018-05-27 11:56:49 -07:00
tengattack 371ad2e397 fix reserve volume on node 2018-01-17 19:53:41 +08:00
Chris Lu 9026b3e86e always remember the max volume id 2017-11-28 17:08:59 -08:00
Chris Lu 58344980e4 preallocate disk space during compaction also, add cleanup for failed compaction 2017-08-29 23:59:53 -07:00
Chris Lu f7c22f0159 fix bug when collecting vacuum errors 2017-08-29 23:11:20 -07:00
Chris Lu 3a8746ca54 better error message 2017-08-10 10:26:39 -07:00
Xinran Xu 4a0939a4e4 Fix panic error when target collection is not exist. 
Fix: interface conversion: interface {} is nil, not *topology.Collection.
 2017-07-14 20:04:33 +08:00
Chris Lu 2c7dad589d stop when dn.parent is nil 
fix https://github.com/chrislusf/seaweedfs/issues/473
 2017-06-16 08:27:50 -07:00
Chris Lu a7faefd4ce fix compilation error 2017-05-22 23:04:43 -07:00
Chris Lu b3513d3167 skip only readonly volumes 
fix https://github.com/chrislusf/seaweedfs/issues/405
 2017-05-22 17:05:27 -07:00
sparklxb 4049c72e6c fix bug: replication, metadata not replicated 2017-03-04 11:24:12 +08:00
Chris Lu 30f138a538 fix tests 2017-02-12 22:18:48 -08:00
Chris Lu 41f686ab9d remove weeds.conf 2017-02-12 21:58:44 -08:00
Chris Lu e767c3ea4f disable master connection timeout 
temporarily disable master connection timeout due to heartbeat
connection timeout
 2017-01-10 01:30:00 -08:00
Chris Lu e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00
Chris Lu ed44f12f6d support Fallocate on linux 2017-01-08 11:01:46 -08:00
sparklxb da9b672d1b support additional header name-value pairs 2017-01-08 22:34:42 +08:00
sparklxb 86a7c56275 support additional header name-value pairs 2017-01-08 09:16:40 +08:00
Chris Lu 9ba52db585 add writable checking 
fix https://github.com/chrislusf/seaweedfs/issues/405
 2016-11-26 12:14:06 -08:00
Chris Lu 0559aa9673 use Lock instead of RLock 
fix https://github.com/chrislusf/seaweedfs/issues/364
 2016-09-07 18:21:05 -07:00
霍晓栋 f04d8fcbcc if replicated volume has one copy in readonly mode at one node,it should be removed from writable list 2016-08-09 20:12:39 +08:00
霍晓栋 079fdbad13 fix volume_layout bug, add new feature for volume etag write,filer pass through query params' 2016-06-29 16:05:00 +08:00
Chris Lu b617b13c43 remember oversized volumes 
fix https://github.com/chrislusf/seaweedfs/issues/331
 2016-06-27 15:28:23 -07:00
Chris Lu d0dbf6d2ea adjust logging 2016-06-26 12:49:10 -07:00
Chris Lu 96d2f11f81 adjust log 2016-06-05 09:05:03 -07:00
Chris Lu 5ce6bbf076 directory structure change to work with glide 
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
 2016-06-02 18:09:14 -07:00
Chris Lu 5071f528f6 testing compilation with remove package 2013-02-10 03:49:51 -08:00
Chris Lu 55f2627fcf testing new directory structure 2013-02-10 03:31:40 -08:00
Chris Lu ab6fb13ad7 avoid the "src" folder 2013-02-10 03:25:35 -08:00