Commit graph

169 commits

Author SHA1 Message Date
Chris Lu 94525aa0fd allocate volume by disk type 2020-12-13 23:08:21 -08:00
Chris Lu a9db24cd05 master allocate volumes if ssd type runs out 2020-12-13 19:44:57 -08:00
Chris Lu 0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu 715b199eeb fix tests 2020-12-13 04:14:50 -08:00
Chris Lu d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu 003b6245e7 fix nil 2020-12-02 00:09:19 -08:00
Chris Lu 965413c21b shell: add volume.vacuum command 2020-11-28 23:18:02 -08:00
Chris Lu c7ebadc25d avoid possible concurrent access inside ensureCorrectWritables() 2020-11-22 17:15:59 -08:00
Chris Lu 8cb8cd4cc5 add locks 2020-11-17 16:59:48 -08:00
Chris Lu c6bd244ebd add TODO 2020-11-11 12:51:27 -08:00
Chris Lu e0002f8dd7 check existing volumes for writable status 2020-10-24 01:34:31 -07:00
Chris Lu 720b1d9b88 adding locking to avoid nil VolumeLocationList
fix panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 00:53:44 bedb-master1 weed[8055]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x17658da]
Oct 22 00:53:44 bedb-master1 weed[8055]: goroutine 310 [running]:
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLocationList).Length(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_location_list.go:35
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).enoughCopies(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:376
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).ensureCorrectWritables(0xc000111d50, 0xc000b55438)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:202 +0x5a
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*Topology).SyncDataNodeRegistration(0xc00042ac60, 0xc001454d30, 0x1, 0x1, 0xc0005fc000, 0xc00135de40, 0x4, 0xc00135de50, 0x10, 0x10d, ...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/topology.go:224 +0x616
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/server.(*MasterServer).SendHeartbeat(0xc000162700, 0x23b97c0, 0xc000ae2c90, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/server/master_grpc_server.go:106 +0x325
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/pb/master_pb._Seaweed_SendHeartbeat_Handler(0x1f8e7c0, 0xc000162700, 0x23b0a60, 0xc00024b440, 0x3172c38, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/pb/master_pb/master.pb.go:4250 +0xad
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0xc0001fea80, 0x311fec0, 0x0, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1329 +0xcd8
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).handleStream(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1409 +0xc5c
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001ce8b0, 0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xa5
Oct 22 00:53:44 bedb-master1 weed[8055]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Failed with result 'exit-code'.
2020-10-21 23:15:48 -07:00
Chris Lu 410b818aa7 master: avoid timer leakage 2020-10-19 14:24:57 -07:00
Chris Lu c7d7b1a0f6
Merge pull request #1485 from LIBA-S/fix_oversized
Correct the oversized state of volume after compaction
2020-09-23 19:24:30 -07:00
LIBA-S eecd6b5d35 Fix a race condition when handle VolumeLocationList 2020-09-23 20:56:51 +08:00
LIBA-S 0157798ebf Correct the oversized state of volume after compaction 2020-09-23 20:27:42 +08:00
Chris Lu 289e62a305 master: better locking of in memory volume data
related to https://github.com/chrislusf/seaweedfs/issues/1436#issuecomment-695880135
2020-09-20 23:07:55 -07:00
Chris Lu 6a92f0bc7a refactoring to typed Size
Go is amazing with refactoring!
2020-08-18 17:04:28 -07:00
Chris Lu 152a6cbc2b minor adjustments 2020-08-10 20:42:27 -07:00
cheng.li01 25fbff5d52 fix bug: two same volumeId in different collections
1, there will be two leader when master server startup in a few seconds
2, raft server will get a leader even there is only one master, so there is no need to do hard code to set the server to be leader
2020-08-10 16:37:47 +08:00
cheng.li01 dad1161c70 fix dn.volumes Iterate when write issue 2020-07-08 19:57:19 +08:00
Chris Lu e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Evgenii Kozlov 0e0db70f55 Set volumes ReadOnly if low free disk space 2020-06-05 18:18:15 +03:00
bingoohuang 1f8782a1ed try showing the first 100 volume ids and an extra ... 2020-05-29 16:15:33 +08:00
bingoohuang 1a642b9876 add Volume Ids column only for max 100 volumes for convenience in the master ui. 2020-05-29 15:37:58 +08:00
Chris Lu e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
James Hartig eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu e39e78ea8d remove println 2020-03-22 18:37:12 -07:00
Chris Lu 35208711e5 logging 2020-03-22 18:32:49 -07:00
Chris Lu c3cb6fa1d7 volume: compaction can cause readonly volumes
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-17 09:43:57 -07:00
Chris Lu 560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu 7edbee6f57 volume: proxy writes to remote volume server, with replication or not
the panic is triggered by uploading a file to a volume server not holding the designated replica.
2020-03-15 10:20:14.365488 I | http: panic serving 127.0.0.1:57124: runtime error: invalid memory address or nil pointer dereference
goroutine 119 [running]:
net/http.(*conn).serve.func1(0xc0001a8000)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1772 +0x139
panic(0x2316fe0, 0x3662900)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:973 +0x396
github.com/chrislusf/seaweedfs/weed/topology.getWritableRemoteReplications(0xc00009c000, 0x2, 0x7ffeefbffbd2, 0xe, 0x0, 0xa, 0x0, 0x0, 0xbb4bf1f7)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:157 +0x53
github.com/chrislusf/seaweedfs/weed/topology.ReplicatedWrite(0x7ffeefbffbd2, 0xe, 0xc00009c000, 0xc000000002, 0xc000472750, 0xc0001b2200, 0x0, 0x1, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:29 +0xc7
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).PostHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers_write.go:52 +0x56f
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).privateStoreHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers.go:37 +0x21f
net/http.HandlerFunc.ServeHTTP(0xc0004420e0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0001fc800, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2387 +0x1a5
net/http.serverHandler.ServeHTTP(0xc0001781c0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0001a8000, 0x2934420, 0xc000212400)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2933 +0x35c
Eg:
server A (datacenter 1) and server B (datacenter 2) hold replica (100) for volume 1.
If you upload a file with a key 1,xxxxx to server C (datacenter 3) will trigger the panic on server C.
The server C should either proxy upload file to the correct volume server or should return an HTTP error code and not panic.
2020-03-15 02:50:42 -07:00
Chris Lu d022b6bc0e fix compilation 2020-03-14 16:32:16 -07:00
Chris Lu 0576a27f44 protect against possible nil
which is unlikely to happen though
2020-03-14 16:06:06 -07:00
Chris Lu 3cc9e85895 volume: vacuum pass preallocate variable 2020-03-13 16:17:44 -07:00
Chris Lu 4ce23e8809 fix compilation error 2020-03-13 15:51:38 -07:00
Chris Lu c90eb0da1f volume: handling readonly volumes after compaction
ensure readonly volumes are not added as writable
2020-03-13 15:41:27 -07:00
Chris Lu 14ae33d642 adjust logging 2020-03-09 10:36:03 -07:00
Chris Lu 2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu 1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu 564629444b format 2020-03-07 06:12:20 -08:00
Chris Lu e3b8bf5588
Merge pull request #1219 from song-zhang/master
schedule new volume by free volume number of nodes
2020-03-07 05:54:06 -08:00
zhangsong 40f70481cd schedule new volume by free volume number of nodes 2020-03-07 21:41:42 +08:00
Chris Lu 13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu 892e726eb9 avoid reusing context object
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-02-25 21:50:12 -08:00
Chris Lu 45156cc2fe add test cases for replication=011 2020-02-20 01:21:11 -08:00
Chris Lu dc786a63ac master: add configurable volume growth toml setting 2020-02-03 18:15:16 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu d1ab16b6e3 treat it as a single node cluster if empty raft server name
possible fix for https://github.com/chrislusf/seaweedfs/issues/1118
2020-01-10 00:37:44 -08:00