gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3156 commits 28 branches 278 tags 135 MiB
Commit graph

1731 commits

Author SHA1 Message Date
Chris Lu 45156cc2fe add test cases for replication=011 2020-02-20 01:21:11 -08:00
Chris Lu b07089cff2 webdav out of "unstable" 2020-02-18 19:38:59 -08:00
Chris Lu dbabdd418e add unhandled request auth type 
fix

2020-02-18 11:43:57.396699 I | http: panic serving 172.28.0.43:50658: runtime error: invalid memory address or nil pointer dereference
goroutine 595 [running]:
net/http.(*conn).serve.func1(0xc0001fe3c0)
    /usr/lib/go/src/net/http/server.go:1767 +0x13b
panic(0x55c4e35f3820, 0x55c4e48b3c40)
    /usr/lib/go/src/runtime/panic.go:679 +0x1b6
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).authRequest(0xc0004b84e0, 0xc000115900, 0xc0000bb650, 0x1, 0x1, 0x55c4e399d740)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:143 +0x11c
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:111 +0x5e
net/http.HandlerFunc.ServeHTTP(0xc0004b87e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /usr/lib/go/src/net/http/server.go:2007 +0x46
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0004ba000, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /root/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe4
net/http.serverHandler.ServeHTTP(0xc00011e0e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /usr/lib/go/src/net/http/server.go:2802 +0xa6
net/http.(*conn).serve(0xc0001fe3c0, 0x55c4e399d680, 0xc000894180)
    /usr/lib/go/src/net/http/server.go:1890 +0x877
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:2927 +0x390
 2020-02-18 09:16:04 -08:00
Chris Lu 443a68484a 1.56 2020-02-17 12:54:45 -08:00
Chris Lu 0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl 
Add s3 ACL
 2020-02-17 12:53:30 -08:00
Chris Lu 4cdde5f569 configuration stores the identity list 2020-02-17 12:31:59 -08:00
Chris Lu 9232d3ac68 ignore error when counting closed connections 2020-02-14 10:28:02 -08:00
Chris Lu 96c1ae8471 refactoring the close http response 2020-02-14 09:46:36 -08:00
Chris Lu cf5064d702 properly close http response 2020-02-14 09:09:15 -08:00
Chris Lu 9b6296e77a volume: add grpc file read operation 
This is added more for performance benchmarking.
 2020-02-14 00:37:32 -08:00
panyc16 7b3764fd9e make loadExistingVolume() return bool 2020-02-13 15:19:00 +08:00
Chris Lu 62c34454d8 1.55 2020-02-11 23:00:31 -08:00
Chris Lu 83836f4299 filer: recursively batch delete file chunks 
fix https://github.com/chrislusf/seaweedfs/issues/1197
 2020-02-11 22:54:10 -08:00
Chris Lu 2a6db0fd43 volume: return 204 for unchanged file uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1196
 2020-02-11 09:45:02 -08:00
Chris Lu 33b3bd467c Revert "HEAD operation changes to fasthttp" 
This reverts commit 58f126fd27.
 2020-02-10 13:43:53 -08:00
Chris Lu 58f126fd27 HEAD operation changes to fasthttp 2020-02-10 09:13:29 -08:00
Chris Lu 29945fad51 1.54 2020-02-09 18:07:17 -08:00
Chris Lu b4abe3c081 unused 2020-02-09 18:02:17 -08:00
Chris Lu f3ce3166ad add streaming v4 2020-02-09 17:42:17 -08:00
Chris Lu b90ad6f452 add v2 support 2020-02-09 16:02:05 -08:00
Chris Lu e7b63d50b1 add s3.config to server.go 2020-02-09 14:31:51 -08:00
Chris Lu 9ed364f053 support acl 2020-02-09 14:30:02 -08:00
Chris Lu 8d94564f41 refactor 2020-02-04 21:16:34 -08:00
Chris Lu c1288e9eb4 volume: sdx generation uses memdb instead of compactMap 
fix https://github.com/chrislusf/seaweedfs/issues/1194
 2020-02-04 21:12:09 -08:00
Chris Lu 08e4702542 sync before closing index file 
merge from 0181f87d91
 2020-02-04 10:38:12 -08:00
Chris Lu 3a35632d58 sync before closing file 
merge changes from 1747fc2d52
 2020-02-04 10:37:14 -08:00
Chris Lu dc786a63ac master: add configurable volume growth toml setting 2020-02-03 18:15:16 -08:00
Chris Lu 382ff218d3 filer: set file size, streaming chunk file uploading 
fix https://github.com/chrislusf/seaweedfs/issues/1193
 2020-02-03 17:04:06 -08:00
Chris Lu 40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu fb19263a71 fix build 2020-01-31 00:59:48 -08:00
Chris Lu beb0a1599e 1.53 2020-01-31 00:32:09 -08:00
Chris Lu b2743afaee remove println 2020-01-31 00:21:18 -08:00
Chris Lu a80ecbfe84 s3: add s3 copy 
fix https://github.com/chrislusf/seaweedfs/issues/1190
 2020-01-31 00:11:12 -08:00
Chris Lu 3b043ead49
Merge pull request #1189 from eryx67/compression-wav 
Support wav files compression
 2020-01-29 22:37:35 -08:00
Chris Lu abf90ad7b7 remove testing for viper 2020-01-29 21:38:53 -08:00
Chris Lu 20b1fb9ab7 fix test 2020-01-29 21:24:06 -08:00
eryx67 2a87087590 support wav files compression 2020-01-30 10:11:47 +05:00
Chris Lu 8925f3305d adjust example 2020-01-29 09:11:07 -08:00
Chris Lu d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu 27b94cb65b fix wrong url 
fix https://github.com/chrislusf/seaweedfs/issues/1187
 2020-01-28 00:49:47 -08:00
Chris Lu f724057051 reduce logs 2020-01-27 21:20:41 -08:00
Chris Lu c8d543ef51 avoid retry delay 2020-01-27 00:58:26 -08:00
Chris Lu 081bc1ea25 mount: able to write to any part of a file 2020-01-27 00:54:52 -08:00
Chris Lu 2f6bb57979 logging 2020-01-27 00:54:21 -08:00
Chris Lu 234f69452b 1.52 2020-01-26 21:04:40 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu 08e4b56a8a mount: able to handle large git clone 2020-01-26 13:01:11 -08:00
Chris Lu 19a05ad174 add test cases 2020-01-26 12:47:23 -08:00
Chris Lu 835da19c09 add logging 2020-01-26 00:50:18 -08:00
Chris Lu c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu ccf3859152 log 2020-01-25 08:16:43 -08:00
Chris Lu 4433c7bdca logs 2020-01-25 07:34:09 -08:00
Chris Lu 8cccccce9f mount: reuse the entry object for the new directory 
avoid mkdir and then query for the same directory
reduces these "context canceled" issues

attr read directory:"/seaweedfs/other/java/s3copier/src/main/java/com" name:"seaweedfs" : rpc error: code = Canceled desc = context canceled
 2020-01-25 01:15:54 -08:00
Chris Lu 90e30c0170 log 2020-01-25 00:32:18 -08:00
Chris Lu 8a42aa8221 adjust logging 2020-01-25 00:31:53 -08:00
Chris Lu 28d1b34856 logging 2020-01-24 22:13:06 -08:00
Chris Lu 6e25acc681 add logging 2020-01-24 22:01:51 -08:00
Chris Lu b6e6ca8595 adjust logs 2020-01-24 20:06:58 -08:00
Chris Lu d4cde5df49 return empty response if not found 2020-01-24 18:07:34 -08:00
Chris Lu 3cdb27fafc adjust logging 2020-01-24 17:55:39 -08:00
Chris Lu 6107509c64 adjust logging 2020-01-24 17:26:18 -08:00
Chris Lu e6f9f5da3a adjust logging 2020-01-24 17:26:03 -08:00
Chris Lu 24f62ebe76 minor changes 2020-01-24 10:02:53 -08:00
Chris Lu 5f1109f143 reduce log 2020-01-24 01:41:51 -08:00
Chris Lu 9d93c83654 mount: return IO error for mkdir 2020-01-24 01:41:31 -08:00
Chris Lu 107e8a56ea retry context canceled request 2020-01-24 01:40:51 -08:00
Chris Lu 2f75264ec7 mount: avoid use uint64 for greater than zero test 2020-01-23 22:12:57 -08:00
Chris Lu c936a12afa mount: update the file size so file Attr() can read 
if file Attr() during file writes, the OS, at least for Mac, may try to resend the data to write, which usually are empty.
 2020-01-23 21:59:58 -08:00
Chris Lu c2e589f202 mount: better combines connected intervals to write to volume servers 2020-01-22 23:00:04 -08:00
Chris Lu 6a5c037099 fix http range requests 2020-01-22 22:59:12 -08:00
Chris Lu 72e1aff237 add lock on dirty pages 2020-01-22 15:38:25 -08:00
Chris Lu 66ded8804a adjust logging 2020-01-22 15:37:59 -08:00
Chris Lu 6b48d246a5 mount: read data that is just written 
able read on data not flushed
multiple file open shares the same file handle

fix https://github.com/chrislusf/seaweedfs/issues/1182 on linux
 2020-01-22 13:42:03 -08:00
Chris Lu 09f4ceef3a mount: reuse previous file node 2020-01-22 11:43:43 -08:00
Chris Lu d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu 9b01a99d9a adjust logging 2020-01-21 22:45:50 -08:00
Chris Lu c8b2dac6c1 volume: avoid sharing volume dat file handle 
possibly help on https://github.com/chrislusf/seaweedfs/issues/1184
 2020-01-21 21:18:01 -08:00
Chris Lu bb1be61602 1.51 2020-01-20 20:30:03 -08:00
Chris Lu fca6152c3a fix print format 2020-01-20 20:24:23 -08:00
Chris Lu a990ef2106 mount: fix problems found in issue 1182 
fix https://github.com/chrislusf/seaweedfs/issues/1182

always use the non-duplicated fs.Node
Forget() the fs.Node
Rename will also use the right fs.Node
Avoid using the same file handle for the same file
 2020-01-20 20:21:01 -08:00
Chris Lu 630f72f8c5 mount: able to trim a file 2020-01-20 00:00:08 -08:00
Chris Lu 1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu 2f15e93466 mount: rename also clear the cache 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-01-19 12:31:56 -08:00
Chris Lu 04019aa161 mount: deletion ignore old vid 2020-01-19 12:07:26 -08:00
Chris Lu 6a40c688e0 mount: modify file size 2020-01-19 12:07:04 -08:00
Chris Lu 39edcfde4c filer: set inode 
avoid "getcwd: No such file or directory"
 2020-01-19 12:06:19 -08:00
Chris Lu e9bb0c179f adjust logs 2020-01-18 13:25:17 -08:00
Chris Lu 061a61e018 logs 2020-01-15 19:09:00 -08:00
Chris Lu 230a0e4a33 filer: no entry if not found 2020-01-15 19:08:54 -08:00
Chris Lu e4b660699f correct comments 
fix https://github.com/chrislusf/seaweedfs/issues/1181
 2020-01-12 20:31:33 -08:00
Chris Lu d1ab16b6e3 treat it as a single node cluster if empty raft server name 
possible fix for https://github.com/chrislusf/seaweedfs/issues/1118
 2020-01-10 00:37:44 -08:00
Chris Lu 30c7148020 1.50 2020-01-08 22:08:35 -08:00
Chris Lu 8dd260623e protect against nil deletion results 
fix https://github.com/chrislusf/seaweedfs/issues/1180
 2020-01-08 21:50:59 -08:00
Chris Lu 89e16bd2e8 skip error when draining reader 
fix https://github.com/chrislusf/seaweedfs/issues/1179
 2020-01-08 18:07:07 -08:00
Chris Lu 66ab09aa4a 1.49 2020-01-08 09:51:35 -08:00
Chris Lu f8a20ef35e add the old way to compact as a comment 2020-01-08 09:45:42 -08:00
Chris Lu 288baf37fd saving .vif files correctly 2020-01-08 09:45:26 -08:00
Chris Lu acf7ca7b93 volume: fix compaction 2020-01-08 09:45:03 -08:00
Chris Lu 943f4986ef fix possible nil 2020-01-08 08:49:18 -08:00
Chris Lu 9995d3bcb5 remove println 2020-01-07 21:49:28 -08:00
Chris Lu dc9457fb87 fix compilation error 2020-01-07 09:03:47 -08:00
Chris Lu aa73364a24 weed fix: add back .idx file generation 2020-01-06 16:29:59 -08:00
Chris Lu 4e731f1c8b volume: copy volumes also include .vif file 2020-01-04 11:28:29 -08:00
Chris Lu df636e4c14 remove .vif files 2020-01-03 12:46:42 -08:00
Chris Lu 3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu 9c139e0601 update help message 2020-01-01 12:38:29 -08:00
Chris Lu 88a80ab557 fix help message 2020-01-01 12:37:38 -08:00
Chris Lu aa1807e082 1.48 2020-01-01 09:27:02 -08:00
Chris Lu 86d4b18eb5 filer: optional recursive deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1176
 2019-12-31 11:52:54 -08:00
Chris Lu 09043c8e5a fs: synchronized meta file writing 
fix https://github.com/chrislusf/seaweedfs/issues/1175
 2019-12-29 20:19:51 -08:00
Chris Lu 509f314350 keep .vif file for ec to normal volume conversion 2019-12-28 21:52:55 -08:00
Chris Lu 672868b460 always create .vif file 2019-12-28 21:52:06 -08:00
Chris Lu 03ab0b1b9a correct .vif file creation 2019-12-28 21:40:12 -08:00
Chris Lu b6f7033478 go fmt 2019-12-28 21:37:29 -08:00
Chris Lu 9134092a8d endure creating .vif files 2019-12-28 21:36:15 -08:00
Chris Lu 0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu 8fbc0a9163 fix edge cases 2019-12-28 12:59:31 -08:00
Chris Lu 37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu c06f7eb48a load volume info from .vif file, use superblock as a backup 2019-12-28 12:28:58 -08:00
Chris Lu f4a74e03d1 refactoring: separating .vif from tier file loading 2019-12-28 11:35:27 -08:00
Chris Lu 2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu e8b357fd53 rename from .tier to .vif 2019-12-28 11:17:39 -08:00
Chris Lu b3c83f9404 fix spelling 2019-12-28 11:16:10 -08:00
Chris Lu 1fd8926ac7 ignore draining error 2019-12-25 21:39:33 -08:00
Chris Lu 48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
divinerapier 4cbb6fa199
feat: drains http body if buffer is too small 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-12-26 11:28:43 +08:00
divinerapier 84640d07b7
fix: handle errors for ReadUrl 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-12-26 10:07:07 +08:00
Chris Lu 1346437d71
Merge pull request #1171 from stlpmo-jn/redundant_type_conversion 
remove the redundant type conversion
 2019-12-25 17:02:05 -08:00
Chris Lu 35393b4a02 cleanly destroy remote files 2019-12-25 16:17:58 -08:00
Chris Lu eecad6062a go fmt 2019-12-25 09:56:42 -08:00
Chris Lu d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
stlpmo f3de4b6c18 remove the redundant type conversion 2019-12-25 10:13:45 +08:00
Chris Lu 3ebeae0c0b ec encode distribute ec data and parity shards evenly 2019-12-24 16:52:21 -08:00
Chris Lu 9ff72f616a go fmt 2019-12-24 14:56:16 -08:00
Chris Lu efd2f50ede compaction changed to .idx based deletion 2019-12-24 14:55:50 -08:00
Chris Lu 7ba6be2312 volume ui add readonly status 2019-12-24 14:55:26 -08:00
Chris Lu d8b39fe92a testing 2019-12-24 11:29:26 -08:00
Chris Lu f42b5bd0f5 rename .sdb to .sdx 2019-12-24 10:19:12 -08:00
Chris Lu abffe857a1 change btree map to in memory level db 2019-12-24 10:18:56 -08:00
Chris Lu 72a561ab7c refactoring 2019-12-24 08:13:01 -08:00
Chris Lu 9114f7f432
Merge pull request #1169 from stlpmo-jn/http_304_has_no_resp_body 
remove the http 304 response body
 2019-12-24 07:08:46 -08:00
Chris Lu 409a3fe41f
Merge pull request #1168 from stlpmo-jn/optimize_DeleteCollectionFromDiskLocation 
decouple the volume.Destroy() from the operation of unmountVolume()
 2019-12-24 07:08:04 -08:00
Chris Lu 37a836c8a7
Merge pull request #1167 from stlpmo-jn/volume_server_core_dump 
resolve issue : create volume on no free space disk, the volume server core dump
 2019-12-24 07:07:41 -08:00
stlpmo f87ccbddb4 remove the http 304 response body 2019-12-24 17:56:50 +08:00
stlpmo 38e4b79125 decouple the volume.Destroy() from the operation of unmountVolume() 2019-12-24 17:20:34 +08:00
stlpmo f300c35266 resolve issue : when the host disk full, the volume server will core dump 2019-12-24 16:36:15 +08:00
Chris Lu 1ad34a2487 ed.decode prefers servers with most data shards 2019-12-24 00:00:45 -08:00
Chris Lu a18f62bbe7 only copy required shards 2019-12-23 18:06:13 -08:00
Chris Lu c7d5a0b00c adjust help message 2019-12-23 17:58:47 -08:00
Chris Lu 8a2d529759 go fmt 2019-12-23 12:48:53 -08:00