gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3235 commits 28 branches 278 tags 135 MiB
Commit graph

1804 commits

Author SHA1 Message Date
Chris Lu 1dc30214cb mark encryptVolumeData as not ready 2020-03-08 17:02:47 -07:00
Chris Lu 9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu 11fceaf2f7 refactoring 2020-03-07 18:09:30 -08:00
Chris Lu 1a75fbfce0 refactoring 2020-03-07 18:06:48 -08:00
Chris Lu afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu 8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu 936e7cdbfb pass in filer.copy cipher option 2020-03-07 12:46:00 -08:00
Chris Lu 9228ff192c encryption works 2020-03-07 11:08:57 -08:00
Chris Lu 0375ce2c2e filer: set mime type from volume server 2020-03-07 07:25:15 -08:00
Chris Lu 1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu 564629444b format 2020-03-07 06:12:20 -08:00
Chris Lu 8c3e25b38a mimee type default "application/octet-stream" to empty string 2020-03-07 06:08:08 -08:00
Chris Lu dba35404e4 filer: HEAD response add content type 2020-03-07 06:08:08 -08:00
Chris Lu ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu e3b8bf5588
Merge pull request #1219 from song-zhang/master 
schedule new volume by free volume number of nodes
 2020-03-07 05:54:06 -08:00
zhangsong 40f70481cd schedule new volume by free volume number of nodes 2020-03-07 21:41:42 +08:00
Chris Lu 13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu 31c481e3fc fix typo 2020-03-05 16:00:20 -08:00
Chris Lu 465d18930b filer redis: directory listing follow ttl 
fix https://github.com/chrislusf/seaweedfs/issues/1217
 2020-03-05 10:35:21 -08:00
Chris Lu e031605248 purge code 2020-03-04 00:44:27 -08:00
Chris Lu f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu 9b7dae8037 filer: fix uploaded chunked file size reporting bug. 2020-03-02 20:27:14 -08:00
Chris Lu 7335e62199 volume: PUT also conditionally gzip compress 2020-03-01 23:37:02 -08:00
Chris Lu 410bce3925 go fmt 2020-03-01 22:39:08 -08:00
Chris Lu dc40413847 fix error 2020-03-01 22:21:13 -08:00
Chris Lu 6a8484b4ae master able to list all master clients by type 2020-03-01 22:13:47 -08:00
Chris Lu 0ca68a2a6d WIP 2020-03-01 22:13:15 -08:00
Chris Lu 4532640ffd 1.58 2020-03-01 17:59:12 -08:00
Chris Lu 151114ff59 volume: fix readonly status reporting 2020-03-01 17:57:37 -08:00
Chris Lu 1df13d0d6d adjust instructions 2020-02-27 13:15:21 -08:00
Chris Lu f87f2045b3 delete old file first on windows 
fix https://github.com/chrislusf/seaweedfs/issues/1210
 2020-02-27 00:59:35 -08:00
Chris Lu 757c7d67ad avoid nil DataBackend 2020-02-27 00:07:25 -08:00
Chris Lu 555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu 430b5a49e2 unused code 2020-02-26 19:35:00 -08:00
Chris Lu 8c1da71402 remove dead code 2020-02-26 17:27:49 -08:00
Chris Lu ed0acd1722 go fmt 2020-02-26 16:52:57 -08:00
Chris Lu 37a3628b2e grpc add PermitWithoutStream to client and server options 2020-02-26 16:51:38 -08:00
Chris Lu 556dd76303 s3: wait to connect to filer 2020-02-26 16:49:47 -08:00
Chris Lu 0156e2975a mount: add mode to run external to SeaweedFS container cluster 2020-02-26 16:46:01 -08:00
Chris Lu 543cf1c80e fix compilation error 2020-02-25 23:59:30 -08:00
Chris Lu b1f377f822 mount: fail fast when connecting to filer 
fix https://github.com/chrislusf/seaweedfs/issues/1034
 2020-02-25 23:51:34 -08:00
Chris Lu 36d6595658 remove ctx 2020-02-25 22:38:27 -08:00
Chris Lu fd9612d66e remove ctx 2020-02-25 22:37:54 -08:00
Chris Lu 86cce3eb58 fix test 2020-02-25 22:29:01 -08:00
Chris Lu 97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu 892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu bd3254b53f adjust logging 2020-02-25 17:24:08 -08:00
Chris Lu 0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu 35dde56711 refactoring 2020-02-25 12:58:45 -08:00
Chris Lu 986d63cd5e s3: increase list objects or parts limit from 1000 to 10000 
10x of Amazon S3 limits
 2020-02-25 11:45:40 -08:00
Chris Lu 7d10fdf737 fix directory lookup nil 2020-02-25 11:13:06 -08:00
Chris Lu e86da5a491 minor 2020-02-25 00:42:48 -08:00
Chris Lu 0644d63748 shell: add commands for bucket 2020-02-24 23:30:01 -08:00
Chris Lu 6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu 5bcb44eda9 filer: move buckets folder to filer.toml since it should not be changed often 2020-02-24 14:42:57 -08:00
Chris Lu d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu b06b7ca6e6 adjust UI 2020-02-23 22:01:03 -08:00
Yoni Nakache 0d60bb4427
display bug, header is Free but data was Used 2020-02-24 07:47:27 +02:00
Chris Lu e96f85ed8a make pb files 2020-02-23 18:06:34 -08:00
Chris Lu 346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server 
Update stats and status volume server
 2020-02-23 18:05:25 -08:00
Chris Lu bbb163c5ad add for https://github.com/chrislusf/seaweedfs/pull/1207 2020-02-23 18:04:31 -08:00
LazyDBA247-Anyvision 4ff513d64d staus route: add DiskStatuses for disk in the volume server status 
whem monitoring server, better to know the status of the disks & volumes in a single route.
 2020-02-23 23:27:09 +02:00
LazyDBA247-Anyvision ab91dbe1d7 fillInDiskStatus: update function to calculate new stats 2020-02-23 23:25:56 +02:00
LazyDBA247-Anyvision ec8619176a volume server protobuf: add precentFree & percentUsage to the DiskStatus 2020-02-23 23:24:24 +02:00
Yoni Nakache cd4c9a365b
DiskStats: adding Total & Percent Usage 
making relevant data visible and readable.
 2020-02-23 22:33:47 +02:00
Chris Lu 6ea394b802 1.57 2020-02-22 21:34:49 -08:00
Chris Lu 4ed6b584e2 s3: access control limited by bucket 2020-02-22 21:34:18 -08:00
Chris Lu e83bfd0a35 adjust log level 2020-02-22 21:23:30 -08:00
Chris Lu 48b7ad5fa8 s3: deny anonymous type 2020-02-22 14:01:04 -08:00
Chris Lu f55f49970f move volume server stats to grpc 2020-02-21 21:45:03 -08:00
Chris Lu 3a1d017de2 shell: ensure dc and rack does not change for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1203
 2020-02-21 21:23:25 -08:00
Chris Lu 621cdbdf58 filer: avoid possible timeouts for updates and deletions 2020-02-20 15:44:17 -08:00
Chris Lu 45156cc2fe add test cases for replication=011 2020-02-20 01:21:11 -08:00
Chris Lu b07089cff2 webdav out of "unstable" 2020-02-18 19:38:59 -08:00
Chris Lu dbabdd418e add unhandled request auth type 
fix

2020-02-18 11:43:57.396699 I | http: panic serving 172.28.0.43:50658: runtime error: invalid memory address or nil pointer dereference
goroutine 595 [running]:
net/http.(*conn).serve.func1(0xc0001fe3c0)
    /usr/lib/go/src/net/http/server.go:1767 +0x13b
panic(0x55c4e35f3820, 0x55c4e48b3c40)
    /usr/lib/go/src/runtime/panic.go:679 +0x1b6
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).authRequest(0xc0004b84e0, 0xc000115900, 0xc0000bb650, 0x1, 0x1, 0x55c4e399d740)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:143 +0x11c
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:111 +0x5e
net/http.HandlerFunc.ServeHTTP(0xc0004b87e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /usr/lib/go/src/net/http/server.go:2007 +0x46
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0004ba000, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /root/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe4
net/http.serverHandler.ServeHTTP(0xc00011e0e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /usr/lib/go/src/net/http/server.go:2802 +0xa6
net/http.(*conn).serve(0xc0001fe3c0, 0x55c4e399d680, 0xc000894180)
    /usr/lib/go/src/net/http/server.go:1890 +0x877
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:2927 +0x390
 2020-02-18 09:16:04 -08:00
Chris Lu 443a68484a 1.56 2020-02-17 12:54:45 -08:00
Chris Lu 0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl 
Add s3 ACL
 2020-02-17 12:53:30 -08:00
Chris Lu 4cdde5f569 configuration stores the identity list 2020-02-17 12:31:59 -08:00
Chris Lu 9232d3ac68 ignore error when counting closed connections 2020-02-14 10:28:02 -08:00
Chris Lu 96c1ae8471 refactoring the close http response 2020-02-14 09:46:36 -08:00
Chris Lu cf5064d702 properly close http response 2020-02-14 09:09:15 -08:00
Chris Lu 9b6296e77a volume: add grpc file read operation 
This is added more for performance benchmarking.
 2020-02-14 00:37:32 -08:00
panyc16 7b3764fd9e make loadExistingVolume() return bool 2020-02-13 15:19:00 +08:00
Chris Lu 62c34454d8 1.55 2020-02-11 23:00:31 -08:00
Chris Lu 83836f4299 filer: recursively batch delete file chunks 
fix https://github.com/chrislusf/seaweedfs/issues/1197
 2020-02-11 22:54:10 -08:00
Chris Lu 2a6db0fd43 volume: return 204 for unchanged file uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1196
 2020-02-11 09:45:02 -08:00
Chris Lu 33b3bd467c Revert "HEAD operation changes to fasthttp" 
This reverts commit 58f126fd27.
 2020-02-10 13:43:53 -08:00
Chris Lu 58f126fd27 HEAD operation changes to fasthttp 2020-02-10 09:13:29 -08:00
Chris Lu 29945fad51 1.54 2020-02-09 18:07:17 -08:00
Chris Lu b4abe3c081 unused 2020-02-09 18:02:17 -08:00
Chris Lu f3ce3166ad add streaming v4 2020-02-09 17:42:17 -08:00
Chris Lu b90ad6f452 add v2 support 2020-02-09 16:02:05 -08:00
Chris Lu e7b63d50b1 add s3.config to server.go 2020-02-09 14:31:51 -08:00
Chris Lu 9ed364f053 support acl 2020-02-09 14:30:02 -08:00
Chris Lu 8d94564f41 refactor 2020-02-04 21:16:34 -08:00
Chris Lu c1288e9eb4 volume: sdx generation uses memdb instead of compactMap 
fix https://github.com/chrislusf/seaweedfs/issues/1194
 2020-02-04 21:12:09 -08:00
Chris Lu 08e4702542 sync before closing index file 
merge from 0181f87d91
 2020-02-04 10:38:12 -08:00
Chris Lu 3a35632d58 sync before closing file 
merge changes from 1747fc2d52
 2020-02-04 10:37:14 -08:00
Chris Lu dc786a63ac master: add configurable volume growth toml setting 2020-02-03 18:15:16 -08:00
Chris Lu 382ff218d3 filer: set file size, streaming chunk file uploading 
fix https://github.com/chrislusf/seaweedfs/issues/1193
 2020-02-03 17:04:06 -08:00
Chris Lu 40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu fb19263a71 fix build 2020-01-31 00:59:48 -08:00
Chris Lu beb0a1599e 1.53 2020-01-31 00:32:09 -08:00
Chris Lu b2743afaee remove println 2020-01-31 00:21:18 -08:00
Chris Lu a80ecbfe84 s3: add s3 copy 
fix https://github.com/chrislusf/seaweedfs/issues/1190
 2020-01-31 00:11:12 -08:00
Chris Lu 3b043ead49
Merge pull request #1189 from eryx67/compression-wav 
Support wav files compression
 2020-01-29 22:37:35 -08:00
Chris Lu abf90ad7b7 remove testing for viper 2020-01-29 21:38:53 -08:00
Chris Lu 20b1fb9ab7 fix test 2020-01-29 21:24:06 -08:00
eryx67 2a87087590 support wav files compression 2020-01-30 10:11:47 +05:00
Chris Lu 8925f3305d adjust example 2020-01-29 09:11:07 -08:00
Chris Lu d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu 27b94cb65b fix wrong url 
fix https://github.com/chrislusf/seaweedfs/issues/1187
 2020-01-28 00:49:47 -08:00
Chris Lu f724057051 reduce logs 2020-01-27 21:20:41 -08:00
Chris Lu c8d543ef51 avoid retry delay 2020-01-27 00:58:26 -08:00
Chris Lu 081bc1ea25 mount: able to write to any part of a file 2020-01-27 00:54:52 -08:00
Chris Lu 2f6bb57979 logging 2020-01-27 00:54:21 -08:00
Chris Lu 234f69452b 1.52 2020-01-26 21:04:40 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu 08e4b56a8a mount: able to handle large git clone 2020-01-26 13:01:11 -08:00
Chris Lu 19a05ad174 add test cases 2020-01-26 12:47:23 -08:00
Chris Lu 835da19c09 add logging 2020-01-26 00:50:18 -08:00
Chris Lu c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu ccf3859152 log 2020-01-25 08:16:43 -08:00
Chris Lu 4433c7bdca logs 2020-01-25 07:34:09 -08:00
Chris Lu 8cccccce9f mount: reuse the entry object for the new directory 
avoid mkdir and then query for the same directory
reduces these "context canceled" issues

attr read directory:"/seaweedfs/other/java/s3copier/src/main/java/com" name:"seaweedfs" : rpc error: code = Canceled desc = context canceled
 2020-01-25 01:15:54 -08:00
Chris Lu 90e30c0170 log 2020-01-25 00:32:18 -08:00
Chris Lu 8a42aa8221 adjust logging 2020-01-25 00:31:53 -08:00
Chris Lu 28d1b34856 logging 2020-01-24 22:13:06 -08:00
Chris Lu 6e25acc681 add logging 2020-01-24 22:01:51 -08:00
Chris Lu b6e6ca8595 adjust logs 2020-01-24 20:06:58 -08:00
Chris Lu d4cde5df49 return empty response if not found 2020-01-24 18:07:34 -08:00
Chris Lu 3cdb27fafc adjust logging 2020-01-24 17:55:39 -08:00
Chris Lu 6107509c64 adjust logging 2020-01-24 17:26:18 -08:00
Chris Lu e6f9f5da3a adjust logging 2020-01-24 17:26:03 -08:00
Chris Lu 24f62ebe76 minor changes 2020-01-24 10:02:53 -08:00
Chris Lu 5f1109f143 reduce log 2020-01-24 01:41:51 -08:00
Chris Lu 9d93c83654 mount: return IO error for mkdir 2020-01-24 01:41:31 -08:00
Chris Lu 107e8a56ea retry context canceled request 2020-01-24 01:40:51 -08:00
Chris Lu 2f75264ec7 mount: avoid use uint64 for greater than zero test 2020-01-23 22:12:57 -08:00
Chris Lu c936a12afa mount: update the file size so file Attr() can read 
if file Attr() during file writes, the OS, at least for Mac, may try to resend the data to write, which usually are empty.
 2020-01-23 21:59:58 -08:00
Chris Lu c2e589f202 mount: better combines connected intervals to write to volume servers 2020-01-22 23:00:04 -08:00
Chris Lu 6a5c037099 fix http range requests 2020-01-22 22:59:12 -08:00
Chris Lu 72e1aff237 add lock on dirty pages 2020-01-22 15:38:25 -08:00
Chris Lu 66ded8804a adjust logging 2020-01-22 15:37:59 -08:00
Chris Lu 6b48d246a5 mount: read data that is just written 
able read on data not flushed
multiple file open shares the same file handle

fix https://github.com/chrislusf/seaweedfs/issues/1182 on linux
 2020-01-22 13:42:03 -08:00
Chris Lu 09f4ceef3a mount: reuse previous file node 2020-01-22 11:43:43 -08:00
Chris Lu d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu 9b01a99d9a adjust logging 2020-01-21 22:45:50 -08:00
Chris Lu c8b2dac6c1 volume: avoid sharing volume dat file handle 
possibly help on https://github.com/chrislusf/seaweedfs/issues/1184
 2020-01-21 21:18:01 -08:00