gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3331 commits 28 branches 278 tags 135 MiB
Commit graph

1888 commits

Author SHA1 Message Date
Chris Lu a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu 826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu 4aa82c95e6 refactor 2020-03-28 13:42:35 -07:00
Chris Lu eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu 005b4ab3fe 1.69 2020-03-27 04:51:31 -07:00
Chris Lu d1439c5bd3 fix FUSE read for large files 
FUSE expects ReadAt do not return partial filled buffer with a nil error.
 2020-03-27 04:50:51 -07:00
Chris Lu f06ca04451 avoid overflow 2020-03-27 04:35:31 -07:00
Chris Lu e1911760a7 refactoring 2020-03-27 00:30:55 -07:00
Chris Lu 2f5ed29c37 clean up 2020-03-27 00:30:39 -07:00
Chris Lu 2286eda575 sampling whether the data can be gzipped 2020-03-26 23:50:48 -07:00
Chris Lu 5463560912 small test 2020-03-26 23:50:07 -07:00
Chris Lu 34a0d424f8 FUSE mount: report max volume count available disk space 2020-03-26 23:06:21 -07:00
Chris Lu a5ce359999 refactoring 2020-03-26 22:46:49 -07:00
Chris Lu 0a05052b5c 1.68 2020-03-26 22:22:36 -07:00
Chris Lu e7e8691720 add some tests 2020-03-26 22:21:52 -07:00
Chris Lu 86398d1efe S3 API: set deletion to recursive 
fix https://github.com/chrislusf/seaweedfs/issues/1251
 2020-03-26 11:05:54 -07:00
Chris Lu 1fef598347 set target Dir object when moving 2020-03-26 10:56:40 -07:00
Chris Lu 9e2e07b238 tweak 2020-03-26 10:56:18 -07:00
Chris Lu cbce793753 rename: set new name for renamed files 2020-03-26 03:30:23 -07:00
Chris Lu c763a492f5 refactor 2020-03-26 03:30:02 -07:00
Chris Lu 712720864d error checking for remove 2020-03-26 02:21:30 -07:00
Chris Lu 8d9a76005d add back running output 2020-03-26 02:14:48 -07:00
Chris Lu dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu 7f0d87b206 tree structured fs cache 
FsCache for FsNode, wrapping fs.Node
 2020-03-26 00:08:14 -07:00
Chris Lu 2e4fadd10a fix fscache move 2020-03-25 22:19:19 -07:00
Chris Lu 966df2ae27 purging skip EC volumes 2020-03-25 02:41:22 -07:00
Chris Lu 6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu 5ebc95b69b refactoring 2020-03-25 02:20:19 -07:00
Chris Lu d79f0a76b1 format output 2020-03-25 00:56:47 -07:00
Chris Lu d9b56ed88b shell: volume.fsck add options 2020-03-24 22:38:36 -07:00
Chris Lu e63a79ade8 better handle lock in case of exception 2020-03-24 18:41:25 -07:00
Chris Lu 4d5554b16f fix compilation on travis 2020-03-24 18:40:52 -07:00
Chris Lu d83e648c0b auto add prefix "fs." 2020-03-24 02:40:51 -07:00
Chris Lu a875f67a08 fix pwd printout 2020-03-24 02:40:29 -07:00
Chris Lu 00d3f01876 better output format 2020-03-24 02:34:28 -07:00
Chris Lu 0820935290 clean up 2020-03-24 02:20:37 -07:00
Chris Lu 49440cbd2f shell: add volume.fsck 
volume.fsck can find out how much space is not used by the filer
 2020-03-24 02:18:13 -07:00
Chris Lu 0f10833e0b fix compilation 2020-03-24 00:16:12 -07:00
Chris Lu 3794f2721f fix logic 2020-03-24 00:09:12 -07:00
Chris Lu 443b229ef7 disable chunk file output 2020-03-24 00:08:02 -07:00
Chris Lu bb9b97e2b4 add comments 2020-03-23 23:07:11 -07:00
Chris Lu 8047ec2f51 shell: fs.meta.save add option to export all fileIds for all files 2020-03-23 22:54:46 -07:00
Chris Lu 782d776d2a refactoring 2020-03-23 22:54:02 -07:00
Chris Lu 38e73463f1 fix du block and byte couting 2020-03-23 21:37:04 -07:00
Chris Lu b51fa81f0e fix directory checking 2020-03-23 21:36:39 -07:00
Chris Lu e666aeece2 simplify parsing filer host and port 2020-03-23 21:26:15 -07:00
Chris Lu d151185b7e shell: desupport filer url in the arguments 2020-03-23 20:46:17 -07:00
Chris Lu 40601953bf comments 2020-03-23 02:54:49 -07:00
Chris Lu daeb18be30 scaffold: fix master.toml typo 
fix https://github.com/chrislusf/seaweedfs/issues/1247
 2020-03-23 01:45:43 -07:00
Chris Lu b97768c51c refactoring 2020-03-23 01:30:22 -07:00
Chris Lu 45ee3736aa refactoring 2020-03-23 01:25:38 -07:00
Chris Lu cddcb5cafe same booleans as before 2020-03-23 01:25:23 -07:00
Chris Lu f84c288852 purge 2020-03-23 01:14:55 -07:00
Chris Lu a5aa8be19b Update webdav_server.go 2020-03-23 01:14:42 -07:00
Chris Lu 7f1e3c843d refactoring 2020-03-23 01:14:21 -07:00
Chris Lu c07bcd5065 refactoring 2020-03-23 00:30:02 -07:00
Chris Lu 654a69ff52 refactoring 2020-03-23 00:06:24 -07:00
Chris Lu c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu fbca6b29bd refactoring 2020-03-22 23:52:55 -07:00
Chris Lu e39e78ea8d remove println 2020-03-22 18:37:12 -07:00
Chris Lu c16dc57a58 fix max volume count reporting 2020-03-22 18:33:19 -07:00
Chris Lu d6412f27a0 logging 2020-03-22 18:32:56 -07:00
Chris Lu 35208711e5 logging 2020-03-22 18:32:49 -07:00
Chris Lu e32999108a add auto configured volume max count help messge 2020-03-22 16:27:15 -07:00
Chris Lu 3137777d83 volume: automatically detect max volume count 2020-03-22 16:21:42 -07:00
Chris Lu 0bf148f49d logging 2020-03-22 16:19:00 -07:00
Chris Lu bda13ed593 commented prints 2020-03-22 13:09:51 -07:00
Chris Lu 74e23e8d64 FUSE mount: rename correctly across folders 2020-03-22 13:09:33 -07:00
Chris Lu 64000f5c45 FUSE mount: rename under root 
fix https://github.com/chrislusf/seaweedfs/issues/1242
 2020-03-22 13:09:16 -07:00
Chris Lu 9f2c9b666b adjust log level 2020-03-22 01:39:08 -07:00
Chris Lu ae2ee379c0 consistent 64bit size 2020-03-22 01:37:46 -07:00
Chris Lu 2bdd936fb6 purge code 2020-03-22 01:33:40 -07:00
Chris Lu e93e986727 webdav: stream read 2020-03-22 01:30:27 -07:00
Chris Lu 65d2ea9fb0 FUSE mount: stream read data with buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1244
 2020-03-22 01:00:36 -07:00
Chris Lu 82bfad5b86 refactoring 2020-03-21 23:48:11 -07:00
Chris Lu 6ff9e2835e filer: skip resizing image if width or height larger than original image 
fix https://github.com/chrislusf/seaweedfs/issues/1239#issuecomment-602140779
 2020-03-21 22:16:00 -07:00
Chris Lu 7c111f7b75 FUSE mount: make "nonempty" optional 
https://github.com/chrislusf/seaweedfs/issues/1094
 2020-03-21 19:14:25 -07:00
Chris Lu 3b3e063f25 1.67 2020-03-20 23:42:49 -07:00
Chris Lu d3e4a31058 filer: fix where deletion can miss under high concurrency 2020-03-20 23:39:32 -07:00
Chris Lu cbfe31a9a8 idx file sync before compaction 2020-03-20 23:38:46 -07:00
Chris Lu 3505b06023 report deletion error in the log 2020-03-20 23:38:24 -07:00
Chris Lu 4bc19f1e52 master: fix connections count reporting 2020-03-20 22:41:37 -07:00
Chris Lu e1b5ac5c04 1.66 2020-03-20 20:39:48 -07:00
Chris Lu 308688c8d0 filer: add back image resizing capability 2020-03-20 20:31:11 -07:00
Chris Lu f251d03673 refactoring 2020-03-20 15:46:16 -07:00
Chris Lu 11372dd548 refactoring 2020-03-20 15:38:04 -07:00
Chris Lu c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu 165b0d22a4 1.65 2020-03-20 00:05:42 -07:00
Chris Lu 81797a059a volume: sync volume file right before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1237
 2020-03-19 23:54:52 -07:00
Chris Lu 709f231e23 tiered storage: add s3 endpoint for private s3 implementation 
fix https://github.com/chrislusf/seaweedfs/issues/1238
 2020-03-19 21:13:56 -07:00
Chris Lu d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu 6566c8e114 weed upload: add usePublicUrl option 
fix https://github.com/chrislusf/seaweedfs/issues/1236
 2020-03-18 10:50:53 -07:00
Chris Lu db9854fa0a 1.64 2020-03-17 10:58:35 -07:00
Chris Lu bec6ec7db6 go fmt 2020-03-17 10:01:55 -07:00
Chris Lu b964bbab3d fix compilation 2020-03-17 10:01:24 -07:00
Chris Lu c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu f9b3750ad1 volume: handle repeated writes response 2020-03-15 04:16:00 -07:00
Chris Lu 22400c6633 consistent gzip logic 
local store gzip same as replicated writes
 2020-03-15 04:15:40 -07:00
Chris Lu 560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu 7edbee6f57 volume: proxy writes to remote volume server, with replication or not 
the panic is triggered by uploading a file to a volume server not holding the designated replica.
2020-03-15 10:20:14.365488 I | http: panic serving 127.0.0.1:57124: runtime error: invalid memory address or nil pointer dereference
goroutine 119 [running]:
net/http.(*conn).serve.func1(0xc0001a8000)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1772 +0x139
panic(0x2316fe0, 0x3662900)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:973 +0x396
github.com/chrislusf/seaweedfs/weed/topology.getWritableRemoteReplications(0xc00009c000, 0x2, 0x7ffeefbffbd2, 0xe, 0x0, 0xa, 0x0, 0x0, 0xbb4bf1f7)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:157 +0x53
github.com/chrislusf/seaweedfs/weed/topology.ReplicatedWrite(0x7ffeefbffbd2, 0xe, 0xc00009c000, 0xc000000002, 0xc000472750, 0xc0001b2200, 0x0, 0x1, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:29 +0xc7
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).PostHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers_write.go:52 +0x56f
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).privateStoreHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers.go:37 +0x21f
net/http.HandlerFunc.ServeHTTP(0xc0004420e0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0001fc800, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2387 +0x1a5
net/http.serverHandler.ServeHTTP(0xc0001781c0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0001a8000, 0x2934420, 0xc000212400)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2933 +0x35c
Eg:
server A (datacenter 1) and server B (datacenter 2) hold replica (100) for volume 1.
If you upload a file with a key 1,xxxxx to server C (datacenter 3) will trigger the panic on server C.
The server C should either proxy upload file to the correct volume server or should return an HTTP error code and not panic.
 2020-03-15 02:50:42 -07:00