gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
1963 commits 28 branches 278 tags 135 MiB
Commit graph

716 commits

Author SHA1 Message Date
Chris Lu 60db731e36 update to 1.01 2018-11-05 08:54:40 -08:00
Chris Lu f9dbaf6292 delete file by name instead of referencing the dataFile pointer 2018-11-05 08:53:38 -08:00
alex8224 8d3f9c0b75 fix invalid point when delete collection 2018-11-05 19:53:55 +08:00
Chris Lu 1a64a817ac update to 1.00 2018-11-04 22:13:16 -08:00
Chris Lu 85d1d99459 default "weed export to current enabled store" 2018-11-04 12:59:29 -08:00
Chris Lu 6c45787a37 selectively export files 2018-11-04 12:07:33 -08:00
Chris Lu ff4189ab25 go fmt 2018-11-04 11:59:08 -08:00
Chris Lu 8dfac6a4cf working b2 sink 2018-11-04 11:58:59 -08:00
Chris Lu f050b22d6c close file in ScanVolumeFile() 
fix https://github.com/chrislusf/seaweedfs/issues/761
 2018-11-04 00:28:24 -07:00
Chris Lu 93e0dddbf3 add function description 2018-11-03 22:32:21 -07:00
Chris Lu a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu 3674ad9f8e go fmt 2018-11-01 01:12:21 -07:00
Chris Lu db584ff7f8 separate into notification.toml, add gcp pub/sub message queue 2018-11-01 01:11:09 -07:00
Chris Lu 6219a9ad1f filer ordered notifications 2018-10-31 21:48:05 -07:00
Chris Lu 4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu 200cbcde62
Merge pull request #756 from zzoe/patch-1 
Update volume_server_handlers_write.go
 2018-10-30 21:52:19 -07:00
zoe a11c28cb4c
Update volume_server_handlers_write.go 
size declared and not used
 2018-10-31 09:36:32 +08:00
Chris Lu fe31f389b0 add configuration instructions 2018-10-30 02:29:11 -07:00
zoe 29eff0ecd7
Update volume_server_handlers_write.go 
返回实际文件大小，而不是needle的大小
 2018-10-29 21:05:51 +08:00
Chris Lu 9e98bba206 go fmt 2018-10-24 00:00:01 -07:00
Chris Lu 6a756136ef go fmt 2018-10-23 23:59:49 -07:00
Chris Lu c58b73ad9b adding BackBlaze, Azure 2018-10-23 23:59:40 -07:00
Chris Lu b861899919 use released version of azblob 
replace https://github.com/chrislusf/seaweedfs/pull/754
 2018-10-23 02:18:46 -07:00
Chris Lu 82a5d39778 add back volume vacuum cleanup 2018-10-18 23:02:58 -07:00
Chris Lu 496a7398d1 adjust logging 2018-10-18 20:47:30 -07:00
Chris Lu 201c24c110 add locking to volume layout vacuum 2018-10-18 20:34:43 -07:00
Chris Lu 46eb77f9bb move DiskStatus and MemStatus to protobuf 2018-10-15 22:25:28 -07:00
Chris Lu eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu f8b2d3cacc move volume mount/unmount on volume server to grpc 2018-10-15 01:48:15 -07:00
Chris Lu 66a353dcb5 remove volume server /admin/volume/delete 2018-10-15 01:26:49 -07:00
Chris Lu fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu 333709657c adjust log level 2018-10-15 00:40:12 -07:00
Chris Lu f9410bcaa7 adjust log level for deletes 2018-10-15 00:39:56 -07:00
Chris Lu 8301519fb0 migrate delete collection to grpc API on volume server 2018-10-15 00:03:55 -07:00
Chris Lu db152ca540 fix error log format 2018-10-14 23:50:54 -07:00
Chris Lu 67d1e572ca rename files 2018-10-14 23:47:29 -07:00
Chris Lu b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu 91ac2e0dd9 go fmt 2018-10-14 00:30:20 -07:00
Chris Lu 249eeccf32 0.99 release 2018-10-14 00:19:04 -07:00
Chris Lu 9db53cf256 use the configurable chunk size 2018-10-14 00:18:52 -07:00
Chris Lu ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu 3ddcd87098 increase weed mount IO perforance 2018-10-14 00:10:17 -07:00
Chris Lu ab85118233 error handling when kafka not ready 2018-10-13 23:30:00 -07:00
Chris Lu 41aba0c02c fix when no files in filer 2018-10-12 00:45:28 -07:00
Chris Lu 4af6b4eb0e fix master static resource display problem 2018-10-12 00:05:15 -07:00
Chris Lu 556382ff5f adding basic grpc to volume server 2018-10-11 01:16:33 -07:00
Chris Lu 64ed47502b rename file 2018-10-11 00:21:32 -07:00
Chris Lu 08266b7256 go fmt 2018-10-11 00:08:13 -07:00
Chris Lu 1d7e01b754 reformat 2018-10-11 00:05:54 -07:00
Chris Lu da6154b29c refactor volume server to startVolumeServer() 2018-10-11 00:04:31 -07:00
Chris Lu 60d2f1557d refactor func name 2018-10-10 23:19:54 -07:00
Chris Lu b7763c1cb6 releasing 0.98 2018-10-10 21:44:10 -07:00
Chris Lu 9f4c2f87d3 adding filer replication sink to Azure 2018-10-09 01:35:48 -07:00
Chris Lu 453d0be4d4 adjust error format 2018-10-09 01:35:30 -07:00
Alex Wang 4db68669b6 [bugfix] Fix interrupt hook overwritten bug 2018-10-09 15:43:32 +08:00
Chris Lu 96a23591e7 remove comments 2018-10-07 22:21:00 -07:00
Chris Lu dbe33236fb embed static resources via statik 2018-10-07 21:42:57 -07:00
Chris Lu fffbc2d490 embed static resources via statik 2018-10-07 10:54:05 -07:00
Chris Lu a5b5887efc add filer replication to aws s3 2018-10-06 17:10:15 -07:00
Chris Lu 84d4a2c8a0 add filer replication to google cloud storage 2018-10-06 13:08:38 -07:00
Chris Lu f8c2704d2b reformat 2018-10-06 13:04:33 -07:00
Chris Lu 14858f6224 add logging 2018-10-06 13:01:38 -07:00
Chris Lu 04da4c8094 add logging 2018-10-06 13:01:29 -07:00
Chris Lu 4a8ef198d7 add logging 2018-10-06 13:00:33 -07:00
Chris Lu 56c5c7b1b6 add google cloud storage 2018-10-04 01:14:44 -07:00
Chris Lu e8ef501f02 add s3 replication sink 2018-10-03 23:36:52 -07:00
Chris Lu 56a5d5af8d ignore duplicated key error if the store has duplicated primary key prevention 
fix https://github.com/chrislusf/seaweedfs/issues/741
 2018-09-30 00:49:52 -07:00
Chris Lu 98384b62fd adjust description 2018-09-28 01:58:34 -07:00
Chris Lu f11d531d45 adjust logging 2018-09-28 01:54:10 -07:00
Chris Lu 06c43286cd skip error when deletion failed 2018-09-28 00:10:49 -07:00
Chris Lu 746017ea4f releasing 0.97 2018-09-27 21:51:41 -07:00
Chris Lu f7ff84f17f Revert "smalller block size for better accuracy" 
This reverts commit 97ecd20fc9.
 2018-09-27 09:42:24 -07:00
Chris Lu 97ecd20fc9 smalller block size for better accuracy 2018-09-27 09:39:17 -07:00
Chris Lu f53632a84b add block size for linux du 
du already working on Mac
 2018-09-27 09:33:36 -07:00
Chris Lu a362261972 avoid possible racing issue if leveldb or boltdb is used. 
remove the to-be-generated files during commit

fix https://github.com/chrislusf/seaweedfs/issues/738
 2018-09-26 18:45:51 -07:00
Chris Lu 31ed352ab6 replication handle cases when entry already exists 2018-09-25 09:27:03 -07:00
Chris Lu 60c1ada4c1 remove filer grpc API GetEntryAttributes 2018-09-25 09:18:06 -07:00
Chris Lu 578f8a6dec fix empty file reading on linux 2018-09-23 10:01:00 -07:00
Chris Lu 200d9b9ac2 add file name to log 2018-09-23 09:48:55 -07:00
Chris Lu a60767c43a add more instructions 2018-09-23 09:26:39 -07:00
Chris Lu 12ba8b9799 generate useable filer.toml 2018-09-23 09:26:25 -07:00
Chris Lu b1b8c4ed32 join via filepath 2018-09-23 01:46:50 -07:00
Chris Lu b16d47030e save and use last processed kafka offsets 2018-09-23 01:34:40 -07:00
Chris Lu 9fe24991d5 refactoring 2018-09-23 00:40:36 -07:00
Chris Lu 7d6b2a4740 add ETag to upload results and chunks 2018-09-22 22:12:21 -07:00
Chris Lu 420f068376 add more instructions 2018-09-22 22:12:06 -07:00
Chris Lu d3e1f20dcc adjust log level 2018-09-22 22:11:49 -07:00
Chris Lu 13d6a74eb2 consolidate file chunks during create 2018-09-22 22:11:37 -07:00
Chris Lu fdc8a267de remove unused function 2018-09-22 22:11:13 -07:00
Chris Lu 9a3d46f8e3 handle duplicated replication when retry happens 2018-09-22 11:14:04 -07:00
Chris Lu 01ceace18e adjust sink options 2018-09-22 00:53:52 -07:00
Chris Lu 402ab598b6 go fmt 2018-09-22 00:12:10 -07:00
Chris Lu 7bb62b9f21 delay file creation without chunks from dir.Create to file.Flush 2018-09-22 00:11:46 -07:00
Chris Lu db69ce89f0 go fmt 2018-09-21 01:56:43 -07:00
Chris Lu a6cfaba018 able to sync the changes 2018-09-21 01:54:29 -07:00
Chris Lu a8cd7df44a let filer choose default replication 2018-09-20 09:00:15 -07:00
Chris Lu d5149e592e add logging on kafka reading 2018-09-19 22:31:45 -07:00
Chris Lu d9b32db8d7 adjust uploads to a hidden folder 2018-09-19 22:03:16 -07:00
Chris Lu f5471bcebf add bucket name in the redirection 2018-09-19 22:01:41 -07:00
Chris Lu 25fb6f9a46 fix compilation 2018-09-17 02:23:21 -07:00
Chris Lu 779641e9d4 adjust replicated entry name 2018-09-17 01:37:24 -07:00
Chris Lu b7a375f5aa set large limit for "weed filer.export" directory listing 2018-09-17 00:42:36 -07:00
Chris Lu 4e1f1a0b92 set large limit for "weed mount" directory listing 2018-09-17 00:41:33 -07:00
Chris Lu 788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu 865a017936 fix when if buffer is not aligned 2018-09-17 00:27:40 -07:00
Chris Lu 97575e7185 do not set attributes if the file is still open 2018-09-16 12:37:06 -07:00
Chris Lu 70e67b7c6d add notification log 2018-09-16 11:20:36 -07:00
Chris Lu 5c6e3f457f adjust log 2018-09-16 11:20:08 -07:00
Chris Lu bc03233364 refactoring 2018-09-16 01:37:35 -07:00
Chris Lu d923ba2206 renaming msgqueue to notification 2018-09-16 01:18:30 -07:00
Chris Lu bea4f6ca14 go fmt 2018-09-12 01:00:57 -07:00
Chris Lu f628953c33 cleanup tmp upload directories 2018-09-12 01:00:51 -07:00
Chris Lu f6d8525d1d working S3 multipart uploads 2018-09-12 00:46:12 -07:00
Chris Lu 439e88b5a6 fix format error 2018-09-11 13:01:51 -07:00
Chris Lu 267201ff44 handle non continuous writes 
fix #728
 2018-09-10 02:39:41 -07:00
Chris Lu 109385e955 simplifying logic to avoid handling non continuous writes 
fix https://github.com/chrislusf/seaweedfs/issues/728
 2018-09-10 02:21:57 -07:00
Chris Lu 296acc4a0a go fmt 2018-09-09 16:26:11 -07:00
Chris Lu 164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu 9b3bf0e46c fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:48:58 -07:00
Chris Lu bc8d34143d Revert "fix "weed backup" rerunning" 
This reverts commit 1937be2bb2.
 2018-09-09 02:38:30 -07:00
Chris Lu 1937be2bb2 fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:31:12 -07:00
Chris Lu d7a4b12d2a fix build error 2018-09-07 14:21:10 -07:00
Chris Lu 83ca595472 adjust version number 2018-09-07 14:19:48 -07:00
Chris Lu 747377142b fix file permission during writes 2018-09-07 13:57:06 -07:00
Chris Lu dd37f0540f add extended attributes 2018-09-07 13:12:52 -07:00
Chris Lu 75e749039b fix data writes error when consecutive calls overlaps with previous writes 2018-09-07 13:11:43 -07:00
Chris Lu f177cec509 avoid slice out of bounds 
avoid this problem

2018/09/04 16:27:14 fuse: panic in handler for Write [ID=0x27c0d Node=0x2 Uid=0 Gid=0 Pid=0] 0x1 131072 @10607788032 fl=WriteCache lock=0 ffl=OpenReadOnly: runtime error: slice bounds out of range
goroutine 211141 [running]:
bazil.org/fuse/fs.(*Server).serve.func2(0x10d3e60, 0xc00014be30, 0xc00052fef8, 0xc00052fe77)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:857 +0x1ac
panic(0xe2d080, 0x17f62b0)
	/home/travis/.gimme/versions/go/src/runtime/panic.go:513 +0x1b9
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0xc0005dc000, 0x20000, 0x1000000, 0x276720000, 0xc0003feaa0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:142 +0x8ec
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveExistingPagesToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:107 +0x6c
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).AddPage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x278460000, 0xc011966050, 0x20000, 0x20fb0, 0x6fc23ac00, 0x4a817c800, 0x0, ...)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:70 +0x8f
github.com/chrislusf/seaweedfs/weed/filesys.(*FileHandle).Write(0xc000548410, 0x10d7ba0, 0xc0003fcc00, 0xc00014be30, 0xc011946af8, 0x47fa01, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/filehandle.go:141 +0x245
bazil.org/fuse/fs.(*Server).handleRequest(0xc0002cc0c0, 0x10d7ba0, 0xc0003fcc00, 0x10cb020, 0xc000394140, 0xc0000acac0, 0x10d3e60, 0xc00014be30, 0xc00052fef8, 0x10ca6a0, ...)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:1265 +0x1599
bazil.org/fuse/fs.(*Server).serve(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:878 +0x410
bazil.org/fuse/fs.(*Server).Serve.func1(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:425 +0x6e
created by bazil.org/fuse/fs.(*Server).Serve
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:423 +0x321
 2018-09-05 02:17:04 -07:00
Chris Lu d614716ff2 fix compilation error 2018-09-04 00:43:46 -07:00
Chris Lu ce6a2140a2 preparing to support S3 multipart uploads 2018-09-04 00:42:44 -07:00
Chris Lu 7f760f16b0 prepare for 0.95 2018-09-03 20:52:52 -07:00
Chris Lu 35bafd9861 remove unused function 2018-09-03 16:50:32 -07:00
Chris Lu 7ad5a90127 go fmt 2018-09-03 16:47:00 -07:00
Chris Lu 2ed6d8cca6 refactoring filer rm 2018-09-03 13:16:26 -07:00
Chris Lu 03f852c799 refactoring 2018-09-03 13:03:16 -07:00
Chris Lu 0b0ece9649 add place holders for multiplarts upload 2018-09-03 11:38:10 -07:00
Chris Lu 0adb2ba6b8 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2018-09-03 00:29:02 -07:00
Chris Lu 06b81c063a fix possible connection counting error 2018-09-03 00:28:58 -07:00
zhangmingfeng aafe8d84cb
Update volume_server_handlers_read.go 
change ext to lower-case
 2018-09-03 11:01:35 +08:00
Chris Lu 88f1d32cc4 add s3ChunkedReader 
fix https://github.com/chrislusf/seaweedfs/issues/718
 2018-09-02 14:20:47 -07:00
Chris Lu 76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu 5ccf8e8078 reformat 2018-08-24 00:30:03 -07:00
Chris Lu ac793a3c5a clean remove all left over files when deleting a collection 2018-08-23 23:33:16 -07:00
Chris Lu d361a1ddf5 fix id String() instead 2018-08-23 22:44:30 -07:00
Chris Lu 694f93de80 parse needle id correctly 
fix https://github.com/chrislusf/seaweedfs/issues/713
 2018-08-23 20:48:43 -07:00
Chris Lu 37e4e022fd prepare for 0.94 release 2018-08-23 00:02:13 -07:00
Chris Lu 6e3f4d1079 add working filer.export command 2018-08-23 00:02:04 -07:00
Chris Lu 98b8f8649d adding filer.export prototype 2018-08-22 23:54:22 -07:00
Chris Lu 2b4277ec18 fix nil bug with filer.toml 2018-08-22 23:54:00 -07:00
Chris Lu ae4e654d2a add example configuration for default redis create-cluster setup 2018-08-22 18:23:19 -07:00