gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
1781 commits 28 branches 278 tags 135 MiB
Commit graph

598 commits

Author SHA1 Message Date
Chris Lu 164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu 9b3bf0e46c fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:48:58 -07:00
Chris Lu bc8d34143d Revert "fix "weed backup" rerunning" 
This reverts commit 1937be2bb2.
 2018-09-09 02:38:30 -07:00
Chris Lu 1937be2bb2 fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:31:12 -07:00
Chris Lu d7a4b12d2a fix build error 2018-09-07 14:21:10 -07:00
Chris Lu 83ca595472 adjust version number 2018-09-07 14:19:48 -07:00
Chris Lu 747377142b fix file permission during writes 2018-09-07 13:57:06 -07:00
Chris Lu dd37f0540f add extended attributes 2018-09-07 13:12:52 -07:00
Chris Lu 75e749039b fix data writes error when consecutive calls overlaps with previous writes 2018-09-07 13:11:43 -07:00
Chris Lu f177cec509 avoid slice out of bounds 
avoid this problem

2018/09/04 16:27:14 fuse: panic in handler for Write [ID=0x27c0d Node=0x2 Uid=0 Gid=0 Pid=0] 0x1 131072 @10607788032 fl=WriteCache lock=0 ffl=OpenReadOnly: runtime error: slice bounds out of range
goroutine 211141 [running]:
bazil.org/fuse/fs.(*Server).serve.func2(0x10d3e60, 0xc00014be30, 0xc00052fef8, 0xc00052fe77)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:857 +0x1ac
panic(0xe2d080, 0x17f62b0)
	/home/travis/.gimme/versions/go/src/runtime/panic.go:513 +0x1b9
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0xc0005dc000, 0x20000, 0x1000000, 0x276720000, 0xc0003feaa0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:142 +0x8ec
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveExistingPagesToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:107 +0x6c
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).AddPage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x278460000, 0xc011966050, 0x20000, 0x20fb0, 0x6fc23ac00, 0x4a817c800, 0x0, ...)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:70 +0x8f
github.com/chrislusf/seaweedfs/weed/filesys.(*FileHandle).Write(0xc000548410, 0x10d7ba0, 0xc0003fcc00, 0xc00014be30, 0xc011946af8, 0x47fa01, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/filehandle.go:141 +0x245
bazil.org/fuse/fs.(*Server).handleRequest(0xc0002cc0c0, 0x10d7ba0, 0xc0003fcc00, 0x10cb020, 0xc000394140, 0xc0000acac0, 0x10d3e60, 0xc00014be30, 0xc00052fef8, 0x10ca6a0, ...)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:1265 +0x1599
bazil.org/fuse/fs.(*Server).serve(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:878 +0x410
bazil.org/fuse/fs.(*Server).Serve.func1(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:425 +0x6e
created by bazil.org/fuse/fs.(*Server).Serve
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:423 +0x321
 2018-09-05 02:17:04 -07:00
Chris Lu d614716ff2 fix compilation error 2018-09-04 00:43:46 -07:00
Chris Lu ce6a2140a2 preparing to support S3 multipart uploads 2018-09-04 00:42:44 -07:00
Chris Lu 7f760f16b0 prepare for 0.95 2018-09-03 20:52:52 -07:00
Chris Lu 35bafd9861 remove unused function 2018-09-03 16:50:32 -07:00
Chris Lu 7ad5a90127 go fmt 2018-09-03 16:47:00 -07:00
Chris Lu 2ed6d8cca6 refactoring filer rm 2018-09-03 13:16:26 -07:00
Chris Lu 03f852c799 refactoring 2018-09-03 13:03:16 -07:00
Chris Lu 0b0ece9649 add place holders for multiplarts upload 2018-09-03 11:38:10 -07:00
Chris Lu 0adb2ba6b8 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2018-09-03 00:29:02 -07:00
Chris Lu 06b81c063a fix possible connection counting error 2018-09-03 00:28:58 -07:00
zhangmingfeng aafe8d84cb
Update volume_server_handlers_read.go 
change ext to lower-case
 2018-09-03 11:01:35 +08:00
Chris Lu 88f1d32cc4 add s3ChunkedReader 
fix https://github.com/chrislusf/seaweedfs/issues/718
 2018-09-02 14:20:47 -07:00
Chris Lu 76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu 5ccf8e8078 reformat 2018-08-24 00:30:03 -07:00
Chris Lu ac793a3c5a clean remove all left over files when deleting a collection 2018-08-23 23:33:16 -07:00
Chris Lu d361a1ddf5 fix id String() instead 2018-08-23 22:44:30 -07:00
Chris Lu 694f93de80 parse needle id correctly 
fix https://github.com/chrislusf/seaweedfs/issues/713
 2018-08-23 20:48:43 -07:00
Chris Lu 37e4e022fd prepare for 0.94 release 2018-08-23 00:02:13 -07:00
Chris Lu 6e3f4d1079 add working filer.export command 2018-08-23 00:02:04 -07:00
Chris Lu 98b8f8649d adding filer.export prototype 2018-08-22 23:54:22 -07:00
Chris Lu 2b4277ec18 fix nil bug with filer.toml 2018-08-22 23:54:00 -07:00
Chris Lu ae4e654d2a add example configuration for default redis create-cluster setup 2018-08-22 18:23:19 -07:00
Chris Lu b23883b3b6 fix a bug on error reporting 2018-08-22 18:19:26 -07:00
Chris Lu 1f030af06f adjusting startup logging 2018-08-22 13:59:04 -07:00
Chris Lu bb92fd1db0 fix test error 2018-08-19 18:52:50 -07:00
Chris Lu 53d171cf5b releasing 0.93 2018-08-19 18:45:26 -07:00
Chris Lu 906472b21d filer add upload and drag&dorp 2018-08-19 18:42:40 -07:00
Chris Lu f22d6b87e1 go fmt 2018-08-19 15:42:50 -07:00
Chris Lu 748870b03f add "weed scaffold" command 2018-08-19 15:36:30 -07:00
Chris Lu 95b997fb35 go fmt 2018-08-19 15:18:37 -07:00
Chris Lu f827ada811 merge notification config with filer.toml 2018-08-19 15:17:55 -07:00
Chris Lu c91372daa6 add one example filer.toml 2018-08-19 14:58:24 -07:00
Chris Lu ee920d4f83 kafka set the right topic 2018-08-19 01:27:30 -07:00
Chris Lu 708acee502 add support for redis cluster 
fix https://github.com/chrislusf/seaweedfs/issues/705
 2018-08-15 00:01:38 -07:00
Chris Lu 9b51ed7af2 fix test 2018-08-13 01:38:18 -07:00
Chris Lu ae9b81c18f fix tests 2018-08-13 01:33:21 -07:00
Chris Lu 5c467083e5 go fmt 2018-08-13 01:22:32 -07:00
Chris Lu f036ef8a3c add filer notification 2018-08-13 01:20:49 -07:00
Chris Lu 75d63db60d randomize raft server startup 
also some go fmt
 2018-08-12 14:27:14 -07:00
Chris Lu eca4b928d2 ensure master server count is odd 2018-08-12 14:25:31 -07:00
Chris Lu b931ced0a3 readable file size 2018-08-10 23:47:31 -07:00
Chris Lu 2e427c66ac small refactoring 2018-08-10 23:47:17 -07:00
Chris Lu c81f1cda47 correctly recursively delete folders 2018-08-01 01:26:41 -07:00
Chris Lu 495a776671 support 128 bit NeedleId 2018-07-31 23:25:26 -07:00
Chris Lu 44d8e35988 add listener for volumd id location for benchmark tool 2018-07-31 19:12:36 -07:00
Chris Lu d09606c65b refactoring 2018-07-29 02:25:24 -07:00
Chris Lu 032f1226db fix compilation error 2018-07-29 02:25:03 -07:00
Chris Lu d3205a0070 go fmt 2018-07-28 21:02:56 -07:00
Chris Lu 7214a8e265 fix init error 2018-07-28 18:40:31 -07:00
Chris Lu cfbfc7cb67 fix compilation error 2018-07-28 18:34:15 -07:00
Chris Lu 4e0522a80c adjust API 2018-07-28 18:17:31 -07:00
Chris Lu 888eb2abb5 filer read write all via locations from MasterClient 2018-07-28 14:51:36 -07:00
Chris Lu 1d779389cb MasterClient replicates all vid locations 2018-07-28 14:22:46 -07:00
Chris Lu 01bcc89803 refactor into MasterClient 2018-07-28 02:10:32 -07:00
Chris Lu 1ab8232b55 filer only talk to the master leader 2018-07-28 01:30:03 -07:00
Chris Lu e8d4be579d send initial list of volume location 2018-07-28 01:17:35 -07:00
Chris Lu a12c7b86b0 broadcast messages of new and deleted volumes 2018-07-27 23:09:55 -07:00
Chris Lu f82ac793b4 Merge branch 'master' into add_topo_listener 2018-07-27 02:26:43 -07:00
Chris Lu 10bb76fe6b go fmt 2018-07-27 02:10:42 -07:00
Chris Lu 4d322df95e avoid reparsing the multipart form 2018-07-27 02:10:10 -07:00
Chris Lu 0d98949199 tmp commit 2018-07-27 01:54:45 -07:00
Chris Lu e4b7e31902 add back recursively delete a folder 2018-07-24 22:33:26 -07:00
Chris Lu 57ded4fd80 speed up leveldb bolddb loading 2018-07-24 22:17:56 -07:00
Chris Lu ea8ecf35f4 set needle AppendAtNs value during writes 2018-07-24 02:44:33 -07:00
Chris Lu 655b449270 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2018-07-24 02:20:02 -07:00
Chris Lu 0fb4f59fa6
Merge pull request #697 from chrislusf/add_version3 
Add version3
 2018-07-24 02:19:06 -07:00
Chris Lu 69d1b88b9e starting on 0.93 beta 2018-07-24 02:18:39 -07:00
Chris Lu e8a5ee4f48 releasing 0.92 
ensure the nil problem is resolved.
 2018-07-24 01:54:15 -07:00
Chris Lu 77fadab560 go fmt 2018-07-24 01:38:08 -07:00
Chris Lu 4f317c7e3d working version3 2018-07-24 01:36:04 -07:00
Chris Lu 0b99557c29 fix compilation error 2018-07-24 01:09:32 -07:00
Chris Lu 9bf93e698e refactoring: avoid common code 2018-07-24 00:58:52 -07:00
Chris Lu aae77e8ea3 initialize read seeker 2018-07-24 00:54:15 -07:00
Chris Lu 5a237af973 fix nil if not images 2018-07-24 00:49:53 -07:00
Chris Lu a9069520ae start 0.92 beta2 2018-07-23 01:59:23 -07:00
Chris Lu 04356ad141 re-releasing 0.91 2018-07-23 01:58:13 -07:00
Chris Lu 6039b25bc9 s3API working with listing objects 2018-07-23 01:55:26 -07:00
Chris Lu 3c7ffa2a2c go fmt 2018-07-23 01:16:17 -07:00
Chris Lu ebad3a44ab s3api do not proxy directory requests 2018-07-23 01:15:59 -07:00
Chris Lu 7f32eb1e25 s3 add https support 2018-07-22 21:28:54 -07:00
Chris Lu c133e72f62 print out unsupported methods 2018-07-22 19:55:53 -07:00
Chris Lu ff331178aa
Merge pull request #695 from zhangmingfeng/master 
增加chunk图片文件支持width和height
 2018-07-22 19:54:04 -07:00
Chris Lu 7e76a7d21b
Merge pull request #694 from himyxia/inprove_switch_case 
better way to handler switch case
 2018-07-22 10:29:37 -07:00
Jimmy 1c77d323b0 better way to handler switch case 2018-07-23 01:27:10 +08:00
Chris Lu 5e55f0da4d add WIP 0.92 beta 2018-07-22 02:15:24 -07:00
Chris Lu 19c537b687 releasing 0.91 2018-07-22 02:12:02 -07:00
Chris Lu 20853972b2 s3 API add ListObjectsV2Handler 2018-07-22 02:04:07 -07:00
Chris Lu 056871242b remove println 2018-07-22 01:17:49 -07:00
Chris Lu 0655151b8c go fmt 2018-07-22 01:15:11 -07:00
Chris Lu 6319d84f42 s3 API add ListObjectsV1 2018-07-22 01:14:36 -07:00
Chris Lu 922c614bde fix compilation problem 2018-07-21 20:42:14 -07:00
Chris Lu 16c66e097b adjust default buckets folder 2018-07-21 20:29:33 -07:00
Chris Lu 49375d6031 adjust DELETE response to S3 format 2018-07-21 19:12:44 -07:00
Chris Lu 5068b6ae7d fix compilation 2018-07-21 19:06:09 -07:00
Chris Lu 12825938d7 unused method 2018-07-21 18:50:56 -07:00
Chris Lu 8f240276db s3 add HEAD DELETE 2018-07-21 18:49:47 -07:00
Chris Lu bfa87b4f50 adjust log level 2018-07-21 18:47:23 -07:00
Chris Lu 648626115a fix printing 2018-07-21 18:47:07 -07:00
Chris Lu 12e6ccc581 remove unused code 2018-07-21 17:54:14 -07:00
Chris Lu 702d7ac424 delegating old chunk purging to CreateEntry 
1. filer add file user id and group id
2. auto-chunking set the file name correctly
3. delegating old chunk purging to CreateEntry
 2018-07-21 17:47:59 -07:00
Chris Lu 852af28f91 needle priting format 2018-07-21 17:41:21 -07:00
Chris Lu 6f30a78a6c update also delete old chunks if different from latest chunks 2018-07-21 17:40:00 -07:00
Chris Lu 7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu dc8dd8b486 Merge branch 'master' into add_s3 2018-07-21 15:59:18 -07:00
Chris Lu c98df05ed0 support PUT 2018-07-21 15:58:48 -07:00
Chris Lu feb8eeb830 refactor: separate multipart parsing file 2018-07-21 12:01:35 -07:00
Chris Lu 8480008a9a add s3 upload, and removing mono and multi part upload analyzer 
removing mono and multi part upload analyzer, which were used just to determine the file name
 2018-07-21 10:39:02 -07:00
Chris Lu 80d80daf64 set filer2.ErrNotFound for not found entry 2018-07-20 02:14:18 -07:00
Chris Lu dad733086a fix string printing 2018-07-20 00:58:10 -07:00
Chris Lu e73b859b1d fix compilation error 2018-07-20 00:46:02 -07:00
Chris Lu 568132c0c9 delete bucket also removes collection data 2018-07-20 00:10:01 -07:00
Chris Lu 13e5541e17 FUSE can change file or folder attributes 
FUSE can change file or folder attributes
 2018-07-19 02:17:36 -07:00
Chris Lu a09ef6002a adjust name to avoid confusion 2018-07-19 01:46:18 -07:00
Chris Lu feea33d5e0 add HeadBucket 2018-07-19 01:43:27 -07:00
Chris Lu d429b6296b fix test 2018-07-19 01:27:45 -07:00
Chris Lu c7228fefa7 add bucket creation and deletion 
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
 2018-07-19 01:21:44 -07:00
Chris Lu 702fbdf731 refactoring 2018-07-18 02:39:12 -07:00
Chris Lu 834a25f084 add list all my buckets 2018-07-18 02:37:09 -07:00
Chris Lu 7abfab8e77 add feature to mount a specific filer path to local directory 2018-07-14 21:09:21 -07:00
Chris Lu 3edfe1d28f extend export command to show tombstone + change output format to CSV 
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
 2018-07-14 20:51:17 -07:00
Chris Lu 85eae57e4c Store and replicate timestamp of file deletion in tombstone marker 
merging changes from https://github.com/chrislusf/seaweedfs/pull/609/files
 2018-07-14 20:26:45 -07:00
Chris Lu 67763fb375 fix data center setting error 2018-07-14 14:09:45 -07:00
Chris Lu 6d58916cfd fix compilation error 2018-07-14 13:39:57 -07:00
Chris Lu 842dab07b4 weed mount can request to prioritize to write to a data center 2018-07-14 13:36:28 -07:00
Chris Lu a51aa4c586 avoid extra log messages 
fix https://github.com/chrislusf/seaweedfs/issues/689
 2018-07-13 22:07:32 -07:00
Chris Lu 9e974f6a1b
Merge pull request #686 from qszxnp/filer2_dir_cache_bugfix 
bugfix:delete Directory cache when filer2 delete Directory
 2018-07-12 01:17:32 -07:00
聂鹏 1ed41f8b21 bugfix:check Directory attr before create it 2018-07-12 00:53:31 -04:00
聂鹏 c523bc4f07 bugfix:delete Directory cache when filer2 delete Directory 2018-07-11 21:27:25 -04:00
Chris Lu 2fd48df9d9 fix test 2018-07-11 12:52:48 -07:00
Chris Lu f7afa80250 also remove volumeLayout if collection is empty 2018-07-11 02:01:33 -07:00
Chris Lu 1e1c368ae8 adjust filer help message 2018-07-10 00:22:11 -07:00
Chris Lu 007c4a1965 alt request for dc preferred requests 2018-07-10 00:20:50 -07:00
Chris Lu 8a12fd9ec4 add error checking when loading dat file 2018-07-09 23:31:25 -07:00
Chris Lu 6c4fee85e3 add primary request and backup request when requesting volumes for a data center 2018-07-09 23:18:20 -07:00
Chris Lu 50d351944f filer add data center option 2018-07-09 02:22:48 -07:00
Chris Lu d0982cafa7 refactoring 2018-07-09 00:22:50 -07:00
Chris Lu 5b1fd374be fix all tests 2018-07-08 02:39:04 -07:00
Chris Lu d4d7ced922 refactoring: add type for needle id, offset 
later the type size can possibly be adjusted
 2018-07-08 02:28:04 -07:00
Chris Lu 922032b9bb add filer sub dir list size limit to "weed server" 2018-07-08 02:11:36 -07:00
Chris Lu 702ff48cda add dir list limit option 2018-07-07 02:18:47 -07:00