gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3239 commits 28 branches 278 tags 135 MiB
Commit graph

381 commits

Author SHA1 Message Date
j.laycock 6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
Tom Maxwell 4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock 1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
j.laycock 595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
xushuxun 5904d78bd4 weed backup: add ttl and replication parameter 2019-08-16 11:05:22 +08:00
Chris Lu 6f75df8660 server: fix starting default master list 2019-08-10 03:10:03 -07:00
Chris Lu b9f6236f8a filer: fix typo 
fix https://github.com/chrislusf/seaweedfs/issues/1031
 2019-08-06 00:42:19 -07:00
Chris Lu c79274f7eb notes on supported meta data store 2019-08-05 16:45:21 -07:00
Feng Ye fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu 8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu 2c6cf72e73 refactoring 2019-07-28 01:55:05 -07:00
Chris Lu 344ce90315 remove weedfuse 2019-07-24 00:42:12 -07:00
Chris Lu 5956dfd08d mount: add umask option 
related to https://github.com/chrislusf/seaweedfs/issues/978
 2019-07-24 00:03:05 -07:00
Chris Lu 61b7a650f4 filer.copy default to 32MB chunks 2019-07-10 21:48:27 -07:00
Chris Lu 23d36dfbaf 1.40 
enable azure replication
 2019-07-02 21:27:05 -07:00
Chris Lu 88fed85bba temp skipping azure 
can not release due to https://github.com/Azure/azure-pipeline-go/issues/23
 2019-06-30 23:17:45 -07:00
Chris Lu 15ecf4e3bf filer: default to leveldb2 2019-06-30 00:44:57 -07:00
Chris Lu c777102da9 go fmt 2019-06-23 15:30:16 -07:00
Chris Lu d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu 6f8b335007 fix duplicated flags 2019-06-23 03:11:21 -07:00
Chris Lu f16375621f big refactoring 2019-06-23 03:08:27 -07:00
Chris Lu 8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu 8b43679ae3 refactoring 2019-06-15 12:21:44 -07:00
Chris Lu 5f6c9825f8 volume server: adds basic metrics 2019-06-14 00:54:56 -07:00
Chris Lu a11525fe4e filer: adds basic metrics pushing to Prometheus gateway 2019-06-13 02:01:54 -07:00
Chris Lu e66bddd84c customizable sleep intervals 2019-06-06 00:39:08 -07:00
Chris Lu 50aa769554 jwt for read access control 2019-06-06 00:29:02 -07:00
Chris Lu 784141c5e6 adjust script parameter 2019-06-05 01:48:03 -07:00
Chris Lu ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu 3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu 17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu 54b835e1ae filer: redis cluster add option for password 
fix https://github.com/chrislusf/seaweedfs/issues/971
 2019-05-20 09:00:30 -07:00
Chris Lu 0094ca3f8a mark the webdav as unstable 2019-05-19 21:08:26 -07:00
Chris Lu 0fe286a6ca filer: optimization for leveldb, add leveldb2 
large filer db may see high CPU and disk usage due to background compaction
 2019-05-17 17:33:49 -07:00
Chris Lu 7e0c3bb732 filer: adjust recommended mysql meta data type to LONGBLOB 
fix https://github.com/chrislusf/seaweedfs/issues/966
 2019-05-17 09:11:08 -07:00
Chris Lu ab22407763 filer ip only use bind ip 2019-05-14 23:43:40 -07:00
Chris Lu 4dfcd2169e weedfuse: adapt to fstab command line pattern 2019-05-11 01:17:55 -07:00
Chris Lu 9a4fb14ea0 weedfuse: daemonize 2019-05-10 17:08:15 -07:00
Chris Lu 1ca1ec906a add weedfuse 2019-05-10 15:03:31 -07:00
Chris Lu d01807ba05 shell: volume.balance add option for all_collections, each_collection 2019-05-06 14:12:19 -07:00
Chris Lu 4c392ca762 use ip.bind to listen for grpc calls 
help to avoid https://github.com/chrislusf/seaweedfs/issues/952
 2019-05-06 08:10:18 -07:00
Chris Lu 25941e0500 master: add jwt expires_after_seconds 2019-05-04 08:42:25 -07:00
Chris Lu 3fbe22b203 better error handling for nil raftServer 
addressing https://github.com/chrislusf/seaweedfs/issues/717
 2019-05-03 22:33:08 -07:00
Chris Lu b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Chris Lu 9d3c945b00
Merge pull request #950 from chrislusf/webdav 
add WebDav
 2019-05-03 01:25:10 -07:00
Chris Lu b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu 47f14775d7 scaffolding, but seems hard to satisfy mac, windows, and different os versions 
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
 2019-05-02 14:22:10 -07:00
Wine93 32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu 82e06a780b mount: allow nonempty fuse mount to work with Samba 
possibly fix https://github.com/chrislusf/seaweedfs/issues/936
 2019-04-29 10:42:09 -07:00
Chris Lu 9774ede3cf adjust usage description 2019-04-24 00:25:20 -07:00
Chris Lu b04d7e3ac0 weed server: also optionally start S3 gateway 2019-04-24 00:18:01 -07:00
Chris Lu dabc9c9a89 filer: listen on specified ip address 2019-04-24 00:01:39 -07:00
Chris Lu 21fe5bde7c filer: remove customizable grpc port, fixed to filer port + 10000 2019-04-23 23:58:52 -07:00
Chris Lu 3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu 730a032137 filer mysql: adjust scaffolding instructions 2019-04-19 09:55:46 -07:00
Chris Lu ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu a2d34d4802 filer.copy: add adjustable compression level 2019-04-18 20:21:28 -07:00
Chris Lu 13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu 8ea1ee6dfa weed shell: add fs.meta.notify, removing filer.export 2019-04-16 01:58:28 -07:00
Chris Lu 2e03f3adef filer: rename -port.public to -port.readonly 2019-04-11 20:42:55 -07:00
Wine93 361912224d typo: remove blank 2019-04-11 09:18:53 +00:00
Chris Lu c0d6cc0067 weed mount: mount to the same folder name 2019-04-10 23:25:41 -07:00
Chris Lu 3555628ad5 weed volume: fix bug, remove boltdb, btree 2019-04-09 10:08:59 -07:00
Chris Lu 8a5ce16e96 weed volume: remove boltdb, btree options, add options for leveldb medium, large 2019-04-09 09:42:06 -07:00
Chris Lu 88ba08cb9e add large disk support 2019-04-08 22:32:42 -07:00
Chris Lu 000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu a41ba79119
Merge pull request #914 from jba/gocdk-receive 
replication: add GoCDK PubSub support
 2019-04-08 10:31:02 -07:00
Chris Lu d14b614407 weed filer.copy: use existing file owner and gropu id 2019-04-07 11:31:50 -07:00
Chris Lu c731d9be6d weed filer.copy: add option to profile the execution 2019-04-07 09:13:24 -07:00
Chris Lu 24c020104a weed filer.copy: parallelize the copying to increase throughput 
improvement https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 23:35:30 -07:00
Jonathan Amsterdam 72920efc20 added entry to scaffold 2019-04-05 13:43:38 -04:00
Chris Lu 300b39b5af weed filer.copy: use cached filer grpc connection 
another attemp for https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 01:09:06 -07:00
Chris Lu 921231268b weed filer.copy: skip files that can not be opened 
such as socket
 2019-04-05 00:04:00 -07:00
Chris Lu 4f714cef22 default weed filer.copy and weed upload chunk size to 32MB 2019-04-04 23:14:32 -07:00
Chris Lu af37b374cb weed shell: fs.cd change current directory 2019-04-04 19:27:51 -07:00
Chris Lu 715a38da1e weed shell: add fs.cd, fs.pwd to change to a directory and print current directory 2019-04-03 00:20:00 -07:00
Chris Lu 2a52e70a41 weed filer: increase default mysql/postgres dir or name length to 65535 2019-04-01 12:37:54 -07:00
Chris Lu 920b4e56aa refactoring 2019-03-29 22:53:35 -07:00
Chris Lu f60d383947 weed filer.copy: support multiple masters 2019-03-29 22:50:14 -07:00
Chris Lu 487ca51491 weed filer.export: fix pagination error 2019-03-28 00:46:15 -07:00
Chris Lu 7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu 70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu a3490b600c weed filer, weed master: add option to disable http 2019-03-21 16:00:46 -07:00
Jonathan Amsterdam e966033098 Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk 2019-03-20 16:02:38 -04:00
Jonathan Amsterdam 8db82e2b75 notification: add Go CDK pubsub support 
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.

Link in all current providers.

Update the notification scaffold.
 2019-03-20 07:57:58 -04:00
chenwanli 45a52b17fd benchmark: add replication flag 2019-03-20 16:38:11 +08:00
Chris Lu da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu 657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu 55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu d30600e6d6 fix filer starting under "weed server" when ip is not localhost 
fix https://github.com/chrislusf/seaweedfs/issues/886
 2019-03-15 09:57:33 -07:00
Chris Lu a4e8aa98a0 weed server: mkdir first, and set the master ip address 
fix https://github.com/chrislusf/seaweedfs/issues/885
 2019-03-15 00:32:42 -07:00
Chris Lu 6fe071175d mount: add option to disable allow others 
fix https://github.com/chrislusf/seaweedfs/issues/877
 2019-03-10 19:45:40 -07:00
Chris Lu 4773497d2c add back "/cluster/status" 
fix https://github.com/chrislusf/seaweedfs/issues/870
 2019-03-06 12:10:45 -08:00
Chris Lu ab32556a51 weed server: fix error coming from moving raft to grpc 
fix https://github.com/chrislusf/seaweedfs/issues/872
 2019-03-06 10:21:49 -08:00
Chris Lu 2812c14520 master: add option to disable http operations 2019-03-02 05:10:05 -08:00
Chris Lu 7bbe24dd28 volume server directly support https 2019-02-25 00:43:36 -08:00
Chris Lu 0fa1be8c4b adjust formatting and comments 2019-02-19 21:10:10 -08:00
Chris Lu 07af52cb6f raft change from http to grpc 
master grpc port is fixed to http port + 10000
 2019-02-18 22:38:14 -08:00
Chris Lu 77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu 98a03b38e5 avoid util package depends on security package 2019-02-16 12:23:35 -08:00
Chris Lu 74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu 215cd27b37 add authorizing fileId write access 
need to secure upload/update/delete for benchmark/filer/mount
need to add secure grpc
 2019-02-14 00:08:20 -08:00
Chris Lu 4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Chris Lu 501bd72b1c wip: add security.toml file 2019-02-09 21:07:12 -08:00
Sergey aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu a6bffc58f9 go fmt 2019-01-18 14:15:10 -08:00
Chris Lu 67e2ea72be master add separate grpc port 
due to https://github.com/soheilhy/cmux/issues/64

fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 14:14:47 -08:00
bingoohuang ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang 6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
Chris Lu 6740427638 weed mount: reduce default chunk size 2019-01-05 15:17:03 -08:00
Chris Lu 55f7d8f801 remove extra log 2018-12-29 18:00:19 -08:00
Chris Lu 170ed53da5 weed mount: mac add AutoXattr 2018-12-29 13:55:38 -08:00
Chris Lu 9f3836072a weed mount: no mac finder broswe mode 2018-12-29 13:16:23 -08:00
Chris Lu d7bf6c5b91 switching_to_seaweedfs_fuse 2018-12-29 10:31:36 -08:00
Chris Lu 03c7447ad6 set mount point to existing permissions 
fix https://github.com/chrislusf/seaweedfs/issues/806
 2018-12-28 23:36:13 -08:00
Chris Lu 2a19511ba8 avoid gzip benchmark data 
related to https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 13:10:46 -08:00
Chris Lu ba923e2864 error if the last arguement does not end with forward slash 2018-12-23 18:20:11 -08:00
Chris Lu 49f40cd145 adjust help message 2018-12-22 15:05:15 -08:00
Chris Lu 852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu 93d0ad09af error log format 2018-12-16 17:37:10 -08:00
Chris Lu c28e8a2397 refactoring 2018-12-06 00:44:41 -08:00
Chris Lu ffa2827ab1 fail fast if two notification queues or inputs are enabled 2018-12-06 00:37:59 -08:00
Chris Lu 52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu fb43a9d7b9 adjust message 2018-12-05 02:03:03 -08:00
Chris Lu 1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu 86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu 6344469a9b set fuse subtype 2018-11-22 23:04:23 -08:00
Chris Lu 60d3681371 default images.fix.orientation to false 2018-11-21 16:50:42 -08:00
Chris Lu c9b3ef58de options to control filer.export verbosity 2018-11-21 15:15:40 -08:00
Chris Lu 2f2e417b68 detailed export 2018-11-21 15:12:47 -08:00
Chris Lu 2e6a3c7b16 reduce memory usage 2018-11-14 22:48:54 -08:00
Chris Lu a62adf62ca a bit faster 2018-11-12 11:19:13 -08:00
Chris Lu affe708961 a bit faster 2018-11-11 00:43:30 -08:00
Chris Lu a09eaf8756 adjusting, seems there are some perf benefit 2018-11-11 00:08:22 -08:00
Chris Lu cbd94b18a5 improve "ls -al" performance for large directory 2018-11-08 07:37:34 -08:00
Chris Lu 85d1d99459 default "weed export to current enabled store" 2018-11-04 12:59:29 -08:00
Chris Lu 6c45787a37 selectively export files 2018-11-04 12:07:33 -08:00
Chris Lu ff4189ab25 go fmt 2018-11-04 11:59:08 -08:00
Chris Lu 8dfac6a4cf working b2 sink 2018-11-04 11:58:59 -08:00
Chris Lu a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu 3674ad9f8e go fmt 2018-11-01 01:12:21 -07:00
Chris Lu db584ff7f8 separate into notification.toml, add gcp pub/sub message queue 2018-11-01 01:11:09 -07:00
Chris Lu 4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu fe31f389b0 add configuration instructions 2018-10-30 02:29:11 -07:00
Chris Lu c58b73ad9b adding BackBlaze, Azure 2018-10-23 23:59:40 -07:00