gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3257 commits 28 branches 278 tags 135 MiB
Commit graph

574 commits

Author SHA1 Message Date
joeslay d8c34b032f
Merge pull request #6 from chrislusf/master 
merge seaweed master
 2019-09-12 11:44:09 +01:00
Chris Lu ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Tom Maxwell 4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
joeslay 6ee65356e3
Merge pull request #3 from chrislusf/master 
merge seaweed master
 2019-09-04 14:58:04 +01:00
j.laycock cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
Chris Lu ee8bcb0c12 remove println 2019-09-03 09:22:56 -07:00
j.laycock 1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
joeslay e91139348e
Merge pull request #2 from chrislusf/master 
merge seaweed master
 2019-09-02 11:39:40 +01:00
j.laycock 595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
Chris Lu 60c9215a00 HCFS: avoid verbose file not found exception stack trace 2019-09-01 21:40:26 -07:00
Chris Lu 0dc43ecdca skip writing content if not modified 
related to https://github.com/chrislusf/seaweedfs/issues/1036
 2019-08-09 14:38:54 -07:00
Feng Ye fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu ca2b81a944 fix test error 2019-07-31 02:09:13 -07:00
Chris Lu 19360ab10a master: redirect clients to the new leader 2019-07-31 01:54:42 -07:00
Chris Lu 8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu 898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu 0264a7f50a set the max file key when ever possible 2019-07-17 23:23:01 -07:00
Chris Lu aff911c00d skip all azuresb 2019-07-17 01:24:20 -07:00
Chris Lu 0ea98dc6a2 disable azure due to api changes 2019-07-17 00:39:36 -07:00
Chris Lu 9e47c4749a shell: add fs.mv 
fix https://github.com/chrislusf/seaweedfs/issues/954
 2019-07-11 09:29:04 -07:00
Chris Lu 1d1b355f9e filer: set file name when uploading to a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1005
 2019-07-11 04:20:03 -07:00
Chris Lu 476a7c269c detect a filer path is a file or a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1004
 2019-07-10 12:11:18 -07:00
Chris Lu 15ecf4e3bf filer: default to leveldb2 2019-06-30 00:44:57 -07:00
Chris Lu 6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu 359563e95a refactoring 2019-06-25 09:49:27 -07:00
Chris Lu c777102da9 go fmt 2019-06-23 15:30:16 -07:00
Chris Lu d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu f16375621f big refactoring 2019-06-23 03:08:27 -07:00
Chris Lu 6c01fb6d2d go fmt 2019-06-23 01:57:51 -07:00
Chris Lu 8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu d1cd8f8c5b add metrics, refactoring 2019-06-22 22:53:52 -07:00
Chris Lu 3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu 613a2e8060 delete garbage only when successful 2019-06-20 23:46:00 -07:00
Chris Lu c74dc2b306 go fmt 2019-06-20 00:55:52 -07:00
Chris Lu e63317fb08 ec deletion code complete, not tested yet 2019-06-20 00:55:30 -07:00
Chris Lu 4cea8aefd0 add grpc VolumeEcBlobDelete 2019-06-20 00:17:11 -07:00
Chris Lu 856da7aae2 ec volume support deletes 2019-06-19 22:57:14 -07:00
Chris Lu d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu ff6a6dd11e refactoring 2019-06-15 21:46:55 -07:00
Chris Lu 8b43679ae3 refactoring 2019-06-15 12:21:44 -07:00
Chris Lu 5336008dcd refactoring 2019-06-14 13:06:01 -07:00
Chris Lu d67189921f go fmt 2019-06-14 00:55:12 -07:00
Chris Lu 5f6c9825f8 volume server: adds basic metrics 2019-06-14 00:54:56 -07:00
Chris Lu a11525fe4e filer: adds basic metrics pushing to Prometheus gateway 2019-06-13 02:01:54 -07:00
Chris Lu 57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu e66bddd84c customizable sleep intervals 2019-06-06 00:39:08 -07:00
Chris Lu 50aa769554 jwt for read access control 2019-06-06 00:29:02 -07:00
Chris Lu d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu 450f4733ce report empty volume or ec shards 2019-06-05 13:32:33 -07:00
Chris Lu 7f6fb9ffd0 every 17 minutes instead of seconds 
TODO: make this configurable
 2019-06-05 13:32:08 -07:00
Chris Lu ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu 2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu b05456fe07 able to purge extra ec shard copies 2019-06-03 20:25:02 -07:00
Chris Lu 11cffb3168 fix ec.rebuild bugs 2019-06-03 11:50:54 -07:00
Chris Lu d85b41b904 fix ec.encode not finding the local ec shards 2019-06-03 10:38:21 -07:00
Chris Lu 7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu 55be09996d fix volume balance bug 2019-06-03 00:13:31 -07:00
Chris Lu 133b772fb5 destroy ec volume if it is empty 2019-06-01 01:51:28 -07:00
Chris Lu ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu f53024d79d delete ec collection 2019-05-30 10:40:22 -07:00
Chris Lu 1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu e941d0c2f4 list ec collections also 2019-05-30 09:17:58 -07:00
Chris Lu 3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu 2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu 713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu 217cde0a3b refactoring 2019-05-27 21:40:51 -07:00
Chris Lu 03b9291e5d volum server passes grpc option and master location to store 2019-05-27 21:22:23 -07:00
Chris Lu ac1ba3b667 remove comments 2019-05-27 21:21:52 -07:00
Chris Lu b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu a4f3d82c57 convert needle id to ec intervals to read from 2019-05-27 01:29:46 -07:00
Chris Lu 92f9c7b1aa adjust ec shard status on disconnect 2019-05-26 01:05:08 -07:00
Chris Lu b58e25e588 broadcast vid->location map to master clients for ec chards 2019-05-26 00:49:15 -07:00
Chris Lu af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu 41e8ae61f8 generate, balance, delete copied shards, delete old volume 2019-05-25 14:02:06 -07:00
Chris Lu 6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu 24b8ad9a14 push the ec shard info to master 2019-05-23 23:47:49 -07:00
Chris Lu 4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu 17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu 698dea779c refactoring 2019-05-19 21:37:49 -07:00
Chris Lu 017d0957c3 avoid switching master node if same ip and host name 
fix https://github.com/chrislusf/seaweedfs/issues/955
 2019-05-11 18:42:46 -07:00
Chris Lu 8ffdd5dcdb writel throttling moving volumes 2019-05-06 13:58:42 -07:00
Chris Lu 3ff92037cc shell: add volume.balance command 2019-05-05 21:17:23 -07:00
Chris Lu 25941e0500 master: add jwt expires_after_seconds 2019-05-04 08:42:25 -07:00
Chris Lu bd8af92b54 remove unused code 2019-05-04 08:20:22 -07:00
Chris Lu 5681208f2d refactor: remove logger 2019-05-03 22:21:06 -07:00
Chris Lu b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Chris Lu f0f981e7c8 webdav: fix directory renaming 2019-05-03 14:12:51 -07:00
Chris Lu bfbecd7253 webdav: fix directory creation 2019-05-03 13:13:11 -07:00
Chris Lu 9d3c945b00
Merge pull request #950 from chrislusf/webdav 
add WebDav
 2019-05-03 01:25:10 -07:00
Chris Lu 7064b2ea48 webdav: can write now 2019-05-03 00:55:52 -07:00
Chris Lu b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu 47f14775d7 scaffolding, but seems hard to satisfy mac, windows, and different os versions 
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
 2019-05-02 14:22:10 -07:00
Wine93 32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu 108d0fb08d adjust log level 2019-04-26 09:32:37 -07:00
Chris Lu 00b6f653fa shell: add fs.cat 2019-04-21 15:43:43 -07:00
Chris Lu 440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu 0302b9496c volume: also check cookie for possible duplicates 2019-04-21 13:31:45 -07:00
Chris Lu 132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu 2ffe98443b refactoring 2019-04-20 12:05:28 -07:00
Chris Lu 6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
Chris Lu 3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu fa176fe80f volume tailing chunks large files 2019-04-18 19:22:13 -07:00
Chris Lu 3dce1016cb add volume tailer 2019-04-18 11:05:02 -07:00
Chris Lu b142f9f1d5 go fmt 2019-04-18 00:19:18 -07:00
Chris Lu b09e8dbb37 add VolumeStreamFollow, but not used yet 2019-04-18 00:18:29 -07:00
Chris Lu 13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu 3b9446606a adjust log level 2019-04-16 23:31:44 -07:00
Chris Lu 338e6d60a5 refactor: prepare for snapshotting 2019-04-16 09:55:37 -07:00
Chris Lu 79c2cca9c1 better error message 2019-04-16 01:15:30 -07:00
Chris Lu 6e116b3d67 volume: reset leader if error happens 2019-04-15 09:09:46 -07:00
Chris Lu e85048bcdc http exhaust and close response body 2019-04-14 23:28:24 -07:00
Chris Lu 3e8a3a8fec fix race detector found problems 2019-04-14 23:00:37 -07:00
Chris Lu 0e33272f56 small refactoring 2019-04-10 23:39:53 -07:00
Chris Lu cb2061f86a
Merge pull request #925 from stlpmo-jn/checkUnhealthReplication 
the implement of repair the error replications of the volume
 2019-04-10 23:37:12 -07:00
Chris Lu 2f76681d62 refactor: adjust grpc API name 2019-04-10 21:41:17 -07:00
stlpmo-jn f2031884f0 fix bug : CI build failed 2019-04-11 09:53:31 +08:00
stlpmo-jn c1a0403da2 repair the error replications of the volume 2019-04-10 19:41:55 +08:00
Chris Lu 000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu 766396d249 weed master: atomic volume counting 
possible fix for https://github.com/chrislusf/seaweedfs/issues/913
 2019-04-04 19:27:00 -07:00
Chris Lu 189c890715 weed replicate: replicate atomic rename to other systems 2019-03-31 08:10:47 -07:00
Chris Lu 78ac2bef3c go fmt 2019-03-30 23:09:16 -07:00
Chris Lu 97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu c7e7b6229f detect mime type 2019-03-27 14:25:18 -07:00
Chris Lu 65757ae2fd weed filer: set content-type consistent with filer store 2019-03-27 11:41:11 -07:00
Chris Lu 7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu 19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu c913583e2d Merge branch 'master' into add_volume_follow 2019-03-25 21:19:34 -07:00
Chris Lu a32797518b weed master: redirect to leader for default admin UI 
fix https://github.com/chrislusf/seaweedfs/issues/898
 2019-03-25 09:20:05 -07:00
Chris Lu 70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu 95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu a3490b600c weed filer, weed master: add option to disable http 2019-03-21 16:00:46 -07:00
Chris Lu 531add52c2 weed master: skip proxied file read in http read only mode 2019-03-21 09:49:04 -07:00
Jonathan Amsterdam e966033098 Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk 2019-03-20 16:02:38 -04:00
Chris Lu 977b30e992 weed filer: http HEAD response header add ETag 
fix https://github.com/chrislusf/seaweedfs/issues/892
 2019-03-20 10:13:53 -07:00
Jonathan Amsterdam 8db82e2b75 notification: add Go CDK pubsub support 
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.

Link in all current providers.

Update the notification scaffold.
 2019-03-20 07:57:58 -04:00
Chris Lu 88ab932f7d refactoring function parameter 2019-03-19 23:01:23 -07:00
Chris Lu da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu eb4a54d9fe refactoring 2019-03-19 05:19:37 -07:00
Chris Lu 104922a3db text wrapping 2019-03-18 00:35:15 -07:00
Chris Lu aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu 657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu 55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu 55c85f3a66 adding context 2019-03-15 15:26:09 -07:00
Chris Lu ad08a52ab6 synchronously upload files 
fix https://github.com/chrislusf/seaweedfs/issues/807
 2019-03-09 13:24:22 -08:00
Chris Lu 4773497d2c add back "/cluster/status" 
fix https://github.com/chrislusf/seaweedfs/issues/870
 2019-03-06 12:10:45 -08:00
Chris Lu 0898e6c38e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-04 13:00:10 -08:00
Chris Lu d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu f9dcf56465 adjust log level for loading configurations 2019-03-04 09:02:58 -08:00
chenwanli f233bbe9c0 Fix https://github.com/chrislusf/seaweedfs/issues/825#issuecomment-469097375 2019-03-04 12:43:43 +08:00
Chris Lu 219b651bc3 jwt check the base file id 
fix https://github.com/chrislusf/seaweedfs/issues/867
 2019-03-03 10:17:44 -08:00
Chris Lu 2812c14520 master: add option to disable http operations 2019-03-02 05:10:05 -08:00
Chris Lu e108688990 avoid grpc 5 seconds timeout 
some operations may take longer than 5 seconds.

only keep the timeout for raft operations
 2019-02-20 01:01:01 -08:00
Chris Lu 07af52cb6f raft change from http to grpc 
master grpc port is fixed to http port + 10000
 2019-02-18 22:38:14 -08:00
Chris Lu d37c3ab7a5 adjust error message 2019-02-18 18:03:27 -08:00
Chris Lu 77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu 7103c1ab7e go fmt 2019-02-15 00:09:48 -08:00
Chris Lu 74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu 215cd27b37 add authorizing fileId write access 
need to secure upload/update/delete for benchmark/filer/mount
need to add secure grpc
 2019-02-14 00:08:20 -08:00
Chris Lu 4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Chris Lu 8afc632484 raft: use the first master to bootstrap the election 2019-02-09 12:52:09 -08:00
Chris Lu 744abc2690 log raft state changes 2019-02-09 12:51:14 -08:00
Chris Lu 1334507595 Revert "randomize based on self address" 
This reverts commit 6230eb28a6.
 2019-01-28 12:12:51 -08:00
Chris Lu 6230eb28a6 randomize based on self address 
fix #851
 2019-01-28 11:55:33 -08:00
Chris Lu 221105eea3 Revert "use the first entry to bootstrap master cluster" 
This reverts commit 40c8725ffa.
 2019-01-28 11:46:46 -08:00
Chris Lu 40c8725ffa use the first entry to bootstrap master cluster 
fix https://github.com/chrislusf/seaweedfs/issues/851
 2019-01-28 10:35:28 -08:00
Chris Lu 834f414af9 add a timeout 2019-01-26 00:15:42 -06:00
Chris Lu 3f56b12ed4 raft: adding idle connection time out 
another attempt to fix https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-22 09:25:25 -08:00
Chris Lu 67e2ea72be master add separate grpc port 
due to https://github.com/soheilhy/cmux/issues/64

fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 14:14:47 -08:00
Chris Lu 1d103e3ed5 timeout http connections 
possible fix for https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-17 23:38:33 -08:00
Chris Lu 2ff95ead57
Merge pull request #832 from PapaYofen/fix-830 
Fix https://github.com/chrislusf/seaweedfs/issues/830
 2019-01-12 17:26:51 -08:00
chenwanli b71d6f4b88 Fix https://github.com/chrislusf/seaweedfs/issues/830 2019-01-12 10:07:19 +08:00
chenwanli 80cf99db1b Set rpc dial timeout to 5s 2019-01-11 18:23:31 +08:00
chenwanli 0a3e83a36a Set timeout for master and volume non-streaming rpc 2019-01-10 19:41:03 +08:00
chenwanli 41fe31b5d0 Speed up heartbeat to new master leader 2019-01-07 16:09:15 +08:00
Chris Lu 7f74577db1 add Last-Modified to HEAD requests 
to work better with restic
 2019-01-02 14:23:25 -08:00
Chris Lu 28a41fda3e assign new file id even on filer PUT operations 2019-01-02 12:58:26 -08:00
Chris Lu e49a38842a set etag in response 2019-01-02 12:57:54 -08:00
Chris Lu 0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu 260b038441 fix url 2018-12-27 14:43:36 -08:00
Chris Lu be946c9e54 filer store original data size instead of data size after gzip 2018-12-22 13:11:07 -08:00
Chris Lu 5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu 52712782a2 adding link 2018-12-17 00:31:23 -08:00
Chris Lu 2fb740a2b9 fix upload path 2018-12-17 00:21:34 -08:00
Chris Lu 7b7f4215aa filer update existing entry 2018-12-17 00:20:00 -08:00
Chris Lu 6071be104b adjust log format 2018-12-17 00:03:14 -08:00
Chris Lu 3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu 93d0ad09af error log format 2018-12-16 17:37:10 -08:00
Chris Lu 52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu 650d1af978 HCFS support change user name and group names 2018-12-04 00:46:00 -08:00
Chris Lu 7ace0efd65 revert prev 2018-12-02 23:25:37 -08:00
Chris Lu 738247d47e do not throw error if not found 2018-12-02 23:20:29 -08:00