gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9910 commits 28 branches 278 tags 135 MiB
Commit graph

1125 commits

Author SHA1 Message Date
Chris Lu 215cd27b37 add authorizing fileId write access 
need to secure upload/update/delete for benchmark/filer/mount
need to add secure grpc
 2019-02-14 00:08:20 -08:00
Chris Lu 4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Chris Lu 501bd72b1c wip: add security.toml file 2019-02-09 21:07:12 -08:00
Sergey aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu a6bffc58f9 go fmt 2019-01-18 14:15:10 -08:00
Chris Lu 67e2ea72be master add separate grpc port 
due to https://github.com/soheilhy/cmux/issues/64

fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 14:14:47 -08:00
bingoohuang ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang 6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
Chris Lu 6740427638 weed mount: reduce default chunk size 2019-01-05 15:17:03 -08:00
Chris Lu 55f7d8f801 remove extra log 2018-12-29 18:00:19 -08:00
Chris Lu 170ed53da5 weed mount: mac add AutoXattr 2018-12-29 13:55:38 -08:00
Chris Lu 9f3836072a weed mount: no mac finder broswe mode 2018-12-29 13:16:23 -08:00
Chris Lu d7bf6c5b91 switching_to_seaweedfs_fuse 2018-12-29 10:31:36 -08:00
Chris Lu 03c7447ad6 set mount point to existing permissions 
fix https://github.com/chrislusf/seaweedfs/issues/806
 2018-12-28 23:36:13 -08:00
Chris Lu 2a19511ba8 avoid gzip benchmark data 
related to https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 13:10:46 -08:00
Chris Lu ba923e2864 error if the last arguement does not end with forward slash 2018-12-23 18:20:11 -08:00
Chris Lu 49f40cd145 adjust help message 2018-12-22 15:05:15 -08:00
Chris Lu 852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu 93d0ad09af error log format 2018-12-16 17:37:10 -08:00
Chris Lu c28e8a2397 refactoring 2018-12-06 00:44:41 -08:00
Chris Lu ffa2827ab1 fail fast if two notification queues or inputs are enabled 2018-12-06 00:37:59 -08:00
Chris Lu 52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu fb43a9d7b9 adjust message 2018-12-05 02:03:03 -08:00
Chris Lu 1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu 86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu 6344469a9b set fuse subtype 2018-11-22 23:04:23 -08:00
Chris Lu 60d3681371 default images.fix.orientation to false 2018-11-21 16:50:42 -08:00
Chris Lu c9b3ef58de options to control filer.export verbosity 2018-11-21 15:15:40 -08:00
Chris Lu 2f2e417b68 detailed export 2018-11-21 15:12:47 -08:00
Chris Lu 2e6a3c7b16 reduce memory usage 2018-11-14 22:48:54 -08:00
Chris Lu a62adf62ca a bit faster 2018-11-12 11:19:13 -08:00
Chris Lu affe708961 a bit faster 2018-11-11 00:43:30 -08:00
Chris Lu a09eaf8756 adjusting, seems there are some perf benefit 2018-11-11 00:08:22 -08:00
Chris Lu cbd94b18a5 improve "ls -al" performance for large directory 2018-11-08 07:37:34 -08:00
Chris Lu 85d1d99459 default "weed export to current enabled store" 2018-11-04 12:59:29 -08:00
Chris Lu 6c45787a37 selectively export files 2018-11-04 12:07:33 -08:00
Chris Lu ff4189ab25 go fmt 2018-11-04 11:59:08 -08:00
Chris Lu 8dfac6a4cf working b2 sink 2018-11-04 11:58:59 -08:00
Chris Lu a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu 3674ad9f8e go fmt 2018-11-01 01:12:21 -07:00
Chris Lu db584ff7f8 separate into notification.toml, add gcp pub/sub message queue 2018-11-01 01:11:09 -07:00
Chris Lu 4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu fe31f389b0 add configuration instructions 2018-10-30 02:29:11 -07:00
Chris Lu c58b73ad9b adding BackBlaze, Azure 2018-10-23 23:59:40 -07:00
Chris Lu fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu 556382ff5f adding basic grpc to volume server 2018-10-11 01:16:33 -07:00
Chris Lu 08266b7256 go fmt 2018-10-11 00:08:13 -07:00
Chris Lu 1d7e01b754 reformat 2018-10-11 00:05:54 -07:00
Chris Lu da6154b29c refactor volume server to startVolumeServer() 2018-10-11 00:04:31 -07:00
Chris Lu 60d2f1557d refactor func name 2018-10-10 23:19:54 -07:00
Chris Lu 9f4c2f87d3 adding filer replication sink to Azure 2018-10-09 01:35:48 -07:00
Chris Lu 96a23591e7 remove comments 2018-10-07 22:21:00 -07:00
Chris Lu a5b5887efc add filer replication to aws s3 2018-10-06 17:10:15 -07:00
Chris Lu 84d4a2c8a0 add filer replication to google cloud storage 2018-10-06 13:08:38 -07:00
Chris Lu 14858f6224 add logging 2018-10-06 13:01:38 -07:00
Chris Lu 56c5c7b1b6 add google cloud storage 2018-10-04 01:14:44 -07:00
Chris Lu e8ef501f02 add s3 replication sink 2018-10-03 23:36:52 -07:00
Chris Lu 98384b62fd adjust description 2018-09-28 01:58:34 -07:00
Chris Lu 12ba8b9799 generate useable filer.toml 2018-09-23 09:26:25 -07:00
Chris Lu b16d47030e save and use last processed kafka offsets 2018-09-23 01:34:40 -07:00
Chris Lu 9fe24991d5 refactoring 2018-09-23 00:40:36 -07:00
Chris Lu 7d6b2a4740 add ETag to upload results and chunks 2018-09-22 22:12:21 -07:00
Chris Lu 420f068376 add more instructions 2018-09-22 22:12:06 -07:00
Chris Lu 01ceace18e adjust sink options 2018-09-22 00:53:52 -07:00
Chris Lu 402ab598b6 go fmt 2018-09-22 00:12:10 -07:00
Chris Lu 7bb62b9f21 delay file creation without chunks from dir.Create to file.Flush 2018-09-22 00:11:46 -07:00
Chris Lu db69ce89f0 go fmt 2018-09-21 01:56:43 -07:00
Chris Lu a6cfaba018 able to sync the changes 2018-09-21 01:54:29 -07:00
Chris Lu a8cd7df44a let filer choose default replication 2018-09-20 09:00:15 -07:00
Chris Lu d5149e592e add logging on kafka reading 2018-09-19 22:31:45 -07:00
Chris Lu 25fb6f9a46 fix compilation 2018-09-17 02:23:21 -07:00
Chris Lu 779641e9d4 adjust replicated entry name 2018-09-17 01:37:24 -07:00
Chris Lu b7a375f5aa set large limit for "weed filer.export" directory listing 2018-09-17 00:42:36 -07:00
Chris Lu 4e1f1a0b92 set large limit for "weed mount" directory listing 2018-09-17 00:41:33 -07:00
Chris Lu 788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu 7ad5a90127 go fmt 2018-09-03 16:47:00 -07:00
Chris Lu 6e3f4d1079 add working filer.export command 2018-08-23 00:02:04 -07:00
Chris Lu 98b8f8649d adding filer.export prototype 2018-08-22 23:54:22 -07:00
Chris Lu ae4e654d2a add example configuration for default redis create-cluster setup 2018-08-22 18:23:19 -07:00
Chris Lu 748870b03f add "weed scaffold" command 2018-08-19 15:36:30 -07:00
Chris Lu f827ada811 merge notification config with filer.toml 2018-08-19 15:17:55 -07:00
Chris Lu f036ef8a3c add filer notification 2018-08-13 01:20:49 -07:00
Chris Lu 75d63db60d randomize raft server startup 
also some go fmt
 2018-08-12 14:27:14 -07:00
Chris Lu eca4b928d2 ensure master server count is odd 2018-08-12 14:25:31 -07:00
Chris Lu 2e427c66ac small refactoring 2018-08-10 23:47:17 -07:00
Chris Lu 44d8e35988 add listener for volumd id location for benchmark tool 2018-07-31 19:12:36 -07:00
Chris Lu d09606c65b refactoring 2018-07-29 02:25:24 -07:00
Chris Lu 4f317c7e3d working version3 2018-07-24 01:36:04 -07:00
Chris Lu 7f32eb1e25 s3 add https support 2018-07-22 21:28:54 -07:00
Chris Lu 056871242b remove println 2018-07-22 01:17:49 -07:00
Chris Lu 0655151b8c go fmt 2018-07-22 01:15:11 -07:00
Chris Lu 6319d84f42 s3 API add ListObjectsV1 2018-07-22 01:14:36 -07:00
Chris Lu 922c614bde fix compilation problem 2018-07-21 20:42:14 -07:00
Chris Lu 16c66e097b adjust default buckets folder 2018-07-21 20:29:33 -07:00
Chris Lu 7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu c7228fefa7 add bucket creation and deletion 
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
 2018-07-19 01:21:44 -07:00
Chris Lu 834a25f084 add list all my buckets 2018-07-18 02:37:09 -07:00
Chris Lu 7abfab8e77 add feature to mount a specific filer path to local directory 2018-07-14 21:09:21 -07:00
Chris Lu 3edfe1d28f extend export command to show tombstone + change output format to CSV 
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
 2018-07-14 20:51:17 -07:00
Chris Lu 6d58916cfd fix compilation error 2018-07-14 13:39:57 -07:00
Chris Lu 842dab07b4 weed mount can request to prioritize to write to a data center 2018-07-14 13:36:28 -07:00
Chris Lu 1e1c368ae8 adjust filer help message 2018-07-10 00:22:11 -07:00
Chris Lu 50d351944f filer add data center option 2018-07-09 02:22:48 -07:00
Chris Lu d4d7ced922 refactoring: add type for needle id, offset 
later the type size can possibly be adjusted
 2018-07-08 02:28:04 -07:00
Chris Lu 922032b9bb add filer sub dir list size limit to "weed server" 2018-07-08 02:11:36 -07:00
Chris Lu 702ff48cda add dir list limit option 2018-07-07 02:18:47 -07:00
Chris Lu 2ad45ca04f refactor 2018-07-07 02:03:25 -07:00
Chris Lu 77fc8c5914 keep alive for gRpc calls 2018-07-03 19:07:55 -07:00
Chris Lu 5bd72696ac weed mount add ttl option 2018-06-11 23:13:33 -07:00
Chris Lu 98110c1697 add collection, replication, ttl to each file 2018-06-10 16:57:32 -07:00
Chris Lu 299312c805 use separate filer grpc port 2018-06-05 23:37:41 -07:00
Chris Lu 43e3f5724c use fixed list of masters in both filer and volume servers 2018-06-01 00:39:39 -07:00
Chris Lu dc13e10637 fix copy error on 0 size files 2018-05-30 22:28:14 -07:00
Chris Lu 44acf4b756 fix file name 
f.Name() is not the base file name
 2018-05-30 20:48:13 -07:00
Chris Lu 0301504184 add mime, use simple insert and update filer store API 
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
 2018-05-30 20:24:57 -07:00
Chris Lu 26e7cd8c75 assign a different volume on large file copying 2018-05-30 01:05:26 -07:00
Chris Lu 2fe0d479f1 filer.copy supports chunking 2018-05-30 00:54:56 -07:00
Chris Lu 2d7dea1a09 a little refactoring 2018-05-29 23:52:27 -07:00
Chris Lu 8a48baa056 add single chunk file copying to new filer 2018-05-29 23:46:45 -07:00
Chris Lu 4e3ea49cff properly working local write buffer 2018-05-29 01:21:21 -07:00
Chris Lu 5c4480ec6c add mountOptions.chunkSizeLimitMB, remove cmdMount.IsDebug 2018-05-28 13:42:25 -07:00
Chris Lu 8ab7dd9d08 weed mount add options for collection and replication 2018-05-28 13:24:48 -07:00
Chris Lu 458ada173e go fmt 2018-05-27 11:52:26 -07:00
Chris Lu 9e77563c99 add leveldb store 
1. switch to viper for filer store configuration
2. simplify FindEntry() return values, removing “found”
3. add leveldb store
 2018-05-26 03:49:46 -07:00
Chris Lu 9dd228747c filer copy added uid/gid 2018-05-21 01:25:30 -07:00
Chris Lu c5cf9bd290 properly working filer 2018-05-13 23:56:16 -07:00
Chris Lu 9f345da20f mv filer proto to filer_pb 2018-05-09 23:18:02 -07:00
Chris Lu 942c2cbd7b mv pb to master_pb 2018-05-09 23:11:54 -07:00
Chris Lu 43a69d20bf change filer API to gRPC 2018-05-08 01:59:43 -07:00
Chris Lu 4936d6c342 adjusting for FUSE 2018-05-07 00:37:47 -07:00
Chris Lu 456738ba64 refactoring fuse 2018-05-05 22:47:16 -07:00
Chris Lu fffb14bc87 better support FUSE Lookup() 2018-05-05 02:01:50 -07:00
Chris Lu 4b9b9547d4 unused option 2018-02-27 13:57:15 -08:00
Chris Lu 61ff330e99 adjust instructions 2018-01-15 20:33:12 -08:00
Chris Lu ef6224e45d filer default to auto chunking file size to 32MB 2017-09-15 08:24:30 -07:00
Chris Lu 58344980e4 preallocate disk space during compaction also, add cleanup for failed compaction 2017-08-29 23:59:53 -07:00
Chris Lu dbee816205 fix typo 
fix https://github.com/chrislusf/seaweedfs/issues/549
 2017-08-10 10:26:39 -07:00
Chris Lu e0dfa7a069 ensure 30GB or less 
avoid https://github.com/chrislusf/seaweedfs/issues/535
 2017-07-16 21:40:55 -07:00
Chris Lu 1fbb8723dc add cpu and mem pprof 2017-06-22 01:33:58 -07:00
Chris Lu 5047bdb4a2 skip bytes cache 2017-06-03 11:44:24 -07:00
Chris Lu 478fe0ecf2 filer add readonly public port 2017-05-27 20:14:22 -07:00
Chris Lu 7ecc0f4b11 rename variables 2017-05-27 18:11:18 -07:00
Chris Lu 82c3ccc8dd add btree for volume index 2017-05-26 22:51:25 -07:00
Chris Lu 4ce6586710 use local random generator to avoid global lock 2017-05-19 23:33:36 -07:00
sparklxb c46e91d229 complement weed mount: add read and delete 2017-03-17 15:02:27 +08:00
sparklxb 3a67b6ddc1 complement weed mount 2017-03-10 14:46:00 +08:00
sparklxb 7f85e15990 complement weed mount 2017-03-10 10:28:19 +08:00
sparklxb 813a4f7456 Fixes #266 2017-03-06 13:04:52 +08:00
Chris Lu 41f686ab9d remove weeds.conf 2017-02-12 21:58:44 -08:00
Chris Lu d953ed442c auto detect volume server ip address 2017-01-12 13:42:53 -08:00
Chris Lu 1be354bdce default IP to "localhost" 
so that the master can know the ip address.
 2017-01-10 23:57:59 -08:00
Chris Lu e767c3ea4f disable master connection timeout 
temporarily disable master connection timeout due to heartbeat
connection timeout
 2017-01-10 01:30:00 -08:00
Chris Lu e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00
sparklxb 66e7013dfe suport uploading files to specific dataCenter 2017-01-09 23:34:47 +08:00
Chris Lu 53cf1b4900 change the basic unit to MB 2017-01-08 14:57:32 -08:00
Chris Lu ed44f12f6d support Fallocate on linux 2017-01-08 11:01:46 -08:00
Chris Lu 13e7069eb9 keep track of total data file size 2017-01-06 10:22:20 -08:00
lixianbin d96d0a87cf fix bug: upload big .gz file more than maxMB 2017-01-04 11:23:40 +08:00
Chris Lu dcaf1796fe add option to enable caching 2016-11-16 07:09:57 -08:00
Chris Lu 1bc041b46d add a new way to manually compact corrupted volume 
fix https://github.com/chrislusf/seaweedfs/issues/371
 2016-09-22 20:31:20 -07:00
霍晓栋 1c2affea8e Merge remote-tracking branch 'upstream/master' 2016-09-08 10:34:24 +08:00
Chris Lu c4b7966dbe minor help message change 2016-09-07 18:21:05 -07:00
霍晓栋 b0035747e3 add filer support 2016-08-31 11:32:30 +08:00
Mike Tolman 01d3f69c52 Adding AutoChunk/MaxMB Support to Filer API 
This is related to the following issue I added to chrislusf/seaweedfs:
https://github.com/chrislusf/seaweedfs/issues/342
 2016-08-05 16:01:30 -06:00
Mike Tolman a89a3c86d0 Revert "Add AutoChunking to the Filer API, so that you can upload really large files through the filer API." 
This reverts commit 09059bfdcc.
 2016-08-05 15:47:46 -06:00
Mike Tolman ce99bb927d Revert "Adding HTTP verb whitelisting options." 
This reverts commit 34837afc7a.
 2016-08-05 15:45:48 -06:00
Mike Tolman 34837afc7a Adding HTTP verb whitelisting options. 2016-08-05 15:23:43 -06:00
Mike Tolman 09059bfdcc Add AutoChunking to the Filer API, so that you can upload really large files through the filer API. 2016-08-05 15:19:10 -06:00
Chris Lu 185a916f5e adjusting command options 2016-07-21 15:00:07 -07:00
Chris Lu cdae9fc680 add "weed copy" command to copy files to filer 2016-07-20 23:45:55 -07:00
Chris Lu d3ae3a8e14 collect proof when stopping 2016-07-05 10:23:14 -07:00
Chris Lu b38ff39e8b add master profiling option 2016-07-03 15:39:57 -07:00
霍晓栋 1f63094542 refactor Volume Assign function 2016-06-26 10:50:18 +08:00
Chris Lu 5ce6bbf076 directory structure change to work with glide 
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
 2016-06-02 18:09:14 -07:00