gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9946 commits 28 branches 278 tags 135 MiB
Commit graph

226 commits

Author SHA1 Message Date
Chris Lu 2ff37ccdbd testing parsing the upload 
The mime type is always the value passed in.

Compress or not depends on the content detection, file name extension, and compression ratio.

If the content is already compressed, need to know the content size.
 2020-06-24 11:39:12 -07:00
Chris Lu fe60db404a prepare for zstd 2020-06-23 09:12:02 -07:00
Chris Lu 3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu 97239ce6f1 rename filechunk is_gzipped to is_compressed 2020-06-20 08:15:49 -07:00
Chris Lu e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu 16fe132a20 detect mime type before replicating to other volume servers 2020-06-19 22:11:36 -07:00
Chris Lu 39e72fb23c go fmt 2020-05-10 03:50:30 -07:00
Chris Lu 78422c2f69
Merge pull request #1298 from song-zhang/master 
add fsync option for benchmark
 2020-04-30 20:16:42 -07:00
Chris Lu 871efa4fc1 refactoring 
some previous chunk etag was using md5, which should be wrong.
 2020-04-30 17:20:47 -07:00
zhangsong d819865fe7 add fsync option for benchmark 2020-04-30 23:31:08 +08:00
Chris Lu 9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu 3f1d79512f only detect when not sure about the types 2020-04-14 11:02:05 -07:00
Chris Lu d2d1597850 volume: detect gzip effectiveness only when content type is empty 
revert part of 2286eda575
 2020-04-14 10:52:53 -07:00
Chris Lu 8a73276602 refactor a bit 2020-04-08 22:33:36 -07:00
Chris Lu bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu e1911760a7 refactoring 2020-03-27 00:30:55 -07:00
Chris Lu 2286eda575 sampling whether the data can be gzipped 2020-03-26 23:50:48 -07:00
Chris Lu a5ce359999 refactoring 2020-03-26 22:46:49 -07:00
Chris Lu 6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu 6ff9e2835e filer: skip resizing image if width or height larger than original image 
fix https://github.com/chrislusf/seaweedfs/issues/1239#issuecomment-602140779
 2020-03-21 22:16:00 -07:00
Chris Lu 308688c8d0 filer: add back image resizing capability 2020-03-20 20:31:11 -07:00
Chris Lu f251d03673 refactoring 2020-03-20 15:46:16 -07:00
Chris Lu 11372dd548 refactoring 2020-03-20 15:38:04 -07:00
Chris Lu d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu 6566c8e114 weed upload: add usePublicUrl option 
fix https://github.com/chrislusf/seaweedfs/issues/1236
 2020-03-18 10:50:53 -07:00
Chris Lu f9b3750ad1 volume: handle repeated writes response 2020-03-15 04:16:00 -07:00
Chris Lu ee635dcc36 refactoring 2020-03-14 14:08:00 -07:00
Chris Lu 18cda6adbb cleaner fix 2020-03-14 13:55:32 -07:00
Chris Lu 17efa361f9 fix possible nil 2020-03-14 13:53:03 -07:00
Chris Lu 3934c69757 clean up 2020-03-08 21:54:47 -07:00
Chris Lu 2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu 5ac6297c68 adjust parameter names 2020-03-08 17:03:37 -07:00
Chris Lu 9228ff192c encryption works 2020-03-07 11:08:57 -08:00
Chris Lu 0375ce2c2e filer: set mime type from volume server 2020-03-07 07:25:15 -08:00
Chris Lu ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu 13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu 892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu 6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu 33b3bd467c Revert "HEAD operation changes to fasthttp" 
This reverts commit 58f126fd27.
 2020-02-10 13:43:53 -08:00
Chris Lu 58f126fd27 HEAD operation changes to fasthttp 2020-02-10 09:13:29 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu 8dd260623e protect against nil deletion results 
fix https://github.com/chrislusf/seaweedfs/issues/1180
 2020-01-08 21:50:59 -08:00
Chris Lu 392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu d0b423bbc0 filer: increase directory listing pagination size 2019-12-12 09:11:10 -08:00
Lei Liu 46755ea1e1 fix master maintenance logic 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-12 14:46:10 +08:00
zhangsong 61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00
Chris Lu 327336ecf3 filer: avoid concurrent modification to result slice 
fix https://github.com/chrislusf/seaweedfs/issues/972
 2019-06-28 01:12:41 -07:00
Chris Lu 713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu 2ffe98443b refactoring 2019-04-20 12:05:28 -07:00
Chris Lu 6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
Chris Lu e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu a2d34d4802 filer.copy: add adjustable compression level 2019-04-18 20:21:28 -07:00
Chris Lu 79c2cca9c1 better error message 2019-04-16 01:15:30 -07:00
Chris Lu e85048bcdc http exhaust and close response body 2019-04-14 23:28:24 -07:00
Chris Lu a32abda1a3 adjust compression to optimize for speed 2019-04-07 11:01:28 -07:00
Chris Lu 174bf1e8b2 more efficient client side gzip compression 2019-04-06 14:14:28 -07:00
Chris Lu 19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu 95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu 55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
chenwanli c668e55e07 Revert "fix https://github.com/chrislusf/seaweedfs/issues/780" 
This reverts commit 31038acccf.
 2019-03-11 11:48:11 +08:00
Chris Lu e108688990 avoid grpc 5 seconds timeout 
some operations may take longer than 5 seconds.

only keep the timeout for raft operations
 2019-02-20 01:01:01 -08:00
Chris Lu 72c2e6bb8f remove unused file 2019-02-19 14:16:19 -08:00
Chris Lu 448645203a remove unused variables 2019-02-18 20:05:55 -08:00
Chris Lu 77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu 7103c1ab7e go fmt 2019-02-15 00:09:48 -08:00
Chris Lu 74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu 4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Sergey aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu 67e2ea72be master add separate grpc port 
due to https://github.com/soheilhy/cmux/issues/64

fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 14:14:47 -08:00
bingoohuang ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
chenwanli 39c7455881 Set http timeout to 5s 2019-01-10 20:21:08 +08:00
chenwanli 0a3e83a36a Set timeout for master and volume non-streaming rpc 2019-01-10 19:41:03 +08:00
Chris Lu 6b5d6bb5a6 adjust error names 2019-01-05 19:52:38 -08:00
Chris Lu e49a38842a set etag in response 2019-01-02 12:57:54 -08:00
Chris Lu d3660ffce8 avoid gzipping files without file extensions 
fix https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 12:17:05 -08:00
Chris Lu c043fd17cb detect more gzippable content 2018-12-22 15:05:31 -08:00
Chris Lu 852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu 5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu 29f1673d97 refactoring 2018-12-07 01:25:01 -08:00
chenwanli 31038acccf fix https://github.com/chrislusf/seaweedfs/issues/780 2018-12-05 14:06:16 +08:00
Chris Lu 86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu 444dfded84 add fs.FSStatfser for SeaweedFS weed mount 2018-11-23 00:24:51 -08:00
Chris Lu b282e34dc2 async file chunk deletion 2018-11-20 20:56:28 -08:00
Chris Lu 5065d4ab2d master add grpc API for fileid assigning 2018-11-20 11:35:45 -08:00
Chris Lu 4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu d3b7965c76 caching grpc clients 2018-11-14 23:50:46 -08:00
Chris Lu 93e0dddbf3 add function description 2018-11-03 22:32:21 -07:00
Chris Lu eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu 91ac2e0dd9 go fmt 2018-10-14 00:30:20 -07:00
Chris Lu ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu 08266b7256 go fmt 2018-10-11 00:08:13 -07:00
Chris Lu 06c43286cd skip error when deletion failed 2018-09-28 00:10:49 -07:00
Chris Lu 7d6b2a4740 add ETag to upload results and chunks 2018-09-22 22:12:21 -07:00
Chris Lu cfbfc7cb67 fix compilation error 2018-07-28 18:34:15 -07:00
Chris Lu 888eb2abb5 filer read write all via locations from MasterClient 2018-07-28 14:51:36 -07:00
Chris Lu 7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu 8480008a9a add s3 upload, and removing mono and multi part upload analyzer 
removing mono and multi part upload analyzer, which were used just to determine the file name
 2018-07-21 10:39:02 -07:00
Chris Lu 6c4fee85e3 add primary request and backup request when requesting volumes for a data center 2018-07-09 23:18:20 -07:00
Chris Lu d4d7ced922 refactoring: add type for needle id, offset 
later the type size can possibly be adjusted
 2018-07-08 02:28:04 -07:00
Chris Lu ebf58709d3 remove unused register file function 2018-06-01 23:24:34 -07:00
Chris Lu 0301504184 add mime, use simple insert and update filer store API 
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
 2018-05-30 20:24:57 -07:00
Chris Lu 9dd228747c filer copy added uid/gid 2018-05-21 01:25:30 -07:00
Chris Lu b303a02461 cp file can work 
1. consolidate to filer_pb.FileChunk
2. dir add file, mkdir
3. file flush, write

updates having issue
 2018-05-16 00:08:44 -07:00
Chris Lu c5cf9bd290 properly working filer 2018-05-13 23:56:16 -07:00
Chris Lu fc3924ac70 set file name to base file name 
fixing https://github.com/chrislusf/seaweedfs/issues/632
 2018-02-26 08:01:23 -08:00
Chris Lu 97f757c669 add debug message 
display the error message for debugging
https://github.com/chrislusf/seaweedfs/issues/603
 2017-11-25 01:31:13 -08:00
Chris Lu 3065506b38 volume servers always connect to the master leader 2017-01-13 10:20:40 -08:00
Chris Lu e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00
sparklxb 66e7013dfe suport uploading files to specific dataCenter 2017-01-09 23:34:47 +08:00
sparklxb da9b672d1b support additional header name-value pairs 2017-01-08 22:34:42 +08:00
sparklxb 86a7c56275 support additional header name-value pairs 2017-01-08 09:16:40 +08:00
lixianbin d96d0a87cf fix bug: upload big .gz file more than maxMB 2017-01-04 11:23:40 +08:00
Chris Lu a5be4a6d40 fix package name 2016-07-21 01:23:56 -07:00
Chris Lu cdae9fc680 add "weed copy" command to copy files to filer 2016-07-20 23:45:55 -07:00
霍晓栋 1f63094542 refactor Volume Assign function 2016-06-26 10:50:18 +08:00
霍晓栋 79b0a58132 add options params for the Assign func 2016-06-23 11:19:09 +08:00
Chris Lu a61453d3ec fix nil crash 
fix https://github.com/chrislusf/seaweedfs/issues/320
 2016-06-13 14:38:35 -07:00
Chris Lu 5ce6bbf076 directory structure change to work with glide 
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
 2016-06-02 18:09:14 -07:00
Chris Lu 5071f528f6 testing compilation with remove package 2013-02-10 03:49:51 -08:00
Chris Lu 55f2627fcf testing new directory structure 2013-02-10 03:31:40 -08:00
Chris Lu ab6fb13ad7 avoid the "src" folder 2013-02-10 03:25:35 -08:00