gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
10029 commits 28 branches 278 tags 135 MiB
Commit graph

126 commits

Author SHA1 Message Date
Chris Lu cbd80253e3 better needle id format 2020-08-04 22:09:07 -07:00
Chris Lu f89186a3cf better logging 2020-08-01 11:19:57 -07:00
Chris Lu 97d97f3528 go code can read and write chunk manifest 2020-07-19 17:59:43 -07:00
李远军 6608cb5f43 Revert "remove fixJpgOrientation" 
This reverts commit de5ca9b2
 2020-07-10 10:08:36 +08:00
Chris Lu 20d9cfae86 volume: following md5 base64 encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1374
 2020-06-28 07:25:50 -07:00
Chris Lu 2ff37ccdbd testing parsing the upload 
The mime type is always the value passed in.

Compress or not depends on the content detection, file name extension, and compression ratio.

If the content is already compressed, need to know the content size.
 2020-06-24 11:39:12 -07:00
Chris Lu c21f4ebfee
Merge pull request #1371 from Kimbsen/content_md5_validation 
Optional md5 validation of uploads
 2020-06-24 11:33:44 -07:00
Kimbsen 4eeab2a379 remove unused imports 2020-06-24 13:37:00 +02:00
Kimbsen ffddecebef do md5 validation AFTER decompression 2020-06-24 13:35:13 +02:00
Chris Lu fe60db404a prepare for zstd 2020-06-23 09:12:02 -07:00
Kimbsen 6b1e93ba0b check request header not part header for content-encoding 2020-06-23 17:04:24 +02:00
Kimbsen 38626cb584 don't forget the last few bytes 2020-06-23 16:53:49 +02:00
Kimbsen 8d1f6cd567 added optional md5 verification of uploaded data 2020-06-23 16:24:11 +02:00
Chris Lu de5ca9b258 remove fixJpgOrientation 2020-06-20 12:50:40 -07:00
Chris Lu 3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu 16fe132a20 detect mime type before replicating to other volume servers 2020-06-19 22:11:36 -07:00
Chris Lu 7ccc31853c hard limit volume file size 2020-06-16 00:27:48 -07:00
zhangsong 3703c7e176 write request keep compatible with before 2020-05-06 21:35:32 +08:00
zhangsong f9e8702bb4 use async write to persistent file to disk - part1 2020-05-04 17:39:44 +08:00
Chris Lu 22400c6633 consistent gzip logic 
local store gzip same as replicated writes
 2020-03-15 04:15:40 -07:00
Chris Lu 9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu 1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu 7335e62199 volume: PUT also conditionally gzip compress 2020-03-01 23:37:02 -08:00
Chris Lu 943f4986ef fix possible nil 2020-01-08 08:49:18 -08:00
Chris Lu 9995d3bcb5 remove println 2020-01-07 21:49:28 -08:00
Chris Lu 3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu 10bd3c6b4b refactoring 2019-12-08 19:44:16 -08:00
Chris Lu f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu 85f8649320 refactor memory mapped file into backend storage 2019-11-09 00:10:59 -08:00
Chris Lu 4b5ba4927f adjust NewDiskFile() paramaters 2019-10-29 22:37:36 -07:00
Chris Lu cd7ce720c7 fix test 2019-10-29 00:37:15 -07:00
Chris Lu 19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu 7a51a9a582 refactor: simplifying to ReadAt() and WriteAt() 2019-10-25 00:11:25 -07:00
Chris Lu 46ed2ca902 refactoring 2019-10-24 23:41:32 -07:00
Chris Lu c9a183eb69 refactor memory map related code 2019-10-22 00:49:42 -07:00
Chris Lu faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
j.laycock eb27c2b037 Make releaseMemory private and return byte array instead, fix other platform compilation issues, reduce in-memory chunk size. 2019-10-01 12:21:44 +01:00
j.laycock 6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
j.laycock 63acc8c972 rename mem_buffer to mBuffer 2019-09-05 13:46:29 +01:00
Tom Maxwell 4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock cc756ddd79 Rename End_of_file variable, implement platform indepenent memory_map.go 2019-09-04 14:24:17 +01:00
j.laycock 1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
j.laycock 595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
j.laycock 840ccdc35d Refactor to pass memory maps by reference instead of value, fix memory maps not being created properly or written to properly 2019-08-30 12:30:00 +01:00
j.laycock 0e5d3b1a70 Move memory_map_windows to new folder, intercept most of the read and write calls to volume dat files 2019-08-30 12:30:00 +01:00
Chris Lu c2960c7a86 defaulting file ttl to volume ttl 
fix https://github.com/chrislusf/seaweedfs/issues/992
 2019-06-26 11:18:23 -07:00
Chris Lu f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu 302d9fbc6d adjust error message for file not found 2019-05-28 14:27:21 -07:00
Chris Lu 4f76342cbc WIP 
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
 2019-05-28 00:51:01 -07:00
Chris Lu b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Wine93 32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
stlpmo-jn 2200ea9cb9 add function ParseFileIdFromString 2019-04-20 18:39:06 +08:00
Chris Lu 0be2d51c96 read volume lastAppendAtNs when loading a volume 2019-04-19 01:56:38 -07:00
Chris Lu ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu b142f9f1d5 go fmt 2019-04-18 00:19:18 -07:00
Chris Lu 7a4b234ea1 divide offset into higher and lower sections 2019-04-08 21:44:06 -07:00
Chris Lu 000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu 95ef4513c8 print out memory size 2019-03-13 23:14:40 -07:00
Chris Lu 346541a101 print out per entry memory usage 2019-03-13 23:07:24 -07:00
Chris Lu 09471b46f9 fix related tests 2019-01-26 00:10:29 -06:00
Chris Lu 5668ed798d memory needle map mark size to be TombstoneFileSize 
fix https://github.com/chrislusf/seaweedfs/issues/850
 2019-01-26 00:05:44 -06:00
Chris Lu 141d302492 fix binarySearchCompactSection 2018-12-22 11:52:58 -08:00
Chris Lu 36d13355bb compact map fix bug when iterating 2018-12-22 11:05:29 -08:00
Chris Lu 42cb9b76c4 reduce needle map memory usage by 25% 2018-12-15 05:55:56 -08:00
Chris Lu 4a490fe35a go fmt 2018-12-09 01:27:25 -08:00
Chris Lu 5741134ca0 change overflow from map to slice 2018-12-09 01:27:11 -08:00
Chris Lu df4b9df142 add memory usage report 2018-12-09 00:15:23 -08:00
Chris Lu ec5a305624 refactor 2018-12-09 00:12:37 -08:00
Chris Lu 9d3be33e5c optimize a little bit 2018-12-08 21:45:14 -08:00
Chris Lu 7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu 5b1fd374be fix all tests 2018-07-08 02:39:04 -07:00
Chris Lu d4d7ced922 refactoring: add type for needle id, offset 
later the type size can possibly be adjusted
 2018-07-08 02:28:04 -07:00
Chris Lu 82c3ccc8dd add btree for volume index 2017-05-26 22:51:25 -07:00