gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Large File Handling (markdown)

Chris Lu 2016-05-20 11:55:06 -07:00
parent b2d40cc67a
commit 5defc5d231
1 changed files with 18 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
Large-File-Handling.md

@ -1,17 +1,14 @@
In order to achieve high concurrency, SeaweedFS tries to read and write the whole file into memory. But this would not work for large files.
The following is implemented in "weed upload" command. For 3rd party clients, here is the spec.
To support large files, SeaweedFS supports these two kinds of files:
* Chunk data File
* Chunk Manifest
* Chunk File. Each chunk file is actually just normal files to SeaweedFS.
* Chunk Manifest. A simple json file with the list of all the chunks.
## Create new large file
SeaweedFS delegates the effort to the client side. The steps are:
1. split large files into chunks
1. upload each file chunks as usual, with mime type "application/octet-stream". Save the related info into ChunkInfo struct.
1. upload the manifest file with mime type "application/json", and add url parameter "cm=true".
This piece of code shows the json file structure:
https://github.com/chrislusf/seaweedfs/blob/master/go/operation/chunked_file.go#L24
```
type ChunkInfo struct {
@ -30,6 +27,17 @@ type ChunkManifest struct {
}
```
When reading Chunk Manifest files, the SeaweedFS will find and send the data file based on the list of ChunkInfo.
## Create new large file
SeaweedFS delegates the effort to the client side. The steps are:
1. split large files into chunks
1. upload each file chunks as usual, with mime type "application/octet-stream". Save the related info into ChunkInfo struct.
1. upload the manifest file with mime type "application/json", and add url parameter "cm=true".
## Update large file
Usually we just append large files. Updating a specific chunk of file is almost the same.
@ -37,4 +45,4 @@ Usually we just append large files. Updating a specific chunk of file is almost
The steps to append a large file:
1. upload the new file chunks as usual, with mime type "application/octet-stream". Save the related info into ChunkInfo struct.
1. update the updated manifest file with mime type "application/json", and add url parameter "cm=true".
1. update the updated manifest file with mime type "application/json", and add url parameter "cm=true".