mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2024-01-19 02:48:24 +00:00
Updated Large File Handling (markdown)
parent
b2d40cc67a
commit
5defc5d231
|
@ -1,17 +1,14 @@
|
||||||
In order to achieve high concurrency, SeaweedFS tries to read and write the whole file into memory. But this would not work for large files.
|
In order to achieve high concurrency, SeaweedFS tries to read and write the whole file into memory. But this would not work for large files.
|
||||||
|
|
||||||
|
The following is implemented in "weed upload" command. For 3rd party clients, here is the spec.
|
||||||
|
|
||||||
To support large files, SeaweedFS supports these two kinds of files:
|
To support large files, SeaweedFS supports these two kinds of files:
|
||||||
* Chunk data File
|
* Chunk File. Each chunk file is actually just normal files to SeaweedFS.
|
||||||
* Chunk Manifest
|
* Chunk Manifest. A simple json file with the list of all the chunks.
|
||||||
|
|
||||||
## Create new large file
|
This piece of code shows the json file structure:
|
||||||
|
|
||||||
SeaweedFS delegates the effort to the client side. The steps are:
|
|
||||||
|
|
||||||
1. split large files into chunks
|
|
||||||
1. upload each file chunks as usual, with mime type "application/octet-stream". Save the related info into ChunkInfo struct.
|
|
||||||
1. upload the manifest file with mime type "application/json", and add url parameter "cm=true".
|
|
||||||
|
|
||||||
|
https://github.com/chrislusf/seaweedfs/blob/master/go/operation/chunked_file.go#L24
|
||||||
|
|
||||||
```
|
```
|
||||||
type ChunkInfo struct {
|
type ChunkInfo struct {
|
||||||
|
@ -30,6 +27,17 @@ type ChunkManifest struct {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
When reading Chunk Manifest files, the SeaweedFS will find and send the data file based on the list of ChunkInfo.
|
||||||
|
|
||||||
|
## Create new large file
|
||||||
|
|
||||||
|
SeaweedFS delegates the effort to the client side. The steps are:
|
||||||
|
|
||||||
|
1. split large files into chunks
|
||||||
|
1. upload each file chunks as usual, with mime type "application/octet-stream". Save the related info into ChunkInfo struct.
|
||||||
|
1. upload the manifest file with mime type "application/json", and add url parameter "cm=true".
|
||||||
|
|
||||||
|
|
||||||
## Update large file
|
## Update large file
|
||||||
|
|
||||||
Usually we just append large files. Updating a specific chunk of file is almost the same.
|
Usually we just append large files. Updating a specific chunk of file is almost the same.
|
||||||
|
|
Loading…
Reference in a new issue