From 5defc5d23172fd94b2ebb43a521928d7577dad00 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Fri, 20 May 2016 11:55:06 -0700 Subject: [PATCH] Updated Large File Handling (markdown) --- Large-File-Handling.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/Large-File-Handling.md b/Large-File-Handling.md index 026ef8f..b7841f6 100644 --- a/Large-File-Handling.md +++ b/Large-File-Handling.md @@ -1,17 +1,14 @@ In order to achieve high concurrency, SeaweedFS tries to read and write the whole file into memory. But this would not work for large files. +The following is implemented in "weed upload" command. For 3rd party clients, here is the spec. + To support large files, SeaweedFS supports these two kinds of files: -* Chunk data File -* Chunk Manifest +* Chunk File. Each chunk file is actually just normal files to SeaweedFS. +* Chunk Manifest. A simple json file with the list of all the chunks. -## Create new large file - -SeaweedFS delegates the effort to the client side. The steps are: - -1. split large files into chunks -1. upload each file chunks as usual, with mime type "application/octet-stream". Save the related info into ChunkInfo struct. -1. upload the manifest file with mime type "application/json", and add url parameter "cm=true". +This piece of code shows the json file structure: +https://github.com/chrislusf/seaweedfs/blob/master/go/operation/chunked_file.go#L24 ``` type ChunkInfo struct { @@ -30,6 +27,17 @@ type ChunkManifest struct { } ``` +When reading Chunk Manifest files, the SeaweedFS will find and send the data file based on the list of ChunkInfo. + +## Create new large file + +SeaweedFS delegates the effort to the client side. The steps are: + +1. split large files into chunks +1. upload each file chunks as usual, with mime type "application/octet-stream". Save the related info into ChunkInfo struct. +1. upload the manifest file with mime type "application/json", and add url parameter "cm=true". + + ## Update large file Usually we just append large files. Updating a specific chunk of file is almost the same. @@ -37,4 +45,4 @@ Usually we just append large files. Updating a specific chunk of file is almost The steps to append a large file: 1. upload the new file chunks as usual, with mime type "application/octet-stream". Save the related info into ChunkInfo struct. -1. update the updated manifest file with mime type "application/json", and add url parameter "cm=true". +1. update the updated manifest file with mime type "application/json", and add url parameter "cm=true". \ No newline at end of file