From 4b58edc827d4eef1be2dc66d94e4ea208e5e965c Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Mon, 20 Jul 2020 22:58:07 -0700 Subject: [PATCH] Updated Data Structure for Large Files (markdown) --- Data-Structure-for-Large-Files.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/Data-Structure-for-Large-Files.md b/Data-Structure-for-Large-Files.md index 6405a4e..458488a 100644 --- a/Data-Structure-for-Large-Files.md +++ b/Data-Structure-for-Large-Files.md @@ -24,6 +24,11 @@ SeaweedFS adds a manifest chunk to hold 1000 pieces of chunk info. This manifest For example, one super large file with 1000 manifest chunks is still 400KB in meta data in the key value store, but it can address a file of 8MB * 1000 * 1000 = 8TB in size. -This manifest chunk can also be easily implemented recursively, which can leads to even larger file size limit. But given the size limit is already very high and is beyond most common use cases, we will add the recursion later when really necessary. - The 8TB for one file is not a real limit, but only used for this discussion. However, if you have the need for files more than 8TB, please prepare some consulting fee and we will help you. :) Or you probably should already split the large files into smaller files. + +## Note + +This manifest chunk can also be easily implemented recursively, which can leads to even larger file size limit. But there are a few reasons of not going there: + +* The recursion leads to multi-level of indirection, and make the access time unpredictable. SeaweedFS tries to minimize this as much as possible. +* The size limit is already very high. It is way beyond what most common use cases would ever need. We will add the recursion later when really necessary.