From 878f28ee73cd2754b6a4c6b5ef16a49ad0b16f9c Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Tue, 19 Jan 2021 18:55:48 -0800 Subject: [PATCH] Updated Filer Stores (markdown) --- Filer-Stores.md | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/Filer-Stores.md b/Filer-Stores.md index cb6ab8b..f1791b3 100644 --- a/Filer-Stores.md +++ b/Filer-Stores.md @@ -2,24 +2,27 @@ The Filer Store persists all file metadata and directory information. -| Filer Store Name | Lookup | number of entries in a folder | Scalability | Directory Renaming | TTL | Note | -| ---------------- | -- | -- | -- | -- | -- | -- | -| memory | O(1) | limited by memory | Local, Fast | | Yes| for testing only, no persistent storage | -| leveldb | O(logN)| unlimited | Local, Very Fast | | Yes| Default, fairly scalable | -| leveldb2 | O(logN)| unlimited | Local, Very Fast | | Yes| Similar to leveldb, part of the lookup key is 128bit MD5 instead of the long full file path | -| leveldb3 | O(logN)| unlimited | Local, Very Fast | | Yes| Similar to leveldb2, separate leveldb instance for each bucket | -| RocksDB | O(logN)| unlimited | Local, Very Fast | | Native| Default, fairly scalable | -| Mongodb | O(logN)| unlimited | Local or Distributed, Fast | | Yes| Easy to manage | -| Redis | O(1) | limited | Local or Distributed, Fastest ||Native| one directory's sub file names are stored in one key~value entry | -| Cassandra | O(logN)| unlimited | Local or Distributed, Very Fast||Native| | -| MySql | O(logN)| unlimited | Local or Distributed, Fast |Atomic| Yes| Easy to manage | -| Postgres | O(logN)| unlimited | Local or Distributed, Fast |Atomic| Yes| Easy to manage | -| MemSql | O(logN)| unlimited | Distributed, Fast |Atomic| Yes| Scalable | -| TiDB | O(logN)| unlimited | Distributed, Fast |Atomic| Yes| Scalable | -| CockroachDB | O(logN)| unlimited | Distributed, Fast |Atomic| Yes| Scalable | -| Etcd | O(logN)| ~10GB | Distributed, 10,000 writes/sec || Yes| No SPOF. High Availability.| -| ElasticSearch| O(logN)| unlimited | Distributed, Fast || Yes| Scalable, Searchable | -| HBase | O(logN)| unlimited | Distributed, Fast | | Native| Scalable | +| Filer Store Name | Lookup | number of entries in a folder | Scalability | Directory Renaming | TTL | Fast Bucket Deletion |Note | +| ---------------- | -- | -- | -- | -- | -- | -- | -- | +| memory | O(1) | limited by memory | Local, Fast | | Yes| | for testing only, no persistent storage | +| leveldb | O(logN)| unlimited | Local, Very Fast | | Yes| | Default, fairly scalable | +| leveldb2 | O(logN)| unlimited | Local, Very Fast | | Yes| | Similar to leveldb, part of the lookup key is 128bit MD5 instead of the long full file path | +| leveldb3 | O(logN)| unlimited | Local, Very Fast | | Yes| Yes| Similar to leveldb2, separate leveldb instance for each bucket | +| RocksDB | O(logN)| unlimited | Local, Very Fast | | Native| | Default, fairly scalable | +| Mongodb | O(logN)| unlimited | Local or Distributed, Fast | | Yes| | Easy to manage | +| Redis | O(1) | limited | Local or Distributed, Fastest ||Native| | one directory's sub file names are stored in one key~value entry | +| Cassandra | O(logN)| unlimited | Local or Distributed, Very Fast||Native| | | +| MySql | O(logN)| unlimited | Local or Distributed, Fast |Atomic| Yes| | Easy to manage | +| MySql2 | O(logN)| unlimited | Local or Distributed, Fast |Atomic| Yes| Yes| Easy to manage | +| Postgres | O(logN)| unlimited | Local or Distributed, Fast |Atomic| Yes| | Easy to manage | +| Postgres2 | O(logN)| unlimited | Local or Distributed, Fast |Atomic| Yes| Yes| Easy to manage | +| MemSql | O(logN)| unlimited | Distributed, Fast |Atomic| Yes| Yes| Scalable | +| TiDB | O(logN)| unlimited | Distributed, Fast |Atomic| Yes| Yes| Scalable | +| CockroachDB | O(logN)| unlimited | Distributed, Fast |Atomic| Yes| Yes| Scalable | +| YugabyteDB | O(logN)| unlimited | Distributed, Fast |Atomic| Yes| Yes| Scalable | +| Etcd | O(logN)| ~10GB | Distributed, 10,000 writes/sec || Yes| | No SPOF. High Availability.| +| ElasticSearch| O(logN)| unlimited | Distributed, Fast || Yes| | Scalable, Searchable | +| HBase | O(logN)| unlimited | Distributed, Fast | | Native| | Scalable | #### Switching between different Stores It is easy to switch between different filer stores.