Merge branch 'master' of github.com:chrislusf/weed-fs

docs/benchmarks.rst

@@ -1,7 +1,10 @@
 Benchmarks
 ======================
 
-Do we really need the benchmark? People always use benchmark to compare systems. But benchmarks are misleading. The resources, e.g., CPU, disk, memory, network, all matter a lot. And with Seaweed File System, single node vs multiple nodes, benchmarking on one machine vs several multiple machines, all matter a lot.
+Do we really need the benchmark? People always use benchmark to compare systems. 
+But benchmarks are misleading. The resources, e.g., CPU, disk, memory, network, 
+all matter a lot. And with Seaweed File System, single node vs multiple nodes, 
+benchmarking on one machine vs several multiple machines, all matter a lot.
 
 Here is the steps on how to run benchmark if you really need some numbers.
 
@@ -25,9 +28,13 @@ For more realistic tests, please start them on different machines.
 What does the test do?
 #############################
 
-By default, the benchmark command would start writing 1 million files, each having 1KB size, uncompressed. For each file, one request is sent to assign a file key, and a second request is sent to post the file to the volume server. The written file keys are stored in a temp file.
+By default, the benchmark command would start writing 1 million files, each having 1KB size, uncompressed. 
+For each file, one request is sent to assign a file key, and a second request is sent to post the file to the volume server. 
+The written file keys are stored in a temp file.
 
-Then the benchmark command would read the list of file keys, randomly read 1 million files. For each volume, the volume id is cached, so there is several request to lookup the volume id, and all the rest requests are to get the file content.
+Then the benchmark command would read the list of file keys, randomly read 1 million files. 
+For each volume, the volume id is cached, so there is several request to lookup the volume id, 
+and all the rest requests are to get the file content.
 
 Many options are options are configurable. Please check the help content:
 
@@ -35,25 +42,11 @@ Many options are options are configurable. Please check the help content:
 
   ./weed benchmark -h
 
-Common Problems
+Different Benchmark Target
 ###############################
 
-The most common
+The default "weed benchmark" uses 1 million 1KB file. This is to stress the number of files per second. 
-I start weed servers in one console for simplicity. Better run servers on different consoles.
+Increasing the file size to 100KB or more can show much larger number of IO throughput in KB/second.
-
-For more realistic tests, please start them on different machines.
-
-.. code-block:: bash
-
-  # prepare directories
-  mkdir 3 4 5
-  # start 3 servers
-  ./weed server -dir=./3 -master.port=9333 -volume.port=8083 &
-  ./weed volume -dir=./4 -port=8084 &
-  ./weed volume -dir=./5 -port=8085 &
-  ./weed benchmark -server=localhost:9333
-
- problem is "too many open files" error. This is because the test itself starts too many network connections on one single machine. In my local macbook, if I ran "random read" following writing right away, the error happens always. I have to run "weed benchmark -write=false" to run the reading test only. Also, changing the concurrency level to "-c=16" would also help.
 
 My own unscientific single machine results
 ###################################################
@@ -171,4 +164,6 @@ Create benchmark volumes directly
      99%      9.4 ms
     100%    256.9 ms
 How can the replication 001 writes faster than no replication?
-I could not tell. Very likely, the computer was in turbo mode. I can not reproduce it consistently either. Posted the number here just to illustrate that number lies. Don't quote on the exact number, just get an idea of the performance would be good enough.
+I could not tell. Very likely, the computer was in turbo mode. 
+I can not reproduce it consistently either. Posted the number here just to illustrate that number lies. 
+Don't quote on the exact number, just get an idea of the performance would be good enough.

docs/changelist.rst

@@ -0,0 +1,275 @@
+Change List
+===================================
+
+Introduction
+############
+This file contains list of recent changes, important features, usage changes, data format changes, etc. Do read this if you upgrade.
+
+
+v0.67
+#####
+1. Increase "weed benchmark" performance to pump in more data. The bottleneck is on the client side. Duh...
+
+v0.65
+#####
+
+1. Reset the cluster configuration if "-peers" is not empty.
+
+v0.64
+#####
+
+1. Add TTL support!
+1. filer: resolve directory log file error, avoid possible race condition
+
+v0.63
+#####
+
+1. Compiled with Go 1.3.1 to fix a rare crashing issue.
+
+v0.62
+#####
+
+1. Add support for Etag.
+2. Add /admin/mv to move a file or a folder.
+3. Add client Go API to pre-process the images.
+
+v0.61
+#####
+
+1. Reduce memory requirements for "weed fix"
+2. Guess mime type by file name extensions when stored mime type is "application/octstream"
+3. Added simple volume id lookup caching expiring by time.
+
+v0.60
+#####
+
+Fix file missing error caused by .idx file overwriting. The problem shows up if the weed volume server is restarted after 2 times. But the actual .idx file may have already been overwritten on second restart.
+
+To fix this issue, please run "weed fix -dir=... -volumeId=..." to re-generate the .idx file.
+
+v0.59
+#####
+
+1. Add option to automatically fix jpeg picture orientation.
+2. Add volume id lookup caching
+3. Support Partial Content and Range Requests. http status code == 206.
+
+v0.57
+#####
+
+Add hidden dynamic image resizing feature
+
+Add an hidden feature: For images, jpg/png/gif, if you specify append these url parameters, &width=xxx or &height=xxx or both, the image will be dynamically resized. However, resizing the image would cause high CPU and memory usage. Not recommended unless special use cases. So this would not be documented anywhere else.
+
+v0.56 Major Command line options change
+#####
+
+
+Adjust command line options.
+
+1. switch to use -publicIp instead of -publicUrl
+2. -ip can be empty. It will listen to all available interfaces.
+3. For "weed server", these options are changed:
+  - -masterPort => -master.port
+  - -peers => -master.peers
+  - -mdir => -master.dir
+  - -volumeSizeLimitMB => -master.volumeSizeLimitMB
+  - -conf => -master.conf
+  - -defaultReplicaPlacement => -master.defaultReplicaPlacement
+  - -port => -volume.port
+  - -max => -volume.max
+
+v0.55 Recursive folder deletion for Filer
+#####
+
+Now folders with sub folders or files can be deleted recursively.
+
+Also, for filer, avoid showing files under the first created directory when listing the root directory.
+
+v0.54 Misc improvements
+#####
+
+No need to persist metadata for master sequence number generation. This shall avoid possible issues where file are lost due to duplicated sequence number generated in rare cases.
+
+More robust handing of "peers" in master node clustering mode.
+
+Added logging instructions.
+
+v0.53  Miscellaneous improvements
+#####
+
+Added retry logic to wait for cluster peers during cluster bootstrapping. Previously the cluster bootstrapping is ordered. This make it tricky to deploy automatically and repeatedly. The fix make the commands repeatable.
+
+Also, when growing volumes, additional preferred "rack" and "dataNode" parameters are also provided, works together with existing "dataCenter" parameter.
+
+Fix important bug where settings for non-"000" replications are read back wrong, if volume server is restarted.
+
+v0.52 Added "filer" server
+#####
+
+A "weed filer" server is added, to provide more "common" file storage. Currently the fullFileName-to-fileId mapping is stored with an efficient embedded leveldb. So it's not linearly scalable yet. But it can handle LOTS of files.
+
+
+.. code-block:: bash
+
+  //POST a file and read it back
+  curl -F "filename=@README.md" "http://localhost:8888/path/to/sources/"
+  curl "http://localhost:8888/path/to/sources/README.md"
+  //POST a file with a new name and read it back
+  curl -F "filename=@Makefile" "http://localhost:8888/path/to/sources/new_name"
+  curl "http://localhost:8888/path/to/sources/new_name"
+  //list sub folders and files
+  curl "http://localhost:8888/path/to/sources/?pretty=y"
+
+
+v0.51 Idle Timeout
+#####
+
+Previously the timeout setting is "-readTimeout", which is the time limit of the whole http connection. This is inconvenient for large files or for slow internet connections.  Now this option is replaced with "-idleTimeout", and default to 10 seconds. Ideally, you should not need to tweak it based on your use case.
+
+v0.50 Improved Locking
+#####
+
+1. All read operation switched to thread-safe pread, no read locks now.
+2. When vacuuming large volumes, a lock was preventing heartbeats to master node. This is fixed now.
+3. Fix volume compaction error for collections.
+
+v0.49 Bug Fixes
+#####
+
+With the new benchmark tool to bombard the system, many bugs are found and fixed, especially on clustering, http connection reuse.
+
+v0.48 added benchmark command!
+#####
+
+Benchmark! Enough said.
+
+v0.47 Improving replication
+#####
+
+Support more replication types.
+
+v0.46 Adding failover master server
+#####
+
+Automatically fail over master servers!
+
+v0.46 Add "weed server" command
+#####
+
+Now you can start one master server and one volume server in just one command!
+
+.. code-block:: bash
+
+ weed server
+
+
+v0.45 Add support for extra large file
+#####
+
+For extra large file, this example will split the file into 100MB chunks.
+
+.. code-block:: bash
+
+ weed upload -maxMB=100 the_file_name
+
+
+Also, Added "download" command, for simple files or chunked files.
+
+.. code-block:: bash
+
+ weed download file_id [file_id3](file_id2)
+
+
+v0.34 Add support for multiple directories on volume server
+#####
+
+For volume server, add support for multiple folders and multiple max limit. For example:
+
+.. code-block:: bash
+
+ weed volume -dir=folder1,folder2,folder3 -max=7,8,9
+
+
+v0.33 Add Nicer URL support
+#####
+
+For HTTP GET request
+
+.. code-block:: bash
+
+  http://localhost:8080/3,01637037d6
+
+Can also be retrieved by
+
+.. code-block:: bash
+
+  http://localhost:8080/3/01637037d6/my_preferred_name.jpg
+
+
+v0.32 Add support for Last-Modified header
+#####
+
+The last modified timestamp is stored with 5 additional bytes.
+
+Return http code 304 if the file is not modified.
+
+Also, the writing are more solid with the fix for issue#26.
+
+v0.31 Allocate File Key on specific data center
+#####
+
+Volume servers can start with a specific data center name.
+
+.. code-block:: bash
+
+ weed volume -dir=/tmp/1 -port=8080 -dataCenter=dc1
+ weed volume -dir=/tmp/2 -port=8081 -dataCenter=dc2
+
+Or the master server can determine the data center via volume server's IP address and settings in weed.conf file.
+
+Now when requesting a file key, an optional "dataCenter" parameter can limit the assigned volume to the specific data center. For example, this specif
+
+.. code-block:: bash
+
+ http://localhost:9333/dir/assign?dataCenter=dc1
+
+v0.26 Storing File Name and Mime Type
+#####
+
+In order to keep one single disk read for each file, a new storage format is implemented to store: is gzipped or not, file name and mime type (used when downloading files), and possibly other future new attributes.  The volumes with old storage format are treated as read only and deprecated.
+
+Also, you can pre-gzip and submit your file directly, for example, gzip "my.css" into "my.css.gz", and submit. In this case, "my.css" will be stored as the file name. This should save some transmission time, and allow you to force gzipped storage or customize the gzip compression level.
+
+v0.25 Adding reclaiming garbage spaces
+
+Garbage spaces are reclaimed by an automatic compacting process. Garbage spaces are generated when updating or deleting files. If they exceed a configurable threshold, 0.3 by default (meaning 30% of the used disk space is garbage), the volume will be marked as readonly, compacted and garbage spaces are reclaimed, and then marked as writable.
+
+v0.19 Adding rack and data center aware replication
+#####
+
+Now when you have one rack, or multiple racks, or multiple data centers, you can choose your own replication strategy.
+
+v0.18 Detect disconnected volume servers
+#####
+
+The disconnected volume servers would not be assigned when generating the file keys. Volume servers by default send a heartbeat to master server every 5~10 seconds. Master thinks the volume server is disconnected after 5 times of the heartbeat interval, or 25 seconds by default.
+
+v0.16 Change to single executable file to do everything
+#####
+
+If you are using v0.15 or earlier, you would use
+
+.. code-block:: bash
+
+  >weedvolume -dir="/tmp" -volumes=0-4 -mserver="localhost:9333" -port=8080 -publicUrl="localhost:8080"
+
+With v0.16 or later, you would need to do this in stead:
+
+.. code-block:: bash
+
+  >weed volume -dir="/tmp" -volumes=0-4 -mserver="localhost:9333" -port=8080 -publicUrl="localhost:8080"
+
+And more new commands, in addition to "server","volume","fix", etc, will be added.
+
+This provides a simple deliverable file, and the file size is much smaller since Go language statically compile the commands. Combining commands into one file would avoid lots of duplication.

docs/clients.rst

@@ -5,7 +5,7 @@ Clients
 ###################################
 +---------------------------------------------------------------------------------+--------------+-----------+
 | Name                                                                            |  Author      |  Language |  
-+---------------------------------------------------------------------------------+--------------+-----------+
++=================================================================================+==============+===========+
 | `WeedPHP <https://github.com/micjohnson/weed-php/>`_                            | Mic Johnson  |  PHP      |
 +---------------------------------------------------------------------------------+--------------+-----------+
 | `Seaweed-FS Symfony bundle <https://github.com/micjohnson/weed-php-bundle>`_    | Mic Johnson  |  PHP      |

docs/conf.py

@@ -51,9 +51,9 @@ copyright = u'2014, chrislusf, ernado'
 # built documents.
 #
 # The short X.Y version.
-version = '0.63'
+version = '0.67'
 # The full version, including alpha/beta/rc tags.
-release = '0.63'
+release = '0.67'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/directories.rst

@@ -39,7 +39,7 @@ A common file system would use inode to store meta data for each folder and file
 
 Seaweed-FS wants to make as small number of disk access as possible, yet still be able to store a lot of file metadata. So we need to think very differently.
 
-From a full file path to get to the file content, there are several steps:
+We can take the following steps to map a full file path to the actual data block:
 
 .. code-block:: bash
 
@@ -48,7 +48,7 @@ From a full file path to get to the file content, there are several steps:
 	file_id => data_block
 
 
-Because default Seaweed-FS only provides file_id=>data_block mapping, the first 2 steps need to be implemented.
+Because default Seaweed-FS only provides file_id=>data_block mapping, only the first 2 steps need to be implemented.
 
 
 There are several data features I noticed:
@@ -72,7 +72,7 @@ I believe these are reasonable assumptions:
 Data structure
 #################
 
-This difference lead to the design that the metadata for directories and files should have different data structure.
+This assumed differences between directories and files lead to the design that the metadata for directories and files should have different data structure.
 
 * Store directories in memory
 
@@ -100,16 +100,18 @@ For file renaming, it's just trivially delete and then add a row in leveldb.
 Details
 ########################
 
-In the current first version, the path_to_file=>file_id mapping is stored with an efficient embedded leveldb. Being embedded, it runs on single machine. So it's not linearly scalable yet. However, it can handle LOTS AND LOTS of files on weed-fs on other servers. Using an external distributed database is possible. Your contribution is welcome!
+In the current first version, the path_to_file=>file_id mapping is stored with an efficient embedded leveldb. Being embedded, it runs on single machine. So it's not linearly scalable yet. However, it can handle LOTS AND LOTS of files on Seaweed-FS on other master/volume servers.
 
-The in-memory directory structure can improve on memory efficiency. Current simple map in memory works when the number of directories is less than 1 million, which will use about 500MB memory. But I would highly doubt any common use case would have more than 100 directories.
+Switching from the embedded leveldb to an external distributed database is very feasible. Your contribution is welcome!
+
+The in-memory directory structure can improve on memory efficiency. Current simple map in memory works when the number of directories is less than 1 million, which will use about 500MB memory. But I would expect common use case would have a few, not even more than 100 directories.
 
 Use Cases
 #########################
 
 Clients can assess one "weed filer" via HTTP, list files under a directory, create files via HTTP POST, read files via HTTP POST directly.
 
-Although one "weed filer" can only sits in one machine, you can start multiple "weed filer" on several machines, each "weed filer" instance running in its own collection, having its own namespace, but sharing the same weed-fs.
+Although one "weed filer" can only sits in one machine, you can start multiple "weed filer" on several machines, each "weed filer" instance running in its own collection, having its own namespace, but sharing the same Seaweed-FS storage.
 
 Future
 ###################
@@ -127,6 +129,6 @@ Later, FUSE or HCFS plugins will be created, to really integrate Seaweed-FS to e
 Helps Wanted
 ########################
 
-This is a big step towards more interesting weed-fs usage and integration with existing systems.
+This is a big step towards more interesting Seaweed-FS usage and integration with existing systems.
 
 If you can help to refactor and implement other directory meta data, or file meta data storage, please do so.

docs/index.rst

@@ -15,7 +15,7 @@ For documents and bug reporting, Please visit
  http://weed-fs.googlecode.com
  
 For pre-compiled releases,
- https://bintray.com/chrislusf/Weed-FS/weed
+ https://bintray.com/chrislusf/Weed-FS/seaweed
 
 
 Contents:
@@ -27,11 +27,13 @@ Contents:
    clients
    api
    replication
+   ttl
    failover
-   usecases
    directories
+   usecases
    optimization
    benchmarks
+   changelist
 
 
 Indices and tables

docs/ttl.rst

@@ -0,0 +1,85 @@
+Store file with a Time To Live
+===================
+
+Introduction
+#############################
+
+Seaweed is a key~file store, and files can optionally expire with a Time To Live (TTL).
+
+How to use it?
+#############################
+
+Assume we want to store a file with TTL of 3 minutes.
+
+First, ask the master to assign a file id to a volume with a 3-minute TTL:
+
+.. code-block:: bash
+
+  > curl http://localhost:9333/dir/assign?ttl=3m
+  {"count":1,"fid":"5,01637037d6","url":"127.0.0.1:8080","publicUrl":"localhost:8080"}
+
+Secondly, use the file id to store on the volume server
+
+.. code-block:: bash
+
+  > curl -F "file=@x.go" http://127.0.0.1:8080/5,01637037d6?ttl=3m
+
+After writing, the file content will be returned as usual if read before the TTL expiry. But if read after the TTL expiry, the file will be reported as missing and return the http response status as not found.
+
+For next writes with ttl=3m, the same set of volumes with ttl=3m will be used until:
+
+1. the ttl=3m volumes are full. If so, new volumes will be created.
+2. there are no write activities for 3 minutes. If so, these volumes will be stopped and deleted.
+
+Advanced Usage
+#############################
+
+As you may have noticed, the "ttl=3m" is used twice! One for assigning file id, and one for uploading the actual file. The first one is for master to pick a matching volume, while the second one is written together with the file.
+
+These two TTL values are not required to be the same. As long as the volume TTL is larger than file TTL, it should be OK.
+
+This gives some flexibility to fine-tune the file TTL, while reducing the number of volume TTL variations, which simplifies managing the TTL volumes.
+
+Supported TTL format
+#############################
+
+The TTL is in the format of one integer number followed by one unit. The unit can be 'm', 'h', 'd', 'w', 'M', 'y'.
+
+Supported TTL format examples:
+
+- 3m: 3 minutes
+-  4h: 4 hours
+-  5d: 5 days
+-  6w: 6 weeks
+-  7M: 7 months
+-  8y: 8 years
+
+
+How efficient it is?
+#############################
+
+TTL seems easy to implement since we just need to report the file as missing if the time is over the TTL. However, the real difficulty is to efficiently reclaim disk space from expired files, similar to JVM memory garbage collection, which is a sophisticated piece of work with many man-years of effort.
+
+Memcached also supports TTL. It gets around this problem by putting entries into fix-sized slabs. If one slab is expired, no work is required and the slab can be overwritten right away. However, this fix-sized slab approach is not applicable to files since the file contents rarely fit in slabs exactly.
+
+Seaweed-FS efficiently resolves this disk space garbage collection problem with great simplicity. One of key differences from "normal" implementation is that the TTL is associated with the volume, together with each specific file.
+
+During the file id assigning step, the file id will be assigned to a volume with matching TTL. The volumes are checked periodically (every 5~10 seconds by default). If the latest expiration time has been reached, all the files in the whole volume will be all expired, and the volume can be safely deleted.
+
+Implementation Details
+#############################
+1. When assigning file key, the master would pick one TTL volume with matching TTL. If such volumes do not exist, create a few.
+2. Volume servers will write the file with expiration time. When serving file, if the file is expired, the file will be reported as not found.
+3. Volume servers will track each volume's largest expiration time, and stop reporting the expired volumes to the master server.
+4. Master server will think the previously existed volumes are dead, and stop assigning write requests to them.
+5. After about 10% of the TTL time, or at most 10 minutes, the volume servers will delete the expired volume.
+
+Deployment
+#############################
+
+For deploying to production, the TTL volume maximum size should be taken into consideration. If the writes are frequent, the TTL volume will grow to the max volume size. So when the disk space is not ample enough, it's better to reduce the maximum volume size.
+
+It's recommended not to mix the TTL volumes and non TTL volumes in the same cluster. This is because the volume maximum size, default to 30GB, is configured on the volume master at the cluster level.
+
+We could implement the configuration for max volume size for each TTL. However, it could get fairly verbose. Maybe later if it is strongly desired.
+