Merge branch 'master' of https://github.com/chrislusf/weed-fs

.project

@@ -5,11 +5,6 @@
 	<projects>
 	</projects>
 	<buildSpec>
-		<buildCommand>
-			<name>com.googlecode.goclipse.goBuilder</name>
-			<arguments>
-			</arguments>
-		</buildCommand>
 	</buildSpec>
 	<natures>
 		<nature>goclipse.goNature</nature>

.travis.yml

@@ -1,6 +1,5 @@
 language: go
 go: 
- - 1.2
  - 1.3
  - release
  - tip

Dockerfile

@@ -1,5 +1,5 @@
 FROM cydev/go
-RUN go get code.google.com/p/weed-fs/go/weed
+RUN go get github.com/chrislusf/weed-fs/go/weed
 EXPOSE 8080
 EXPOSE 9333
 VOLUME /data

README.md

@@ -5,16 +5,6 @@ Seaweed File System
 [![GoDoc](https://godoc.org/github.com/chrislusf/weed-fs/go?status.svg)](https://godoc.org/github.com/chrislusf/weed-fs/go)
 [![RTD](https://readthedocs.org/projects/weed-fs/badge/?version=latest)](http://weed-fs.readthedocs.org/en/latest/)
 
-## Usage
-
-```
-go get github.com/chrislusf/weed-fs/go/weed
-```
-
-## Reference
-
-For pre-compiled releases,
- https://bintray.com/chrislusf/Weed-FS/seaweed
 
 ## Introduction
 
@@ -246,12 +236,34 @@ More tools and documentation, on how to maintain and scale the system. For examp
 
 This is a super exciting project! And I need helpers!
 
-## Contributions ##
 
-To make contributions easier, I have mirrored a repo in github.com
+## Installation guide for users who are not familiar with golang
-```
+
- https://github.com/chrislusf/weed-fs.git
+step 1: install go on your machine and setup the environment by following the instructions from the following link:
-```
+
+https://golang.org/doc/install
+
+make sure you set up your $GOPATH
+
+
+step 2: also you may need to install Mercurial by following the instructions below
+
+http://mercurial.selenic.com/downloads
+
+
+step 3: download, compile, and install the project by executing the following command 
+
+go get github.com/chrislusf/weed-fs/go/weed
+
+once this is done, you should see the executable "weed" under $GOPATH/bin
+
+step 4: after you modify your code locally, you could start a local build by calling "go install" under $GOPATH/src/github.com/chrislusf/weed-fs/go/weed
+
+## Reference
+
+For pre-compiled releases,
+ https://bintray.com/chrislusf/Weed-FS/seaweed
+
 ## Disk Related topics ##
 
 ### Hard Drive Performance ###

docs/api.rst

@@ -187,10 +187,11 @@ Upload File Directly
 
 .. code-block:: bash
 
-	curl -F file=@/home/chris/myphoto.jpg http://localhost:8080/submit
+	curl -F file=@/home/chris/myphoto.jpg http://localhost:9333/submit
 	{"fid":"3,01fbe0dc6f1f38","fileName":"myphoto.jpg","fileUrl":"localhost:8080/3,01fbe0dc6f1f38","size":68231}
 
-This API is a little convenient. The volume server would contact the master to get an file id and store it to the right volume server(not necessarily itself).
+This API is just for convenience. The master server would get an file id and store the file to the right volume server.
+It is a convenient API and does not support different parameters when assigning file id. (or you can add the support and send a push request.)
 
 Delete File
 ***********************************

docs/benchmarks.rst

@@ -1,7 +1,7 @@
 Benchmarks
 ======================
 
-Do we really need the benchmark? People always use benchmark to compare systems. But benchmarks are misleading. The resources, e.g., CPU, disk, memory, network, all matter a lot. And with Weed File System, single node vs multiple nodes, benchmarking on one machine vs several multiple machines, all matter a lot.
+Do we really need the benchmark? People always use benchmark to compare systems. But benchmarks are misleading. The resources, e.g., CPU, disk, memory, network, all matter a lot. And with Seaweed File System, single node vs multiple nodes, benchmarking on one machine vs several multiple machines, all matter a lot.
 
 Here is the steps on how to run benchmark if you really need some numbers.
 
@@ -38,7 +38,22 @@ Many options are options are configurable. Please check the help content:
 Common Problems
 ###############################
 
-The most common problem is "too many open files" error. This is because the test itself starts too many network connections on one single machine. In my local macbook, if I ran "random read" following writing right away, the error happens always. I have to run "weed benchmark -write=false" to run the reading test only. Also, changing the concurrency level to "-c=16" would also help.
+The most common
+I start weed servers in one console for simplicity. Better run servers on different consoles.
+
+For more realistic tests, please start them on different machines.
+
+.. code-block:: bash
+
+  # prepare directories
+  mkdir 3 4 5
+  # start 3 servers
+  ./weed server -dir=./3 -master.port=9333 -volume.port=8083 &
+  ./weed volume -dir=./4 -port=8084 &
+  ./weed volume -dir=./5 -port=8085 &
+  ./weed benchmark -server=localhost:9333
+
+ problem is "too many open files" error. This is because the test itself starts too many network connections on one single machine. In my local macbook, if I ran "random read" following writing right away, the error happens always. I have to run "weed benchmark -write=false" to run the reading test only. Also, changing the concurrency level to "-c=16" would also help.
 
 My own unscientific single machine results
 ###################################################

docs/directories.rst

@@ -1,7 +1,7 @@
 Directories and files
 ===========================
 
-When talking about file systems, many people would assume directories, list files under a directory, etc. These are expected if we want to hook up Weed File System with linux by FUSE, or with Hadoop, etc.
+When talking about file systems, many people would assume directories, list files under a directory, etc. These are expected if we want to hook up Seaweed File System with linux by FUSE, or with Hadoop, etc.
 
 Sample usage
 #####################

docs/failover.rst

@@ -7,7 +7,7 @@ Introduction
 
 Some user will ask for no single point of failure. Although google runs its file system with a single master for years, no SPOF seems becoming a criteria for architects to pick solutions.
 
-Luckily, it's not too difficult to enable Weed File System with failover master servers.
+Luckily, it's not too difficult to enable Seaweed File System with failover master servers.
 
 Cheat Sheet: Startup multiple servers
 ########################################

docs/gettingstarted.rst

@@ -1,6 +1,6 @@
 Getting started
 ===================================
-Installing Weed-Fs
+Installing Seaweed-FS
 ###################################
 
 Download a proper version from  `Seaweed-FS download page <https://bintray.com/chrislusf/Weed-FS/weed/>`_.
@@ -57,7 +57,7 @@ Actually, forget about previous commands. You can setup one master server and on
 	# use "weed server -h" to find out more
 	./weed server -master.port=9333 -volume.port=8080 -dir="./data"
 
-Testing Weed-Fs
+Testing Seaweed-FS
 ###################################
 
 With the master and volume server up, now what? Let's pump in a lot of files into the system!
@@ -77,7 +77,7 @@ Then, you can simply check "du -m -s /some/big/folder" to see the actual disk us
 
 Now you can use your tools to hit weed-fs as hard as you can.
 
-Using Weed-Fs in docker
+Using Seaweed-FS in docker
 ####################################
 
 You can use image "cydev/weed" or build your own with  `dockerfile <https://github.com/chrislusf/weed-fs/blob/master/Dockerfile>`_  in the root of repo.

go/filer/client_operations.go

@@ -1,12 +1,12 @@
 package filer
 
-import ()
-
 import (
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
 	"fmt"
+
+	"github.com/chrislusf/weed-fs/go/util"
+
 	"net/url"
 )
 

go/filer/directory.go

@@ -1,7 +1,5 @@
 package filer
 
-import ()
-
 type DirectoryId int32
 
 type DirectoryEntry struct {

go/filer/directory_in_map.go

@@ -2,7 +2,6 @@ package filer
 
 import (
 	"bufio"
-	"github.com/chrislusf/weed-fs/go/util"
 	"fmt"
 	"io"
 	"os"
@@ -10,6 +9,8 @@ import (
 	"strconv"
 	"strings"
 	"sync"
+
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 var writeLock sync.Mutex //serialize changes to dir.log

go/filer/filer.go

@@ -1,7 +1,5 @@
 package filer
 
-import ()
-
 type FileId string //file id on weedfs
 
 type FileEntry struct {

go/filer/filer_embedded.go

@@ -1,11 +1,12 @@
 package filer
 
 import (
-	"github.com/chrislusf/weed-fs/go/operation"
 	"errors"
 	"fmt"
 	"path/filepath"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/operation"
 )
 
 type FilerEmbedded struct {

go/filer/files_in_leveldb.go

@@ -2,6 +2,7 @@ package filer
 
 import (
 	"bytes"
+
 	"github.com/chrislusf/weed-fs/go/glog"
 	"github.com/syndtr/goleveldb/leveldb"
 	"github.com/syndtr/goleveldb/leveldb/util"

go/glog/convenient_api.go

@@ -1,7 +1,5 @@
 package glog
 
-import ()
-
 /*
 Copying the original glog because it is missing several convenient methods.
 1. remove nano time in log format

go/images/orientation.go

@@ -2,11 +2,12 @@ package images
 
 import (
 	"bytes"
-	"github.com/rwcarlsen/goexif/exif"
 	"image"
 	"image/draw"
 	"image/jpeg"
 	"log"
+
+	"github.com/rwcarlsen/goexif/exif"
 )
 
 //many code is copied from http://camlistore.org/pkg/images/images.go

go/images/resizing.go

@@ -2,11 +2,12 @@ package images
 
 import (
 	"bytes"
-	"github.com/disintegration/imaging"
 	"image"
 	"image/gif"
 	"image/jpeg"
 	"image/png"
+
+	"github.com/disintegration/imaging"
 )
 
 func Resized(ext string, data []byte, width, height int) (resized []byte, w int, h int) {

go/operation/assign_file_id.go

@@ -1,12 +1,13 @@
 package operation
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
 	"net/url"
 	"strconv"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 type AssignResult struct {

go/operation/data_struts.go

@@ -1,7 +1,5 @@
 package operation
 
-import ()
-
 type JoinResult struct {
 	VolumeSizeLimit uint64 `json:"VolumeSizeLimit,omitempty"`
 	Error           string `json:"error,omitempty"`

go/operation/delete_content.go

@@ -1,12 +1,13 @@
 package operation
 
 import (
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
 	"net/url"
 	"strings"
 	"sync"
+
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 type DeleteResult struct {

go/operation/list_masters.go

@@ -1,9 +1,10 @@
 package operation
 
 import (
+	"encoding/json"
+
 	"github.com/chrislusf/weed-fs/go/glog"
 	"github.com/chrislusf/weed-fs/go/util"
-	"encoding/json"
 )
 
 type ClusterStatusResult struct {

go/operation/lookup.go

@@ -1,7 +1,6 @@
 package operation
 
 import (
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
 	_ "fmt"
@@ -9,6 +8,8 @@ import (
 	"net/url"
 	"strings"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 type Location struct {

go/operation/submit.go

@@ -2,13 +2,14 @@ package operation
 
 import (
 	"bytes"
-	"github.com/chrislusf/weed-fs/go/glog"
 	"io"
 	"mime"
 	"os"
 	"path"
 	"strconv"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 type FilePart struct {

go/operation/system_message_test.go

@@ -1,10 +1,11 @@
 package operation
 
 import (
-	proto "code.google.com/p/goprotobuf/proto"
 	"encoding/json"
 	"log"
 	"testing"
+
+	proto "code.google.com/p/goprotobuf/proto"
 )
 
 func TestSerialDeserial(t *testing.T) {

go/operation/upload_content.go

@@ -2,7 +2,6 @@ package operation
 
 import (
 	"bytes"
-	"github.com/chrislusf/weed-fs/go/glog"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -14,6 +13,8 @@ import (
 	"net/textproto"
 	"path/filepath"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 type UploadResult struct {

go/sequence/sequence.go

@@ -1,7 +1,5 @@
 package sequence
 
-import ()
-
 type Sequencer interface {
 	NextFileId(count int) (uint64, int)
 	SetMax(uint64)

go/stats/disk.go

@@ -1,7 +1,5 @@
 package stats
 
-import ()
-
 type DiskStatus struct {
 	Dir  string
 	All  uint64

go/stats/disk_notsupported.go

@@ -2,8 +2,6 @@
 
 package stats
 
-import ()
-
 func (disk *DiskStatus) fillInStatus() {
 	return
 }

go/stats/memory_notsupported.go

@@ -2,8 +2,6 @@
 
 package stats
 
-import ()
-
 func (mem *MemStatus) fillInStatus() {
 	return
 }

go/storage/cdb_map.go

@@ -1,13 +1,14 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
 	"fmt"
-	"github.com/tgulacsi/go-cdb"
 	"os"
 	"path/filepath"
+
+	"github.com/chrislusf/weed-fs/go/util"
+	"github.com/tgulacsi/go-cdb"
 )
 
 // CDB-backed read-only needle map

go/storage/cdb_map_test.go

@@ -1,11 +1,12 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
 	"math/rand"
 	"os"
 	"runtime"
 	"testing"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 var testIndexFilename string = "../../test/sample.idx"

go/storage/compact_map.go

@@ -1,7 +1,5 @@
 package storage
 
-import ()
-
 type NeedleValue struct {
 	Key    Key
 	Offset uint32 `comment:"Volume offset"` //since aligned to 8 bytes, range is 4G*8=32G

go/storage/compact_map_perf_test.go

@@ -1,11 +1,12 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/util"
 	"log"
 	"os"
 	"testing"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 func TestMemoryUsage(t *testing.T) {

go/storage/compress.go

@@ -2,11 +2,12 @@ package storage
 
 import (
 	"bytes"
-	"github.com/chrislusf/weed-fs/go/glog"
 	"compress/flate"
 	"compress/gzip"
 	"io/ioutil"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 /*

go/storage/crc.go

@@ -1,9 +1,10 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/util"
 	"fmt"
 	"hash/crc32"
+
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 var table = crc32.MakeTable(crc32.Castagnoli)

go/storage/file_id.go

@@ -1,11 +1,12 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/hex"
 	"errors"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 type FileId struct {

go/storage/needle.go

@@ -1,9 +1,6 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/images"
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/hex"
 	"errors"
 	"io/ioutil"
@@ -13,6 +10,10 @@ import (
 	"strconv"
 	"strings"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/images"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 const (

go/storage/needle_map.go

@@ -1,11 +1,12 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/util"
 	"fmt"
 	"io"
 	"os"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 type NeedleMapper interface {

go/storage/needle_read_write.go

@@ -1,12 +1,13 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/util"
 	"errors"
 	"fmt"
 	"io"
 	"os"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 const (

go/storage/store.go

@@ -1,10 +1,6 @@
 package storage
 
 import (
-	proto "code.google.com/p/goprotobuf/proto"
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/operation"
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -12,6 +8,11 @@ import (
 	"math/rand"
 	"strconv"
 	"strings"
+
+	proto "code.google.com/p/goprotobuf/proto"
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/operation"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 const (

go/storage/store_vacuum.go

@@ -1,9 +1,10 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
 	"fmt"
 	"strconv"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 func (s *Store) CheckCompactVolume(volumeIdString string, garbageThresholdString string) (error, bool) {

go/storage/volume.go

@@ -2,7 +2,6 @@ package storage
 
 import (
 	"bytes"
-	"github.com/chrislusf/weed-fs/go/glog"
 	"errors"
 	"fmt"
 	"io"
@@ -10,6 +9,8 @@ import (
 	"path"
 	"sync"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 type Volume struct {
@@ -72,7 +73,7 @@ func (v *Volume) load(alsoLoadIndex bool, createDatIfMissing bool) error {
 
 	if e != nil {
 		if !os.IsPermission(e) {
-			return fmt.Errorf("cannot load Volume Data %s.dat: %s", fileName, e.Error())
+			return fmt.Errorf("cannot load Volume Data %s.dat: %v", fileName, e)
 		}
 	}
 
@@ -92,12 +93,12 @@ func (v *Volume) load(alsoLoadIndex bool, createDatIfMissing bool) error {
 		if v.readOnly {
 			glog.V(1).Infoln("open to read file", fileName+".idx")
 			if indexFile, e = os.OpenFile(fileName+".idx", os.O_RDONLY, 0644); e != nil {
-				return fmt.Errorf("cannot read Volume Index %s.idx: %s", fileName, e.Error())
+				return fmt.Errorf("cannot read Volume Index %s.idx: %v", fileName, e)
 			}
 		} else {
 			glog.V(1).Infoln("open to write file", fileName+".idx")
 			if indexFile, e = os.OpenFile(fileName+".idx", os.O_RDWR|os.O_CREATE, 0644); e != nil {
-				return fmt.Errorf("cannot write Volume Index %s.idx: %s", fileName, e.Error())
+				return fmt.Errorf("cannot write Volume Index %s.idx: %v", fileName, e)
 			}
 		}
 		glog.V(0).Infoln("loading file", fileName+".idx", "readonly", v.readOnly)
@@ -115,7 +116,7 @@ func (v *Volume) Size() int64 {
 	if e == nil {
 		return stat.Size()
 	}
-	glog.V(0).Infof("Failed to read file size %s %s", v.dataFile.Name(), e.Error())
+	glog.V(0).Infof("Failed to read file size %s %v", v.dataFile.Name(), e)
 	return -1
 }
 func (v *Volume) Close() {
@@ -170,6 +171,7 @@ func (v *Volume) write(n *Needle) (size uint32, err error) {
 	}
 	var offset int64
 	if offset, err = v.dataFile.Seek(0, 2); err != nil {
+		glog.V(0).Infof("faile to seek the end of file: %v", err)
 		return
 	}
 
@@ -177,21 +179,21 @@ func (v *Volume) write(n *Needle) (size uint32, err error) {
 	if offset%NeedlePaddingSize != 0 {
 		offset = offset + (NeedlePaddingSize - offset%NeedlePaddingSize)
 		if offset, err = v.dataFile.Seek(offset, 0); err != nil {
-			glog.V(4).Infof("failed to align in datafile %s: %s", v.dataFile.Name(), err.Error())
+			glog.V(0).Infof("failed to align in datafile %s: %v", v.dataFile.Name(), err)
 			return
 		}
 	}
 
 	if size, err = n.Append(v.dataFile, v.Version()); err != nil {
 		if e := v.dataFile.Truncate(offset); e != nil {
-			err = fmt.Errorf("%s\ncannot truncate %s: %s", err, v.dataFile.Name(), e.Error())
+			err = fmt.Errorf("%s\ncannot truncate %s: %v", err, v.dataFile.Name(), e)
 		}
 		return
 	}
 	nv, ok := v.nm.Get(n.Id)
 	if !ok || int64(nv.Offset)*NeedlePaddingSize < offset {
 		if _, err = v.nm.Put(n.Id, uint32(offset/NeedlePaddingSize), n.Size); err != nil {
-			glog.V(4).Infof("failed to save in needle map %d: %s", n.Id, err.Error())
+			glog.V(4).Infof("failed to save in needle map %d: %v", n.Id, err)
 		}
 	}
 	if v.lastModifiedTime < n.LastModified {
@@ -292,13 +294,13 @@ func ScanVolumeFile(dirname string, collection string, id VolumeId,
 	offset := int64(SuperBlockSize)
 	n, rest, e := ReadNeedleHeader(v.dataFile, version, offset)
 	if e != nil {
-		err = fmt.Errorf("cannot read needle header: %s", e)
+		err = fmt.Errorf("cannot read needle header: %v", e)
 		return
 	}
 	for n != nil {
 		if readNeedleBody {
 			if err = n.ReadNeedleBody(v.dataFile, version, offset+int64(NeedleHeaderSize), rest); err != nil {
-				err = fmt.Errorf("cannot read needle body: %s", err)
+				err = fmt.Errorf("cannot read needle body: %v", err)
 				return
 			}
 		}
@@ -310,7 +312,7 @@ func ScanVolumeFile(dirname string, collection string, id VolumeId,
 			if err == io.EOF {
 				return nil
 			}
-			return fmt.Errorf("cannot read needle header: %s", err)
+			return fmt.Errorf("cannot read needle header: %v", err)
 		}
 	}
 
@@ -360,7 +362,7 @@ func (v *Volume) ensureConvertIdxToCdb(fileName string) (cdbCanRead bool) {
 	defer indexFile.Close()
 	glog.V(0).Infof("converting %s.idx to %s.cdb", fileName, fileName)
 	if e = ConvertIndexToCdb(fileName+".cdb", indexFile); e != nil {
-		glog.V(0).Infof("error converting %s.idx to %s.cdb: %s", fileName, fileName, e.Error())
+		glog.V(0).Infof("error converting %s.idx to %s.cdb: %v", fileName, fileName, e)
 		return false
 	}
 	return true

go/storage/volume_super_block.go

@@ -1,9 +1,10 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
 	"fmt"
 	"os"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 const (

go/storage/volume_vacuum.go

@@ -1,10 +1,11 @@
 package storage
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
 	"fmt"
 	"os"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 func (v *Volume) garbageLevel() float64 {

go/storage/volume_version.go

@@ -1,7 +1,5 @@
 package storage
 
-import ()
-
 type Version uint8
 
 const (

go/tools/read_index.go

@@ -1,11 +1,12 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/storage"
 	"flag"
 	"fmt"
 	"log"
 	"os"
+
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 var (

go/topology/allocate_volume.go

@@ -1,11 +1,12 @@
 package topology
 
 import (
-	"github.com/chrislusf/weed-fs/go/storage"
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
 	"net/url"
+
+	"github.com/chrislusf/weed-fs/go/storage"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 type AllocateVolumeResult struct {

go/topology/collection.go

@@ -2,17 +2,18 @@ package topology
 
 import (
 	"github.com/chrislusf/weed-fs/go/storage"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 type Collection struct {
 	Name                     string
 	volumeSizeLimit          uint64
-	storageType2VolumeLayout map[string]*VolumeLayout
+	storageType2VolumeLayout *util.ConcurrentReadMap
 }
 
 func NewCollection(name string, volumeSizeLimit uint64) *Collection {
 	c := &Collection{Name: name, volumeSizeLimit: volumeSizeLimit}
-	c.storageType2VolumeLayout = make(map[string]*VolumeLayout)
+	c.storageType2VolumeLayout = util.NewConcurrentReadMap()
 	return c
 }
 
@@ -21,16 +22,16 @@ func (c *Collection) GetOrCreateVolumeLayout(rp *storage.ReplicaPlacement, ttl *
 	if ttl != nil {
 		keyString += ttl.String()
 	}
-	if c.storageType2VolumeLayout[keyString] == nil {
+	vl := c.storageType2VolumeLayout.Get(keyString, func() interface{} {
-		c.storageType2VolumeLayout[keyString] = NewVolumeLayout(rp, ttl, c.volumeSizeLimit)
+		return NewVolumeLayout(rp, ttl, c.volumeSizeLimit)
-	}
+	})
-	return c.storageType2VolumeLayout[keyString]
+	return vl.(*VolumeLayout)
 }
 
 func (c *Collection) Lookup(vid storage.VolumeId) []*DataNode {
-	for _, vl := range c.storageType2VolumeLayout {
+	for _, vl := range c.storageType2VolumeLayout.Items {
 		if vl != nil {
-			if list := vl.Lookup(vid); list != nil {
+			if list := vl.(*VolumeLayout).Lookup(vid); list != nil {
 				return list
 			}
 		}
@@ -39,9 +40,9 @@ func (c *Collection) Lookup(vid storage.VolumeId) []*DataNode {
 }
 
 func (c *Collection) ListVolumeServers() (nodes []*DataNode) {
-	for _, vl := range c.storageType2VolumeLayout {
+	for _, vl := range c.storageType2VolumeLayout.Items {
 		if vl != nil {
-			if list := vl.ListVolumeServers(); list != nil {
+			if list := vl.(*VolumeLayout).ListVolumeServers(); list != nil {
 				nodes = append(nodes, list...)
 			}
 		}

go/topology/data_center.go

@@ -1,7 +1,5 @@
 package topology
 
-import ()
-
 type DataCenter struct {
 	NodeImpl
 }

go/topology/data_node.go

@@ -1,9 +1,10 @@
 package topology
 
 import (
+	"strconv"
+
 	"github.com/chrislusf/weed-fs/go/glog"
 	"github.com/chrislusf/weed-fs/go/storage"
-	"strconv"
 )
 
 type DataNode struct {

go/topology/node.go

@@ -1,11 +1,12 @@
 package topology
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/storage"
 	"errors"
 	"math/rand"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 type NodeId string

go/topology/store_replicate.go

@@ -2,12 +2,13 @@ package topology
 
 import (
 	"bytes"
+	"net/http"
+	"strconv"
+
 	"github.com/chrislusf/weed-fs/go/glog"
 	"github.com/chrislusf/weed-fs/go/operation"
 	"github.com/chrislusf/weed-fs/go/storage"
 	"github.com/chrislusf/weed-fs/go/util"
-	"net/http"
-	"strconv"
 )
 
 func ReplicatedWrite(masterNode string, s *storage.Store, volumeId storage.VolumeId, needle *storage.Needle, r *http.Request) (size uint32, errorStatus string) {

go/topology/topology.go

@@ -1,20 +1,22 @@
 package topology
 
 import (
+	"errors"
+	"io/ioutil"
+	"math/rand"
+
 	"github.com/chrislusf/weed-fs/go/glog"
 	"github.com/chrislusf/weed-fs/go/operation"
 	"github.com/chrislusf/weed-fs/go/sequence"
 	"github.com/chrislusf/weed-fs/go/storage"
-	"errors"
+	"github.com/chrislusf/weed-fs/go/util"
 	"github.com/goraft/raft"
-	"io/ioutil"
-	"math/rand"
 )
 
 type Topology struct {
 	NodeImpl
 
-	collectionMap map[string]*Collection
+	collectionMap *util.ConcurrentReadMap
 
 	pulse int64
 
@@ -37,7 +39,7 @@ func NewTopology(id string, confFile string, seq sequence.Sequencer, volumeSizeL
 	t.nodeType = "Topology"
 	t.NodeImpl.value = t
 	t.children = make(map[NodeId]Node)
-	t.collectionMap = make(map[string]*Collection)
+	t.collectionMap = util.NewConcurrentReadMap()
 	t.pulse = int64(pulse)
 	t.volumeSizeLimit = volumeSizeLimit
 
@@ -89,14 +91,14 @@ func (t *Topology) loadConfiguration(configurationFile string) error {
 func (t *Topology) Lookup(collection string, vid storage.VolumeId) []*DataNode {
 	//maybe an issue if lots of collections?
 	if collection == "" {
-		for _, c := range t.collectionMap {
+		for _, c := range t.collectionMap.Items {
-			if list := c.Lookup(vid); list != nil {
+			if list := c.(*Collection).Lookup(vid); list != nil {
 				return list
 			}
 		}
 	} else {
-		if c, ok := t.collectionMap[collection]; ok {
+		if c, ok := t.collectionMap.Items[collection]; ok {
-			return c.Lookup(vid)
+			return c.(*Collection).Lookup(vid)
 		}
 	}
 	return nil
@@ -109,7 +111,7 @@ func (t *Topology) NextVolumeId() storage.VolumeId {
 	return next
 }
 
-func (t *Topology) HasWriableVolume(option *VolumeGrowOption) bool {
+func (t *Topology) HasWritableVolume(option *VolumeGrowOption) bool {
 	vl := t.GetVolumeLayout(option.Collection, option.ReplicaPlacement, option.Ttl)
 	return vl.GetActiveVolumeCount(option) > 0
 }
@@ -124,20 +126,18 @@ func (t *Topology) PickForWrite(count int, option *VolumeGrowOption) (string, in
 }
 
 func (t *Topology) GetVolumeLayout(collectionName string, rp *storage.ReplicaPlacement, ttl *storage.TTL) *VolumeLayout {
-	_, ok := t.collectionMap[collectionName]
+	return t.collectionMap.Get(collectionName, func() interface{} {
-	if !ok {
+		return NewCollection(collectionName, t.volumeSizeLimit)
-		t.collectionMap[collectionName] = NewCollection(collectionName, t.volumeSizeLimit)
+	}).(*Collection).GetOrCreateVolumeLayout(rp, ttl)
-	}
-	return t.collectionMap[collectionName].GetOrCreateVolumeLayout(rp, ttl)
 }
 
-func (t *Topology) GetCollection(collectionName string) (collection *Collection, ok bool) {
+func (t *Topology) GetCollection(collectionName string) (*Collection, bool) {
-	collection, ok = t.collectionMap[collectionName]
+	c, hasCollection := t.collectionMap.Items[collectionName]
-	return
+	return c.(*Collection), hasCollection
 }
 
 func (t *Topology) DeleteCollection(collectionName string) {
-	delete(t.collectionMap, collectionName)
+	delete(t.collectionMap.Items, collectionName)
 }
 
 func (t *Topology) RegisterVolumeLayout(v storage.VolumeInfo, dn *DataNode) {

go/topology/topology_event_handling.go

@@ -1,10 +1,11 @@
 package topology
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/storage"
 	"math/rand"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 func (t *Topology) StartRefreshWritableVolumes(garbageThreshold string) {

go/topology/topology_map.go

@@ -1,7 +1,5 @@
 package topology
 
-import ()
-
 func (t *Topology) ToMap() interface{} {
 	m := make(map[string]interface{})
 	m["Max"] = t.GetMaxVolumeCount()
@@ -13,10 +11,11 @@ func (t *Topology) ToMap() interface{} {
 	}
 	m["DataCenters"] = dcs
 	var layouts []interface{}
-	for _, c := range t.collectionMap {
+	for _, col := range t.collectionMap.Items {
-		for _, layout := range c.storageType2VolumeLayout {
+		c := col.(*Collection)
+		for _, layout := range c.storageType2VolumeLayout.Items {
 			if layout != nil {
-				tmp := layout.ToMap()
+				tmp := layout.(*VolumeLayout).ToMap()
 				tmp["collection"] = c.Name
 				layouts = append(layouts, tmp)
 			}

go/topology/topology_vacuum.go

@@ -1,13 +1,14 @@
 package topology
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/storage"
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
 	"net/url"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/storage"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 func batchVacuumVolumeCheck(vl *VolumeLayout, vid storage.VolumeId, locationlist *VolumeLocationList, garbageThreshold string) bool {
@@ -79,13 +80,15 @@ func batchVacuumVolumeCommit(vl *VolumeLayout, vid storage.VolumeId, locationlis
 	return isCommitSuccess
 }
 func (t *Topology) Vacuum(garbageThreshold string) int {
-	for _, c := range t.collectionMap {
+	for _, col := range t.collectionMap.Items {
-		for _, vl := range c.storageType2VolumeLayout {
+		c := col.(*Collection)
+		for _, vl := range c.storageType2VolumeLayout.Items {
 			if vl != nil {
-				for vid, locationlist := range vl.vid2location {
+				volumeLayout := vl.(*VolumeLayout)
-					if batchVacuumVolumeCheck(vl, vid, locationlist, garbageThreshold) {
+				for vid, locationlist := range volumeLayout.vid2location {
-						if batchVacuumVolumeCompact(vl, vid, locationlist) {
+					if batchVacuumVolumeCheck(volumeLayout, vid, locationlist, garbageThreshold) {
-							batchVacuumVolumeCommit(vl, vid, locationlist)
+						if batchVacuumVolumeCompact(volumeLayout, vid, locationlist) {
+							batchVacuumVolumeCommit(volumeLayout, vid, locationlist)
 						}
 					}
 				}

go/topology/volume_growth.go

@@ -1,11 +1,12 @@
 package topology
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/storage"
 	"fmt"
 	"math/rand"
 	"sync"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 /*

go/topology/volume_growth_test.go

@@ -1,11 +1,12 @@
 package topology
 
 import (
-	"github.com/chrislusf/weed-fs/go/sequence"
-	"github.com/chrislusf/weed-fs/go/storage"
 	"encoding/json"
 	"fmt"
 	"testing"
+
+	"github.com/chrislusf/weed-fs/go/sequence"
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 var topologyLayout = `

go/topology/volume_layout.go

@@ -1,11 +1,12 @@
 package topology
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/storage"
 	"errors"
 	"math/rand"
 	"sync"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 // mapping from volume to its locations, inverted from server to volume

go/topology/volume_location_list.go

@@ -1,7 +1,5 @@
 package topology
 
-import ()
-
 type VolumeLocationList struct {
 	list []*DataNode
 }

go/util/concurrent_read_map.go

@@ -0,0 +1,37 @@
+package util
+
+import "sync"
+
+// A mostly for read map, which can thread-safely
+// initialize the map entries.
+type ConcurrentReadMap struct {
+	rmutex sync.RWMutex
+	mutex  sync.Mutex
+	Items  map[string]interface{}
+}
+
+func NewConcurrentReadMap() *ConcurrentReadMap {
+	return &ConcurrentReadMap{Items: make(map[string]interface{})}
+}
+
+func (m *ConcurrentReadMap) initMapEntry(key string, newEntry func() interface{}) (value interface{}) {
+	m.mutex.Lock()
+	defer m.mutex.Unlock()
+	if value, ok := m.Items[key]; ok {
+		return value
+	}
+	value = newEntry()
+	m.Items[key] = value
+	return value
+}
+
+func (m *ConcurrentReadMap) Get(key string, newEntry func() interface{}) interface{} {
+	m.rmutex.RLock()
+	if value, ok := m.Items[key]; ok {
+		m.rmutex.RUnlock()
+		return value
+	} else {
+		m.rmutex.RUnlock()
+		return m.initMapEntry(key, newEntry)
+	}
+}

go/util/config.go

@@ -10,9 +10,10 @@ package util
 
 import (
 	"bytes"
-	"github.com/chrislusf/weed-fs/go/glog"
 	"encoding/json"
 	"os"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 type Config struct {

go/util/constants.go

@@ -1,7 +1,5 @@
 package util
 
-import ()
-
 const (
-	VERSION = "0.64"
+	VERSION = "0.67"
 )

go/util/file_util.go

@@ -2,9 +2,10 @@ package util
 
 import (
 	"bufio"
-	"github.com/chrislusf/weed-fs/go/glog"
 	"errors"
 	"os"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 func TestFolderWritable(folder string) (err error) {

go/util/net_timeout.go

@@ -1,9 +1,10 @@
 package util
 
 import (
-	"github.com/chrislusf/weed-fs/go/stats"
 	"net"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/stats"
 )
 
 // Listener wraps a net.Listener, and gives a place to store the timeout

go/weed/benchmark.go

@@ -2,9 +2,6 @@ package main
 
 import (
 	"bufio"
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/operation"
-	"github.com/chrislusf/weed-fs/go/util"
 	"fmt"
 	"io"
 	"math"
@@ -16,6 +13,10 @@ import (
 	"strings"
 	"sync"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/operation"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 type BenchmarkOptions struct {
@@ -30,11 +31,14 @@ type BenchmarkOptions struct {
 	sequentialRead   *bool
 	collection       *string
 	cpuprofile       *string
+	maxCpu           *int
 	vid2server       map[string]string //cache for vid locations
+
 }
 
 var (
-	b BenchmarkOptions
+	b           BenchmarkOptions
+	sharedBytes []byte
 )
 
 func init() {
@@ -50,33 +54,35 @@ func init() {
 	b.read = cmdBenchmark.Flag.Bool("read", true, "enable read")
 	b.sequentialRead = cmdBenchmark.Flag.Bool("readSequentially", false, "randomly read by ids from \"-list\" specified file")
 	b.collection = cmdBenchmark.Flag.String("collection", "benchmark", "write data to this collection")
-	b.cpuprofile = cmdBenchmark.Flag.String("cpuprofile", "", "write cpu profile to file")
+	b.cpuprofile = cmdBenchmark.Flag.String("cpuprofile", "", "cpu profile output file")
+	b.maxCpu = cmdBenchmark.Flag.Int("maxCpu", 0, "maximum number of CPUs. 0 means all available CPUs")
 	b.vid2server = make(map[string]string)
+	sharedBytes = make([]byte, 1024)
 }
 
 var cmdBenchmark = &Command{
 	UsageLine: "benchmark -server=localhost:9333 -c=10 -n=100000",
 	Short:     "benchmark on writing millions of files and read out",
 	Long: `benchmark on an empty weed file system.
-  
+
   Two tests during benchmark:
   1) write lots of small files to the system
   2) read the files out
-  
+
   The file content is mostly zero, but no compression is done.
-    
+
   You can choose to only benchmark read or write.
   During write, the list of uploaded file ids is stored in "-list" specified file.
   You can also use your own list of file ids to run read test.
-  
+
   Write speed and read speed will be collected.
   The numbers are used to get a sense of the system.
   Usually your network or the hard drive is the real bottleneck.
-  
+
   Another thing to watch is whether the volumes are evenly distributed
   to each volume server. Because the 7 more benchmark volumes are randomly distributed
   to servers with free slots, it's highly possible some servers have uneven amount of
-  benchmark volumes. To remedy this, you can use this to grow the benchmark volumes 
+  benchmark volumes. To remedy this, you can use this to grow the benchmark volumes
   before starting the benchmark command:
     http://localhost:9333/vol/grow?collection=benchmark&count=5
 
@@ -87,18 +93,17 @@ var cmdBenchmark = &Command{
 }
 
 var (
-	wait            sync.WaitGroup
+	wait       sync.WaitGroup
-	writeStats      *stats
+	writeStats *stats
-	readStats       *stats
+	readStats  *stats
-	serverLimitChan map[string]chan bool
 )
 
-func init() {
-	serverLimitChan = make(map[string]chan bool)
-}
-
 func runbenchmark(cmd *Command, args []string) bool {
 	fmt.Printf("This is Seaweed File System version %s %s %s\n", util.VERSION, runtime.GOOS, runtime.GOARCH)
+	if *b.maxCpu < 1 {
+		*b.maxCpu = runtime.NumCPU()
+	}
+	runtime.GOMAXPROCS(*b.maxCpu)
 	if *b.cpuprofile != "" {
 		f, err := os.Create(*b.cpuprofile)
 		if err != nil {
@@ -122,12 +127,12 @@ func runbenchmark(cmd *Command, args []string) bool {
 func bench_write() {
 	fileIdLineChan := make(chan string)
 	finishChan := make(chan bool)
-	writeStats = newStats()
+	writeStats = newStats(*b.concurrency)
 	idChan := make(chan int)
-	wait.Add(*b.concurrency)
 	go writeFileIds(*b.idListFile, fileIdLineChan, finishChan)
 	for i := 0; i < *b.concurrency; i++ {
-		go writeFiles(idChan, fileIdLineChan, writeStats)
+		wait.Add(1)
+		go writeFiles(idChan, fileIdLineChan, &writeStats.localStats[i])
 	}
 	writeStats.start = time.Now()
 	writeStats.total = *b.numberOfFiles
@@ -138,28 +143,30 @@ func bench_write() {
 	close(idChan)
 	wait.Wait()
 	writeStats.end = time.Now()
-	wait.Add(1)
+	wait.Add(2)
 	finishChan <- true
 	finishChan <- true
-	close(finishChan)
 	wait.Wait()
+	close(finishChan)
 	writeStats.printStats()
 }
 
 func bench_read() {
 	fileIdLineChan := make(chan string)
 	finishChan := make(chan bool)
-	readStats = newStats()
+	readStats = newStats(*b.concurrency)
-	wait.Add(*b.concurrency)
 	go readFileIds(*b.idListFile, fileIdLineChan)
 	readStats.start = time.Now()
 	readStats.total = *b.numberOfFiles
 	go readStats.checkProgress("Randomly Reading Benchmark", finishChan)
 	for i := 0; i < *b.concurrency; i++ {
-		go readFiles(fileIdLineChan, readStats)
+		wait.Add(1)
+		go readFiles(fileIdLineChan, &readStats.localStats[i])
 	}
 	wait.Wait()
+	wait.Add(1)
 	finishChan <- true
+	wait.Wait()
 	close(finishChan)
 	readStats.end = time.Now()
 	readStats.printStats()
@@ -170,126 +177,102 @@ type delayedFile struct {
 	fp        *operation.FilePart
 }
 
-func writeFiles(idChan chan int, fileIdLineChan chan string, s *stats) {
+func writeFiles(idChan chan int, fileIdLineChan chan string, s *stat) {
+	defer wait.Done()
 	delayedDeleteChan := make(chan *delayedFile, 100)
 	var waitForDeletions sync.WaitGroup
 	for i := 0; i < 7; i++ {
+		waitForDeletions.Add(1)
 		go func() {
-			waitForDeletions.Add(1)
+			defer waitForDeletions.Done()
 			for df := range delayedDeleteChan {
-				if df == nil {
-					break
-				}
 				if df.enterTime.After(time.Now()) {
 					time.Sleep(df.enterTime.Sub(time.Now()))
 				}
-				fp := df.fp
+				if e := util.Delete("http://" + df.fp.Server + "/" + df.fp.Fid); e == nil {
-				serverLimitChan[fp.Server] <- true
-				if e := util.Delete("http://" + fp.Server + "/" + fp.Fid); e == nil {
 					s.completed++
 				} else {
 					s.failed++
 				}
-				<-serverLimitChan[fp.Server]
 			}
-			waitForDeletions.Done()
 		}()
 	}
 
-	for {
+	for id := range idChan {
-		if id, ok := <-idChan; ok {
+		start := time.Now()
-			start := time.Now()
+		fileSize := int64(*b.fileSize + rand.Intn(64))
-			fileSize := int64(*b.fileSize + rand.Intn(64))
+		fp := &operation.FilePart{Reader: &FakeReader{id: uint64(id), size: fileSize}, FileSize: fileSize}
-			fp := &operation.FilePart{Reader: &FakeReader{id: uint64(id), size: fileSize}, FileSize: fileSize}
+		if assignResult, err := operation.Assign(*b.server, 1, "", *b.collection, ""); err == nil {
-			if assignResult, err := operation.Assign(*b.server, 1, "", *b.collection, ""); err == nil {
+			fp.Server, fp.Fid, fp.Collection = assignResult.PublicUrl, assignResult.Fid, *b.collection
-				fp.Server, fp.Fid, fp.Collection = assignResult.PublicUrl, assignResult.Fid, *b.collection
+			if _, err := fp.Upload(0, *b.server); err == nil {
-				if _, ok := serverLimitChan[fp.Server]; !ok {
+				if rand.Intn(100) < *b.deletePercentage {
-					serverLimitChan[fp.Server] = make(chan bool, 7)
+					s.total++
-				}
+					delayedDeleteChan <- &delayedFile{time.Now().Add(time.Second), fp}
-				serverLimitChan[fp.Server] <- true
-				if _, err := fp.Upload(0, *b.server); err == nil {
-					if rand.Intn(100) < *b.deletePercentage {
-						s.total++
-						delayedDeleteChan <- &delayedFile{time.Now().Add(time.Second), fp}
-					} else {
-						fileIdLineChan <- fp.Fid
-					}
-					s.completed++
-					s.transferred += fileSize
 				} else {
-					s.failed++
+					fileIdLineChan <- fp.Fid
-				}
-				writeStats.addSample(time.Now().Sub(start))
-				<-serverLimitChan[fp.Server]
-				if *cmdBenchmark.IsDebug {
-					fmt.Printf("writing %d file %s\n", id, fp.Fid)
 				}
+				s.completed++
+				s.transferred += fileSize
 			} else {
 				s.failed++
-				println("writing file error:", err.Error())
+				fmt.Printf("Failed to write with error:%v\n", err)
+			}
+			writeStats.addSample(time.Now().Sub(start))
+			if *cmdBenchmark.IsDebug {
+				fmt.Printf("writing %d file %s\n", id, fp.Fid)
 			}
 		} else {
-			break
+			s.failed++
+			println("writing file error:", err.Error())
 		}
 	}
 	close(delayedDeleteChan)
 	waitForDeletions.Wait()
-	wait.Done()
 }
 
-func readFiles(fileIdLineChan chan string, s *stats) {
+func readFiles(fileIdLineChan chan string, s *stat) {
-	serverLimitChan := make(map[string]chan bool)
+	defer wait.Done()
 	masterLimitChan := make(chan bool, 1)
-	for {
+	for fid := range fileIdLineChan {
-		if fid, ok := <-fileIdLineChan; ok {
+		if len(fid) == 0 {
-			if len(fid) == 0 {
+			continue
-				continue
+		}
-			}
+		if fid[0] == '#' {
-			if fid[0] == '#' {
+			continue
-				continue
+		}
-			}
+		if *cmdBenchmark.IsDebug {
-			if *cmdBenchmark.IsDebug {
+			fmt.Printf("reading file %s\n", fid)
-				fmt.Printf("reading file %s\n", fid)
+		}
-			}
+		parts := strings.SplitN(fid, ",", 2)
-			parts := strings.SplitN(fid, ",", 2)
+		vid := parts[0]
-			vid := parts[0]
+		start := time.Now()
-			start := time.Now()
+		if server, ok := b.vid2server[vid]; !ok {
-			if server, ok := b.vid2server[vid]; !ok {
+			masterLimitChan <- true
-				masterLimitChan <- true
+			if _, now_ok := b.vid2server[vid]; !now_ok {
-				if _, now_ok := b.vid2server[vid]; !now_ok {
+				if ret, err := operation.Lookup(*b.server, vid); err == nil {
-					if ret, err := operation.Lookup(*b.server, vid); err == nil {
+					if len(ret.Locations) > 0 {
-						if len(ret.Locations) > 0 {
+						server = ret.Locations[0].PublicUrl
-							server = ret.Locations[0].PublicUrl
+						b.vid2server[vid] = server
-							b.vid2server[vid] = server
-						}
 					}
 				}
-				<-masterLimitChan
 			}
-			if server, ok := b.vid2server[vid]; ok {
+			<-masterLimitChan
-				if _, ok := serverLimitChan[server]; !ok {
+		}
-					serverLimitChan[server] = make(chan bool, 7)
+		if server, ok := b.vid2server[vid]; ok {
-				}
+			url := "http://" + server + "/" + fid
-				serverLimitChan[server] <- true
+			if bytesRead, err := util.Get(url); err == nil {
-				url := "http://" + server + "/" + fid
+				s.completed++
-				if bytesRead, err := util.Get(url); err == nil {
+				s.transferred += int64(len(bytesRead))
-					s.completed++
+				readStats.addSample(time.Now().Sub(start))
-					s.transferred += int64(len(bytesRead))
-					readStats.addSample(time.Now().Sub(start))
-				} else {
-					s.failed++
-					println("!!!! Failed to read from ", url, " !!!!!")
-				}
-				<-serverLimitChan[server]
 			} else {
 				s.failed++
-				println("!!!! volume id ", vid, " location not found!!!!!")
+				fmt.Printf("Failed to read %s error:%v\n", url, err)
 			}
 		} else {
-			break
+			s.failed++
+			println("!!!! volume id ", vid, " location not found!!!!!")
 		}
 	}
-	wait.Done()
 }
 
 func writeFileIds(fileName string, fileIdLineChan chan string, finishChan chan bool) {
@@ -353,20 +336,28 @@ const (
 
 // An efficient statics collecting and rendering
 type stats struct {
-	data        []int
+	data       []int
-	overflow    []int
+	overflow   []int
+	localStats []stat
+	start      time.Time
+	end        time.Time
+	total      int
+}
+type stat struct {
 	completed   int
 	failed      int
 	total       int
 	transferred int64
-	start       time.Time
-	end         time.Time
 }
 
 var percentages = []int{50, 66, 75, 80, 90, 95, 98, 99, 100}
 
-func newStats() *stats {
+func newStats(n int) *stats {
-	return &stats{data: make([]int, benchResolution), overflow: make([]int, 0)}
+	return &stats{
+		data:       make([]int, benchResolution),
+		overflow:   make([]int, 0),
+		localStats: make([]stat, n),
+	}
 }
 
 func (s *stats) addSample(d time.Duration) {
@@ -387,28 +378,41 @@ func (s *stats) checkProgress(testName string, finishChan chan bool) {
 	for {
 		select {
 		case <-finishChan:
+			wait.Done()
 			return
 		case t := <-ticker:
-			completed, transferred, taken := s.completed-lastCompleted, s.transferred-lastTransferred, t.Sub(lastTime)
+			completed, transferred, taken, total := 0, int64(0), t.Sub(lastTime), s.total
+			for _, localStat := range s.localStats {
+				completed += localStat.completed
+				transferred += localStat.transferred
+				total += localStat.total
+			}
 			fmt.Printf("Completed %d of %d requests, %3.1f%% %3.1f/s %3.1fMB/s\n",
-				s.completed, s.total, float64(s.completed)*100/float64(s.total),
+				completed, total, float64(completed)*100/float64(total),
-				float64(completed)*float64(int64(time.Second))/float64(int64(taken)),
+				float64(completed-lastCompleted)*float64(int64(time.Second))/float64(int64(taken)),
-				float64(transferred)*float64(int64(time.Second))/float64(int64(taken))/float64(1024*1024),
+				float64(transferred-lastTransferred)*float64(int64(time.Second))/float64(int64(taken))/float64(1024*1024),
 			)
-			lastCompleted, lastTransferred, lastTime = s.completed, s.transferred, t
+			lastCompleted, lastTransferred, lastTime = completed, transferred, t
 		}
 	}
 }
 
 func (s *stats) printStats() {
+	completed, failed, transferred, total := 0, 0, int64(0), s.total
+	for _, localStat := range s.localStats {
+		completed += localStat.completed
+		failed += localStat.failed
+		transferred += localStat.transferred
+		total += localStat.total
+	}
 	timeTaken := float64(int64(s.end.Sub(s.start))) / 1000000000
 	fmt.Printf("\nConcurrency Level:      %d\n", *b.concurrency)
 	fmt.Printf("Time taken for tests:   %.3f seconds\n", timeTaken)
-	fmt.Printf("Complete requests:      %d\n", s.completed)
+	fmt.Printf("Complete requests:      %d\n", completed)
-	fmt.Printf("Failed requests:        %d\n", s.failed)
+	fmt.Printf("Failed requests:        %d\n", failed)
-	fmt.Printf("Total transferred:      %d bytes\n", s.transferred)
+	fmt.Printf("Total transferred:      %d bytes\n", transferred)
-	fmt.Printf("Requests per second:    %.2f [#/sec]\n", float64(s.completed)/timeTaken)
+	fmt.Printf("Requests per second:    %.2f [#/sec]\n", float64(completed)/timeTaken)
-	fmt.Printf("Transfer rate:          %.2f [Kbytes/sec]\n", float64(s.transferred)/1024/timeTaken)
+	fmt.Printf("Transfer rate:          %.2f [Kbytes/sec]\n", float64(transferred)/1024/timeTaken)
 	n, sum := 0, 0
 	min, max := 10000000, 0
 	for i := 0; i < len(s.data); i++ {
@@ -496,15 +500,32 @@ func (l *FakeReader) Read(p []byte) (n int, err error) {
 	} else {
 		n = len(p)
 	}
-	for i := 0; i < n-8; i += 8 {
+	if n >= 8 {
-		for s := uint(0); s < 8; s++ {
+		for i := 0; i < 8; i++ {
-			p[i] = byte(l.id >> (s * 8))
+			p[i] = byte(l.id >> uint(i*8))
 		}
 	}
 	l.size -= int64(n)
 	return
 }
 
+func (l *FakeReader) WriteTo(w io.Writer) (n int64, err error) {
+	size := int(l.size)
+	bufferSize := len(sharedBytes)
+	for size > 0 {
+		tempBuffer := sharedBytes
+		if size < bufferSize {
+			tempBuffer = sharedBytes[0:size]
+		}
+		count, e := w.Write(tempBuffer)
+		if e != nil {
+			return int64(size), e
+		}
+		size -= count
+	}
+	return l.size, nil
+}
+
 func Readln(r *bufio.Reader) ([]byte, error) {
 	var (
 		isPrefix bool  = true

go/weed/compact.go

@@ -12,7 +12,7 @@ func init() {
 
 var cmdCompact = &Command{
 	UsageLine: "compact -dir=/tmp -volumeId=234",
-	Short:     "run weed tool compact on volume file if corrupted",
+	Short:     "run weed tool compact on volume file",
 	Long: `Force an compaction to remove deleted files from volume files.
   The compacted .dat file is stored as .cpd file.
   The compacted .idx file is stored as .cpx file.

go/weed/download.go

@@ -1,14 +1,15 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/operation"
-	"github.com/chrislusf/weed-fs/go/util"
 	"fmt"
 	"io"
 	"io/ioutil"
 	"os"
 	"path"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/operation"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 var (

go/weed/export.go

@@ -3,8 +3,6 @@ package main
 import (
 	"archive/tar"
 	"bytes"
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/storage"
 	"fmt"
 	"os"
 	"path"
@@ -12,6 +10,9 @@ import (
 	"strings"
 	"text/template"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 func init() {
@@ -36,7 +37,7 @@ var cmdExport = &Command{
 var (
 	exportVolumePath = cmdExport.Flag.String("dir", "/tmp", "input data directory to store volume data files")
 	exportCollection = cmdExport.Flag.String("collection", "", "the volume collection name")
-	exportVolumeId   = cmdExport.Flag.Int("volumeId", -1, "a volume id. The volume should already exist in the dir. The volume index file should not exist.")
+	exportVolumeId   = cmdExport.Flag.Int("volumeId", -1, "a volume id. The volume .dat and .idx files should already exist in the dir.")
 	dest             = cmdExport.Flag.String("o", "", "output tar file name, must ends with .tar, or just a \"-\" for stdout")
 	format           = cmdExport.Flag.String("fileNameFormat", defaultFnFormat, "filename format, default to {{.Mime}}/{{.Id}}:{{.Name}}")
 	tarFh            *tar.Writer

go/weed/filer.go

@@ -1,13 +1,14 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/util"
-	"github.com/chrislusf/weed-fs/go/weed/weed_server"
 	"net/http"
 	"os"
 	"strconv"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/util"
+	"github.com/chrislusf/weed-fs/go/weed/weed_server"
 )
 
 var (
@@ -20,6 +21,7 @@ type FilerOptions struct {
 	collection              *string
 	defaultReplicaPlacement *string
 	dir                     *string
+	redirectOnRead          *bool
 }
 
 func init() {
@@ -28,7 +30,8 @@ func init() {
 	f.collection = cmdFiler.Flag.String("collection", "", "all data will be stored in this collection")
 	f.port = cmdFiler.Flag.Int("port", 8888, "filer server http listen port")
 	f.dir = cmdFiler.Flag.String("dir", os.TempDir(), "directory to store meta data")
-	f.defaultReplicaPlacement = cmdFiler.Flag.String("defaultReplicaPlacement", "000", "Default replication type if not specified.")
+	f.defaultReplicaPlacement = cmdFiler.Flag.String("defaultReplicaPlacement", "000", "default replication type if not specified")
+	f.redirectOnRead = cmdFiler.Flag.Bool("redirectOnRead", false, "whether proxy or redirect to volume server during file GET request")
 }
 
 var cmdFiler = &Command{
@@ -59,7 +62,9 @@ func runFiler(cmd *Command, args []string) bool {
 	}
 
 	r := http.NewServeMux()
-	_, nfs_err := weed_server.NewFilerServer(r, *f.port, *f.master, *f.dir, *f.collection)
+	_, nfs_err := weed_server.NewFilerServer(r, *f.port, *f.master, *f.dir, *f.collection,
+		*f.defaultReplicaPlacement, *f.redirectOnRead,
+	)
 	if nfs_err != nil {
 		glog.Fatalf(nfs_err.Error())
 	}

go/weed/fix.go

@@ -1,11 +1,12 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/storage"
 	"os"
 	"path"
 	"strconv"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 func init() {
@@ -16,7 +17,7 @@ func init() {
 var cmdFix = &Command{
 	UsageLine: "fix -dir=/tmp -volumeId=234",
 	Short:     "run weed tool fix on index file if corrupted",
-	Long: `Fix runs the WeedFS fix command to re-create the index .idx file.
+	Long: `Fix runs the SeeweedFS fix command to re-create the index .idx file.
 
   `,
 }

go/weed/master.go

@@ -1,16 +1,17 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/util"
-	"github.com/chrislusf/weed-fs/go/weed/weed_server"
-	"github.com/gorilla/mux"
 	"net/http"
 	"os"
 	"runtime"
 	"strconv"
 	"strings"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/util"
+	"github.com/chrislusf/weed-fs/go/weed/weed_server"
+	"github.com/gorilla/mux"
 )
 
 func init() {
@@ -29,6 +30,7 @@ var cmdMaster = &Command{
 var (
 	mport                   = cmdMaster.Flag.Int("port", 9333, "http listen port")
 	masterIp                = cmdMaster.Flag.String("ip", "", "master listening ip address, default to listen on all network interfaces")
+	masterBindIp            = cmdMaster.Flag.String("ip.bind", "0.0.0.0", "ip address to bind to")
 	mPublicIp               = cmdMaster.Flag.String("publicIp", "", "peer accessible <ip>|<server_name>")
 	metaFolder              = cmdMaster.Flag.String("mdir", os.TempDir(), "data directory to store meta data")
 	masterPeers             = cmdMaster.Flag.String("peers", "", "other master nodes in comma separated ip:port list")
@@ -61,7 +63,7 @@ func runMaster(cmd *Command, args []string) bool {
 		*volumeSizeLimitMB, *mpulse, *confFile, *defaultReplicaPlacement, *garbageThreshold, masterWhiteList,
 	)
 
-	listeningAddress := *masterIp + ":" + strconv.Itoa(*mport)
+	listeningAddress := *masterBindIp + ":" + strconv.Itoa(*mport)
 
 	glog.V(0).Infoln("Start Seaweed Master", util.VERSION, "at", listeningAddress)
 

go/weed/mount.go

@@ -1,7 +1,5 @@
 package main
 
-import ()
-
 type MountOptions struct {
 	filer *string
 	dir   *string

go/weed/mount_std.go

@@ -3,15 +3,16 @@
 package main
 
 import (
+	"fmt"
+	"os"
+	"runtime"
+
 	"bazil.org/fuse"
 	"bazil.org/fuse/fs"
 	"github.com/chrislusf/weed-fs/go/filer"
 	"github.com/chrislusf/weed-fs/go/glog"
 	"github.com/chrislusf/weed-fs/go/storage"
 	"github.com/chrislusf/weed-fs/go/util"
-	"fmt"
-	"os"
-	"runtime"
 )
 
 func runMount(cmd *Command, args []string) bool {

go/weed/server.go

@@ -1,10 +1,6 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/util"
-	"github.com/chrislusf/weed-fs/go/weed/weed_server"
-	"github.com/gorilla/mux"
 	"net/http"
 	"os"
 	"runtime"
@@ -13,6 +9,11 @@ import (
 	"strings"
 	"sync"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/util"
+	"github.com/chrislusf/weed-fs/go/weed/weed_server"
+	"github.com/gorilla/mux"
 )
 
 type ServerOptions struct {
@@ -31,17 +32,17 @@ func init() {
 var cmdServer = &Command{
 	UsageLine: "server -port=8080 -dir=/tmp -volume.max=5 -ip=server_name",
 	Short:     "start a server, including volume server, and automatically elect a master server",
-	Long: `start both a volume server to provide storage spaces 
+	Long: `start both a volume server to provide storage spaces
   and a master server to provide volume=>location mapping service and sequence number of file ids
-  
+
   This is provided as a convenient way to start both volume server and master server.
   The servers are exactly the same as starting them separately.
 
   So other volume servers can use this embedded master server also.
-  
+
   Optionally, one filer server can be started. Logically, filer servers should not be in a cluster.
   They run with meta data on disk, not shared. So each filer server is different.
-  
+
   `,
 }
 
@@ -72,12 +73,14 @@ var (
 )
 
 func init() {
-	serverOptions.cpuprofile = cmdServer.Flag.String("cpuprofile", "", "write cpu profile to file")
+	serverOptions.cpuprofile = cmdServer.Flag.String("cpuprofile", "", "cpu profile output file")
 	filerOptions.master = cmdServer.Flag.String("filer.master", "", "default to current master server")
 	filerOptions.collection = cmdServer.Flag.String("filer.collection", "", "all data will be stored in this collection")
 	filerOptions.port = cmdServer.Flag.Int("filer.port", 8888, "filer server http listen port")
 	filerOptions.dir = cmdServer.Flag.String("filer.dir", "", "directory to store meta data, default to a 'filer' sub directory of what -mdir is specified")
 	filerOptions.defaultReplicaPlacement = cmdServer.Flag.String("filer.defaultReplicaPlacement", "", "Default replication type if not specified during runtime.")
+	filerOptions.redirectOnRead = cmdServer.Flag.Bool("filer.redirectOnRead", false, "whether proxy or redirect to volume server during file GET request")
+
 }
 
 func runServer(cmd *Command, args []string) bool {
@@ -149,7 +152,9 @@ func runServer(cmd *Command, args []string) bool {
 	if *isStartingFiler {
 		go func() {
 			r := http.NewServeMux()
-			_, nfs_err := weed_server.NewFilerServer(r, *filerOptions.port, *filerOptions.master, *filerOptions.dir, *filerOptions.collection)
+			_, nfs_err := weed_server.NewFilerServer(r, *filerOptions.port, *filerOptions.master, *filerOptions.dir, *filerOptions.collection,
+				*filerOptions.defaultReplicaPlacement, *filerOptions.redirectOnRead,
+			)
 			if nfs_err != nil {
 				glog.Fatalf(nfs_err.Error())
 			}

go/weed/shell.go

@@ -2,9 +2,10 @@ package main
 
 import (
 	"bufio"
-	"github.com/chrislusf/weed-fs/go/glog"
 	"fmt"
 	"os"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 func init() {

go/weed/signal_handling_notsupported.go

@@ -2,7 +2,5 @@
 
 package main
 
-import ()
-
 func OnInterrupt(fn func()) {
 }

go/weed/upload.go

@@ -1,11 +1,12 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/operation"
 	"encoding/json"
 	"fmt"
 	"os"
 	"path/filepath"
+
+	"github.com/chrislusf/weed-fs/go/operation"
 )
 
 var (

go/weed/version.go

@@ -1,9 +1,10 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/util"
 	"fmt"
 	"runtime"
+
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 var cmdVersion = &Command{

go/weed/volume.go

@@ -1,15 +1,16 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/util"
-	"github.com/chrislusf/weed-fs/go/weed/weed_server"
 	"net/http"
 	"os"
 	"runtime"
 	"strconv"
 	"strings"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/util"
+	"github.com/chrislusf/weed-fs/go/weed/weed_server"
 )
 
 func init() {
@@ -30,7 +31,7 @@ var (
 	maxVolumeCounts       = cmdVolume.Flag.String("max", "7", "maximum numbers of volumes, count[,count]...")
 	ip                    = cmdVolume.Flag.String("ip", "", "ip or server name")
 	publicIp              = cmdVolume.Flag.String("publicIp", "", "Publicly accessible <ip|server_name>")
-	bindIp                = cmdVolume.Flag.String("ip.bind", "0.0.0.0", "ip address to bind to")
+	volumeBindIp          = cmdVolume.Flag.String("ip.bind", "0.0.0.0", "ip address to bind to")
 	masterNode            = cmdVolume.Flag.String("mserver", "localhost:9333", "master server location")
 	vpulse                = cmdVolume.Flag.Int("pulseSeconds", 5, "number of seconds between heartbeats, must be smaller than or equal to the master's setting")
 	vTimeout              = cmdVolume.Flag.Int("idleTimeout", 10, "connection idle seconds")
@@ -85,7 +86,7 @@ func runVolume(cmd *Command, args []string) bool {
 		*fixJpgOrientation,
 	)
 
-	listeningAddress := *bindIp + ":" + strconv.Itoa(*vport)
+	listeningAddress := *volumeBindIp + ":" + strconv.Itoa(*vport)
 
 	glog.V(0).Infoln("Start Seaweed volume server", util.VERSION, "at", listeningAddress)
 

go/weed/volume_test.go

@@ -1,10 +1,11 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
 	"net/http"
 	"testing"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 func TestXYZ(t *testing.T) {

go/weed/weed.go

@@ -1,7 +1,6 @@
 package main
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
 	"flag"
 	"fmt"
 	"io"
@@ -13,6 +12,8 @@ import (
 	"time"
 	"unicode"
 	"unicode/utf8"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 var IsDebug *bool

go/weed/weed_server/common.go

@@ -2,11 +2,6 @@ package weed_server
 
 import (
 	"bytes"
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/operation"
-	"github.com/chrislusf/weed-fs/go/stats"
-	"github.com/chrislusf/weed-fs/go/storage"
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"fmt"
 	"net"
@@ -14,6 +9,12 @@ import (
 	"path/filepath"
 	"strconv"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/operation"
+	"github.com/chrislusf/weed-fs/go/stats"
+	"github.com/chrislusf/weed-fs/go/storage"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 var serverStats *stats.ServerStats

go/weed/weed_server/filer_server.go

@@ -1,24 +1,31 @@
 package weed_server
 
 import (
-	"github.com/chrislusf/weed-fs/go/filer"
-	"github.com/chrislusf/weed-fs/go/glog"
 	"net/http"
 	"strconv"
+
+	"github.com/chrislusf/weed-fs/go/filer"
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 type FilerServer struct {
-	port       string
+	port               string
-	master     string
+	master             string
-	collection string
+	collection         string
-	filer      filer.Filer
+	defaultReplication string
+	redirectOnRead     bool
+	filer              filer.Filer
 }
 
-func NewFilerServer(r *http.ServeMux, port int, master string, dir string, collection string) (fs *FilerServer, err error) {
+func NewFilerServer(r *http.ServeMux, port int, master string, dir string, collection string,
+	replication string, redirectOnRead bool,
+) (fs *FilerServer, err error) {
 	fs = &FilerServer{
-		master:     master,
+		master:             master,
-		collection: collection,
+		collection:         collection,
-		port:       ":" + strconv.Itoa(port),
+		defaultReplication: replication,
+		redirectOnRead:     redirectOnRead,
+		port:               ":" + strconv.Itoa(port),
 	}
 
 	if fs.filer, err = filer.NewFilerEmbedded(master, dir); err != nil {

go/weed/weed_server/filer_server_handlers.go

@@ -1,12 +1,8 @@
 package weed_server
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/operation"
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
-	"github.com/syndtr/goleveldb/leveldb"
 	"io"
 	"io/ioutil"
 	"math/rand"
@@ -14,6 +10,11 @@ import (
 	"net/url"
 	"strconv"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/operation"
+	"github.com/chrislusf/weed-fs/go/util"
+	"github.com/syndtr/goleveldb/leveldb"
 )
 
 func (fs *FilerServer) filerHandler(w http.ResponseWriter, r *http.Request) {
@@ -80,7 +81,12 @@ func (fs *FilerServer) GetOrHeadHandler(w http.ResponseWriter, r *http.Request,
 		return
 	}
 	urlLocation := lookup.Locations[rand.Intn(len(lookup.Locations))].PublicUrl
-	u, _ := url.Parse("http://" + urlLocation + "/" + fileId)
+	urlString := "http://" + urlLocation + "/" + fileId
+	if fs.redirectOnRead {
+		http.Redirect(w, r, urlString, http.StatusFound)
+		return
+	}
+	u, _ := url.Parse(urlString)
 	request := &http.Request{
 		Method:        r.Method,
 		URL:           u,
@@ -109,7 +115,11 @@ func (fs *FilerServer) GetOrHeadHandler(w http.ResponseWriter, r *http.Request,
 
 func (fs *FilerServer) PostHandler(w http.ResponseWriter, r *http.Request) {
 	query := r.URL.Query()
-	assignResult, ae := operation.Assign(fs.master, 1, query.Get("replication"), fs.collection, query.Get("ttl"))
+	replication := query.Get("replication")
+	if replication == "" {
+		replication = fs.defaultReplication
+	}
+	assignResult, ae := operation.Assign(fs.master, 1, replication, fs.collection, query.Get("ttl"))
 	if ae != nil {
 		glog.V(0).Infoln("failing to assign a file id", ae.Error())
 		writeJsonError(w, r, ae)

go/weed/weed_server/filer_server_handlers_admin.go

@@ -1,8 +1,9 @@
 package weed_server
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
 	"net/http"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 /*

go/weed/weed_server/master_server.go

@@ -1,16 +1,17 @@
 package weed_server
 
 import (
+	"net/http"
+	"net/http/httputil"
+	"net/url"
+	"sync"
+
 	"github.com/chrislusf/weed-fs/go/glog"
 	"github.com/chrislusf/weed-fs/go/sequence"
 	"github.com/chrislusf/weed-fs/go/topology"
 	"github.com/chrislusf/weed-fs/go/util"
 	"github.com/goraft/raft"
 	"github.com/gorilla/mux"
-	"net/http"
-	"net/http/httputil"
-	"net/url"
-	"sync"
 )
 
 type MasterServer struct {

go/weed/weed_server/master_server_handlers.go

@@ -1,12 +1,13 @@
 package weed_server
 
 import (
-	"github.com/chrislusf/weed-fs/go/operation"
-	"github.com/chrislusf/weed-fs/go/stats"
-	"github.com/chrislusf/weed-fs/go/storage"
 	"net/http"
 	"strconv"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/operation"
+	"github.com/chrislusf/weed-fs/go/stats"
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 func (ms *MasterServer) lookupVolumeId(vids []string, collection string) (volumeLocations map[string]operation.LookupResult) {
@@ -78,7 +79,7 @@ func (ms *MasterServer) dirAssignHandler(w http.ResponseWriter, r *http.Request)
 		return
 	}
 
-	if !ms.Topo.HasWriableVolume(option) {
+	if !ms.Topo.HasWritableVolume(option) {
 		if ms.Topo.FreeSpace() <= 0 {
 			w.WriteHeader(http.StatusNotFound)
 			writeJsonQuiet(w, r, operation.AssignResult{Error: "No free volumes left!"})
@@ -86,7 +87,7 @@ func (ms *MasterServer) dirAssignHandler(w http.ResponseWriter, r *http.Request)
 		} else {
 			ms.vgLock.Lock()
 			defer ms.vgLock.Unlock()
-			if !ms.Topo.HasWriableVolume(option) {
+			if !ms.Topo.HasWritableVolume(option) {
 				if _, err = ms.vg.AutomaticGrowByType(option, ms.Topo); err != nil {
 					writeJsonQuiet(w, r, operation.AssignResult{Error: "Cannot grow volume group! " + err.Error()})
 					return

go/weed/weed_server/master_server_handlers_admin.go

@@ -1,18 +1,19 @@
 package weed_server
 
 import (
-	proto "code.google.com/p/goprotobuf/proto"
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/operation"
-	"github.com/chrislusf/weed-fs/go/storage"
-	"github.com/chrislusf/weed-fs/go/topology"
-	"github.com/chrislusf/weed-fs/go/util"
 	"encoding/json"
 	"errors"
 	"io/ioutil"
 	"net/http"
 	"strconv"
 	"strings"
+
+	proto "code.google.com/p/goprotobuf/proto"
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/operation"
+	"github.com/chrislusf/weed-fs/go/storage"
+	"github.com/chrislusf/weed-fs/go/topology"
+	"github.com/chrislusf/weed-fs/go/util"
 )
 
 func (ms *MasterServer) collectionDeleteHandler(w http.ResponseWriter, r *http.Request) {
@@ -143,7 +144,7 @@ func (ms *MasterServer) deleteFromMasterServerHandler(w http.ResponseWriter, r *
 	}
 }
 
-func (ms *MasterServer) hasWriableVolume(option *topology.VolumeGrowOption) bool {
+func (ms *MasterServer) HasWritableVolume(option *topology.VolumeGrowOption) bool {
 	vl := ms.Topo.GetVolumeLayout(option.Collection, option.ReplicaPlacement, option.Ttl)
 	return vl.GetActiveVolumeCount(option) > 0
 }

go/weed/weed_server/raft_server.go

@@ -2,19 +2,22 @@ package weed_server
 
 import (
 	"bytes"
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/topology"
 	"encoding/json"
 	"errors"
 	"fmt"
-	"github.com/goraft/raft"
-	"github.com/gorilla/mux"
 	"io/ioutil"
 	"math/rand"
 	"net/http"
 	"net/url"
+	"os"
+	"path"
 	"strings"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/topology"
+	"github.com/goraft/raft"
+	"github.com/gorilla/mux"
 )
 
 type RaftServer struct {
@@ -46,6 +49,13 @@ func NewRaftServer(r *mux.Router, peers []string, httpAddr string, dataDir strin
 	transporter.Transport.MaxIdleConnsPerHost = 1024
 	glog.V(1).Infof("Starting RaftServer with IP:%v:", httpAddr)
 
+	// Clear old cluster configurations if peers are set
+	if len(s.peers) > 0 {
+		os.RemoveAll(path.Join(s.dataDir, "conf"))
+		os.RemoveAll(path.Join(s.dataDir, "log"))
+		os.RemoveAll(path.Join(s.dataDir, "snapshot"))
+	}
+
 	s.raftServer, err = raft.NewServer(s.httpAddr, s.dataDir, transporter, nil, topo, "")
 	if err != nil {
 		glog.V(0).Infoln(err)
@@ -53,35 +63,30 @@ func NewRaftServer(r *mux.Router, peers []string, httpAddr string, dataDir strin
 	}
 	transporter.Install(s.raftServer, s)
 	s.raftServer.SetHeartbeatInterval(1 * time.Second)
-	s.raftServer.SetElectionTimeout(time.Duration(pulseSeconds) * 1150 * time.Millisecond)
+	s.raftServer.SetElectionTimeout(time.Duration(pulseSeconds) * 3450 * time.Millisecond)
 	s.raftServer.Start()
 
 	s.router.HandleFunc("/cluster/join", s.joinHandler).Methods("POST")
 	s.router.HandleFunc("/cluster/status", s.statusHandler).Methods("GET")
 
-	// Join to leader if specified.
 	if len(s.peers) > 0 {
-		if !s.raftServer.IsLogEmpty() {
+		// Join to leader if specified.
-			glog.V(0).Infoln("Starting cluster with existing logs.")
+		glog.V(0).Infoln("Joining cluster:", strings.Join(s.peers, ","))
-		} else {
+		time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
-			glog.V(0).Infoln("Joining cluster:", strings.Join(s.peers, ","))
+		firstJoinError := s.Join(s.peers)
-			time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
+		if firstJoinError != nil {
-			firstJoinError := s.Join(s.peers)
+			glog.V(0).Infoln("No existing server found. Starting as leader in the new cluster.")
-			if firstJoinError != nil {
+			_, err := s.raftServer.Do(&raft.DefaultJoinCommand{
-				glog.V(0).Infoln("No existing server found. Starting as leader in the new cluster.")
+				Name:             s.raftServer.Name(),
-				_, err := s.raftServer.Do(&raft.DefaultJoinCommand{
+				ConnectionString: "http://" + s.httpAddr,
-					Name:             s.raftServer.Name(),
+			})
-					ConnectionString: "http://" + s.httpAddr,
+			if err != nil {
-				})
+				glog.V(0).Infoln(err)
-				if err != nil {
+				return nil
-					glog.V(0).Infoln(err)
-					return nil
-				}
 			}
 		}
-
-		// Initialize the server by joining itself.
 	} else if s.raftServer.IsLogEmpty() {
+		// Initialize the server by joining itself.
 		glog.V(0).Infoln("Initializing new cluster")
 
 		_, err := s.raftServer.Do(&raft.DefaultJoinCommand{
@@ -95,7 +100,7 @@ func NewRaftServer(r *mux.Router, peers []string, httpAddr string, dataDir strin
 		}
 
 	} else {
-		glog.V(0).Infoln("Recovered from log")
+		glog.V(0).Infoln("Old conf,log,snapshot should have been removed.")
 	}
 
 	return s

go/weed/weed_server/raft_server_handlers.go

@@ -1,13 +1,14 @@
 package weed_server
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/operation"
 	"encoding/json"
-	"github.com/goraft/raft"
 	"io/ioutil"
 	"net/http"
 	"strings"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/operation"
+	"github.com/goraft/raft"
 )
 
 // Handles incoming RAFT joins.

go/weed/weed_server/volume_server.go

@@ -1,12 +1,13 @@
 package weed_server
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/storage"
 	"math/rand"
 	"net/http"
 	"strconv"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/storage"
 )
 
 type VolumeServer struct {

go/weed/weed_server/volume_server_handlers.go

@@ -1,12 +1,6 @@
 package weed_server
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
-	"github.com/chrislusf/weed-fs/go/images"
-	"github.com/chrislusf/weed-fs/go/operation"
-	"github.com/chrislusf/weed-fs/go/stats"
-	"github.com/chrislusf/weed-fs/go/storage"
-	"github.com/chrislusf/weed-fs/go/topology"
 	"io"
 	"mime"
 	"mime/multipart"
@@ -14,6 +8,13 @@ import (
 	"strconv"
 	"strings"
 	"time"
+
+	"github.com/chrislusf/weed-fs/go/glog"
+	"github.com/chrislusf/weed-fs/go/images"
+	"github.com/chrislusf/weed-fs/go/operation"
+	"github.com/chrislusf/weed-fs/go/stats"
+	"github.com/chrislusf/weed-fs/go/storage"
+	"github.com/chrislusf/weed-fs/go/topology"
 )
 
 var fileNameEscaper = strings.NewReplacer("\\", "\\\\", "\"", "\\\"")

go/weed/weed_server/volume_server_handlers_admin.go

@@ -1,11 +1,12 @@
 package weed_server
 
 import (
+	"net/http"
+	"path/filepath"
+
 	"github.com/chrislusf/weed-fs/go/glog"
 	"github.com/chrislusf/weed-fs/go/stats"
 	"github.com/chrislusf/weed-fs/go/util"
-	"net/http"
-	"path/filepath"
 )
 
 func (vs *VolumeServer) statusHandler(w http.ResponseWriter, r *http.Request) {

go/weed/weed_server/volume_server_handlers_vacuum.go

@@ -1,8 +1,9 @@
 package weed_server
 
 import (
-	"github.com/chrislusf/weed-fs/go/glog"
 	"net/http"
+
+	"github.com/chrislusf/weed-fs/go/glog"
 )
 
 func (vs *VolumeServer) vacuumVolumeCheckHandler(w http.ResponseWriter, r *http.Request) {