Chris Lu
1cbd53c01c
WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete
2018-11-25 13:43:26 -08:00
Chris Lu
86dd933596
go fmt
2018-11-23 00:26:15 -08:00
Chris Lu
6344469a9b
set fuse subtype
2018-11-22 23:04:23 -08:00
Chris Lu
60d3681371
default images.fix.orientation to false
2018-11-21 16:50:42 -08:00
Chris Lu
c9b3ef58de
options to control filer.export verbosity
2018-11-21 15:15:40 -08:00
Chris Lu
2f2e417b68
detailed export
2018-11-21 15:12:47 -08:00
Chris Lu
2e6a3c7b16
reduce memory usage
2018-11-14 22:48:54 -08:00
Chris Lu
a62adf62ca
a bit faster
2018-11-12 11:19:13 -08:00
Chris Lu
affe708961
a bit faster
2018-11-11 00:43:30 -08:00
Chris Lu
a09eaf8756
adjusting, seems there are some perf benefit
2018-11-11 00:08:22 -08:00
Chris Lu
cbd94b18a5
improve "ls -al" performance for large directory
2018-11-08 07:37:34 -08:00
Chris Lu
85d1d99459
default "weed export to current enabled store"
2018-11-04 12:59:29 -08:00
Chris Lu
6c45787a37
selectively export files
2018-11-04 12:07:33 -08:00
Chris Lu
ff4189ab25
go fmt
2018-11-04 11:59:08 -08:00
Chris Lu
8dfac6a4cf
working b2 sink
2018-11-04 11:58:59 -08:00
Chris Lu
a64613172d
bootstrap filer replication with weed filer.export -targetStore=notification
2018-11-03 12:43:45 -07:00
Chris Lu
3674ad9f8e
go fmt
2018-11-01 01:12:21 -07:00
Chris Lu
db584ff7f8
separate into notification.toml, add gcp pub/sub message queue
2018-11-01 01:11:09 -07:00
Chris Lu
4c97ff3717
support AWS SQS as file change notification message queue
2018-10-31 01:11:19 -07:00
Chris Lu
fe31f389b0
add configuration instructions
2018-10-30 02:29:11 -07:00
Chris Lu
c58b73ad9b
adding BackBlaze, Azure
2018-10-23 23:59:40 -07:00
Chris Lu
fda771c83f
migrate volume sync status to grpc API on volume server
2018-10-15 01:19:15 -07:00
Chris Lu
b1daede91b
move volume vacuum to gRpc
2018-10-14 23:12:43 -07:00
Chris Lu
556382ff5f
adding basic grpc to volume server
2018-10-11 01:16:33 -07:00
Chris Lu
08266b7256
go fmt
2018-10-11 00:08:13 -07:00
Chris Lu
1d7e01b754
reformat
2018-10-11 00:05:54 -07:00
Chris Lu
da6154b29c
refactor volume server to startVolumeServer()
2018-10-11 00:04:31 -07:00
Chris Lu
60d2f1557d
refactor func name
2018-10-10 23:19:54 -07:00
Chris Lu
9f4c2f87d3
adding filer replication sink to Azure
2018-10-09 01:35:48 -07:00
Chris Lu
96a23591e7
remove comments
2018-10-07 22:21:00 -07:00
Chris Lu
a5b5887efc
add filer replication to aws s3
2018-10-06 17:10:15 -07:00
Chris Lu
84d4a2c8a0
add filer replication to google cloud storage
2018-10-06 13:08:38 -07:00
Chris Lu
14858f6224
add logging
2018-10-06 13:01:38 -07:00
Chris Lu
56c5c7b1b6
add google cloud storage
2018-10-04 01:14:44 -07:00
Chris Lu
e8ef501f02
add s3 replication sink
2018-10-03 23:36:52 -07:00
Chris Lu
98384b62fd
adjust description
2018-09-28 01:58:34 -07:00
Chris Lu
12ba8b9799
generate useable filer.toml
2018-09-23 09:26:25 -07:00
Chris Lu
b16d47030e
save and use last processed kafka offsets
2018-09-23 01:34:40 -07:00
Chris Lu
9fe24991d5
refactoring
2018-09-23 00:40:36 -07:00
Chris Lu
7d6b2a4740
add ETag to upload results and chunks
2018-09-22 22:12:21 -07:00
Chris Lu
420f068376
add more instructions
2018-09-22 22:12:06 -07:00
Chris Lu
01ceace18e
adjust sink options
2018-09-22 00:53:52 -07:00
Chris Lu
402ab598b6
go fmt
2018-09-22 00:12:10 -07:00
Chris Lu
7bb62b9f21
delay file creation without chunks from dir.Create to file.Flush
2018-09-22 00:11:46 -07:00
Chris Lu
db69ce89f0
go fmt
2018-09-21 01:56:43 -07:00
Chris Lu
a6cfaba018
able to sync the changes
2018-09-21 01:54:29 -07:00
Chris Lu
a8cd7df44a
let filer choose default replication
2018-09-20 09:00:15 -07:00
Chris Lu
d5149e592e
add logging on kafka reading
2018-09-19 22:31:45 -07:00
Chris Lu
25fb6f9a46
fix compilation
2018-09-17 02:23:21 -07:00
Chris Lu
779641e9d4
adjust replicated entry name
2018-09-17 01:37:24 -07:00
Chris Lu
b7a375f5aa
set large limit for "weed filer.export" directory listing
2018-09-17 00:42:36 -07:00
Chris Lu
4e1f1a0b92
set large limit for "weed mount" directory listing
2018-09-17 00:41:33 -07:00
Chris Lu
788acdf527
add WIP filer.replicate
2018-09-17 00:27:56 -07:00
Chris Lu
7ad5a90127
go fmt
2018-09-03 16:47:00 -07:00
Chris Lu
6e3f4d1079
add working filer.export command
2018-08-23 00:02:04 -07:00
Chris Lu
98b8f8649d
adding filer.export prototype
2018-08-22 23:54:22 -07:00
Chris Lu
ae4e654d2a
add example configuration for default redis create-cluster setup
2018-08-22 18:23:19 -07:00
Chris Lu
748870b03f
add "weed scaffold" command
2018-08-19 15:36:30 -07:00
Chris Lu
f827ada811
merge notification config with filer.toml
2018-08-19 15:17:55 -07:00
Chris Lu
f036ef8a3c
add filer notification
2018-08-13 01:20:49 -07:00
Chris Lu
75d63db60d
randomize raft server startup
...
also some go fmt
2018-08-12 14:27:14 -07:00
Chris Lu
eca4b928d2
ensure master server count is odd
2018-08-12 14:25:31 -07:00
Chris Lu
2e427c66ac
small refactoring
2018-08-10 23:47:17 -07:00
Chris Lu
44d8e35988
add listener for volumd id location for benchmark tool
2018-07-31 19:12:36 -07:00
Chris Lu
d09606c65b
refactoring
2018-07-29 02:25:24 -07:00
Chris Lu
4f317c7e3d
working version3
2018-07-24 01:36:04 -07:00
Chris Lu
7f32eb1e25
s3 add https support
2018-07-22 21:28:54 -07:00
Chris Lu
056871242b
remove println
2018-07-22 01:17:49 -07:00
Chris Lu
0655151b8c
go fmt
2018-07-22 01:15:11 -07:00
Chris Lu
6319d84f42
s3 API add ListObjectsV1
2018-07-22 01:14:36 -07:00
Chris Lu
922c614bde
fix compilation problem
2018-07-21 20:42:14 -07:00
Chris Lu
16c66e097b
adjust default buckets folder
2018-07-21 20:29:33 -07:00
Chris Lu
7e2031b18f
go fmt
2018-07-21 17:39:10 -07:00
Chris Lu
c7228fefa7
add bucket creation and deletion
...
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
2018-07-19 01:21:44 -07:00
Chris Lu
834a25f084
add list all my buckets
2018-07-18 02:37:09 -07:00
Chris Lu
7abfab8e77
add feature to mount a specific filer path to local directory
2018-07-14 21:09:21 -07:00
Chris Lu
3edfe1d28f
extend export command to show tombstone + change output format to CSV
...
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
2018-07-14 20:51:17 -07:00
Chris Lu
6d58916cfd
fix compilation error
2018-07-14 13:39:57 -07:00
Chris Lu
842dab07b4
weed mount can request to prioritize to write to a data center
2018-07-14 13:36:28 -07:00
Chris Lu
1e1c368ae8
adjust filer help message
2018-07-10 00:22:11 -07:00
Chris Lu
50d351944f
filer add data center option
2018-07-09 02:22:48 -07:00
Chris Lu
d4d7ced922
refactoring: add type for needle id, offset
...
later the type size can possibly be adjusted
2018-07-08 02:28:04 -07:00
Chris Lu
922032b9bb
add filer sub dir list size limit to "weed server"
2018-07-08 02:11:36 -07:00
Chris Lu
702ff48cda
add dir list limit option
2018-07-07 02:18:47 -07:00
Chris Lu
2ad45ca04f
refactor
2018-07-07 02:03:25 -07:00
Chris Lu
77fc8c5914
keep alive for gRpc calls
2018-07-03 19:07:55 -07:00
Chris Lu
5bd72696ac
weed mount add ttl option
2018-06-11 23:13:33 -07:00
Chris Lu
98110c1697
add collection, replication, ttl to each file
2018-06-10 16:57:32 -07:00
Chris Lu
299312c805
use separate filer grpc port
2018-06-05 23:37:41 -07:00
Chris Lu
43e3f5724c
use fixed list of masters in both filer and volume servers
2018-06-01 00:39:39 -07:00
Chris Lu
dc13e10637
fix copy error on 0 size files
2018-05-30 22:28:14 -07:00
Chris Lu
44acf4b756
fix file name
...
f.Name() is not the base file name
2018-05-30 20:48:13 -07:00
Chris Lu
0301504184
add mime, use simple insert and update filer store API
...
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
2018-05-30 20:24:57 -07:00
Chris Lu
26e7cd8c75
assign a different volume on large file copying
2018-05-30 01:05:26 -07:00
Chris Lu
2fe0d479f1
filer.copy supports chunking
2018-05-30 00:54:56 -07:00
Chris Lu
2d7dea1a09
a little refactoring
2018-05-29 23:52:27 -07:00
Chris Lu
8a48baa056
add single chunk file copying to new filer
2018-05-29 23:46:45 -07:00
Chris Lu
4e3ea49cff
properly working local write buffer
2018-05-29 01:21:21 -07:00
Chris Lu
5c4480ec6c
add mountOptions.chunkSizeLimitMB, remove cmdMount.IsDebug
2018-05-28 13:42:25 -07:00
Chris Lu
8ab7dd9d08
weed mount add options for collection and replication
2018-05-28 13:24:48 -07:00
Chris Lu
458ada173e
go fmt
2018-05-27 11:52:26 -07:00
Chris Lu
9e77563c99
add leveldb store
...
1. switch to viper for filer store configuration
2. simplify FindEntry() return values, removing “found”
3. add leveldb store
2018-05-26 03:49:46 -07:00
Chris Lu
9dd228747c
filer copy added uid/gid
2018-05-21 01:25:30 -07:00
Chris Lu
c5cf9bd290
properly working filer
2018-05-13 23:56:16 -07:00
Chris Lu
9f345da20f
mv filer proto to filer_pb
2018-05-09 23:18:02 -07:00
Chris Lu
942c2cbd7b
mv pb to master_pb
2018-05-09 23:11:54 -07:00
Chris Lu
43a69d20bf
change filer API to gRPC
2018-05-08 01:59:43 -07:00
Chris Lu
4936d6c342
adjusting for FUSE
2018-05-07 00:37:47 -07:00
Chris Lu
456738ba64
refactoring fuse
2018-05-05 22:47:16 -07:00
Chris Lu
fffb14bc87
better support FUSE Lookup()
2018-05-05 02:01:50 -07:00
Chris Lu
4b9b9547d4
unused option
2018-02-27 13:57:15 -08:00
Chris Lu
61ff330e99
adjust instructions
2018-01-15 20:33:12 -08:00
Chris Lu
ef6224e45d
filer default to auto chunking file size to 32MB
2017-09-15 08:24:30 -07:00
Chris Lu
58344980e4
preallocate disk space during compaction also, add cleanup for failed compaction
2017-08-29 23:59:53 -07:00
Chris Lu
dbee816205
fix typo
...
fix https://github.com/chrislusf/seaweedfs/issues/549
2017-08-10 10:26:39 -07:00
Chris Lu
e0dfa7a069
ensure 30GB or less
...
avoid https://github.com/chrislusf/seaweedfs/issues/535
2017-07-16 21:40:55 -07:00
Chris Lu
1fbb8723dc
add cpu and mem pprof
2017-06-22 01:33:58 -07:00
Chris Lu
5047bdb4a2
skip bytes cache
2017-06-03 11:44:24 -07:00
Chris Lu
478fe0ecf2
filer add readonly public port
2017-05-27 20:14:22 -07:00
Chris Lu
7ecc0f4b11
rename variables
2017-05-27 18:11:18 -07:00
Chris Lu
82c3ccc8dd
add btree for volume index
2017-05-26 22:51:25 -07:00
Chris Lu
4ce6586710
use local random generator to avoid global lock
2017-05-19 23:33:36 -07:00
sparklxb
c46e91d229
complement weed mount: add read and delete
2017-03-17 15:02:27 +08:00
sparklxb
3a67b6ddc1
complement weed mount
2017-03-10 14:46:00 +08:00
sparklxb
7f85e15990
complement weed mount
2017-03-10 10:28:19 +08:00
sparklxb
813a4f7456
Fixes #266
2017-03-06 13:04:52 +08:00
Chris Lu
41f686ab9d
remove weeds.conf
2017-02-12 21:58:44 -08:00
Chris Lu
d953ed442c
auto detect volume server ip address
2017-01-12 13:42:53 -08:00
Chris Lu
1be354bdce
default IP to "localhost"
...
so that the master can know the ip address.
2017-01-10 23:57:59 -08:00
Chris Lu
e767c3ea4f
disable master connection timeout
...
temporarily disable master connection timeout due to heartbeat
connection timeout
2017-01-10 01:30:00 -08:00
Chris Lu
e46c3415f7
gRpc for master~volume heartbeat
2017-01-10 01:01:12 -08:00
sparklxb
66e7013dfe
suport uploading files to specific dataCenter
2017-01-09 23:34:47 +08:00
Chris Lu
53cf1b4900
change the basic unit to MB
2017-01-08 14:57:32 -08:00
Chris Lu
ed44f12f6d
support Fallocate on linux
2017-01-08 11:01:46 -08:00
Chris Lu
13e7069eb9
keep track of total data file size
2017-01-06 10:22:20 -08:00
lixianbin
d96d0a87cf
fix bug: upload big .gz file more than maxMB
2017-01-04 11:23:40 +08:00
Chris Lu
dcaf1796fe
add option to enable caching
2016-11-16 07:09:57 -08:00
Chris Lu
1bc041b46d
add a new way to manually compact corrupted volume
...
fix https://github.com/chrislusf/seaweedfs/issues/371
2016-09-22 20:31:20 -07:00
霍晓栋
1c2affea8e
Merge remote-tracking branch 'upstream/master'
2016-09-08 10:34:24 +08:00
Chris Lu
c4b7966dbe
minor help message change
2016-09-07 18:21:05 -07:00
霍晓栋
b0035747e3
add filer support
2016-08-31 11:32:30 +08:00
Mike Tolman
01d3f69c52
Adding AutoChunk/MaxMB Support to Filer API
...
This is related to the following issue I added to chrislusf/seaweedfs:
https://github.com/chrislusf/seaweedfs/issues/342
2016-08-05 16:01:30 -06:00
Mike Tolman
a89a3c86d0
Revert "Add AutoChunking to the Filer API, so that you can upload really large files through the filer API."
...
This reverts commit 09059bfdcc
.
2016-08-05 15:47:46 -06:00
Mike Tolman
ce99bb927d
Revert "Adding HTTP verb whitelisting options."
...
This reverts commit 34837afc7a
.
2016-08-05 15:45:48 -06:00
Mike Tolman
34837afc7a
Adding HTTP verb whitelisting options.
2016-08-05 15:23:43 -06:00
Mike Tolman
09059bfdcc
Add AutoChunking to the Filer API, so that you can upload really large files through the filer API.
2016-08-05 15:19:10 -06:00
Chris Lu
185a916f5e
adjusting command options
2016-07-21 15:00:07 -07:00
Chris Lu
cdae9fc680
add "weed copy" command to copy files to filer
2016-07-20 23:45:55 -07:00
Chris Lu
d3ae3a8e14
collect proof when stopping
2016-07-05 10:23:14 -07:00
Chris Lu
b38ff39e8b
add master profiling option
2016-07-03 15:39:57 -07:00
霍晓栋
1f63094542
refactor Volume Assign function
2016-06-26 10:50:18 +08:00
Chris Lu
5ce6bbf076
directory structure change to work with glide
...
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
2016-06-02 18:09:14 -07:00