Chris Lu
450222dd64
add remote to filer.Entry and filer_pb entry, add RemoteConf
2021-07-19 02:47:27 -07:00
Chris Lu
8dc5def435
proto: add remote
2021-07-19 01:12:31 -07:00
Chris Lu
b194f91f47
add version to filer configuration response
2021-07-12 01:23:20 -07:00
lyg
1a4db87e19
fix s3 metadata error with multipart upload
2021-07-02 11:00:42 +08:00
Chris Lu
215b169562
mount: recursively rename locally
2021-07-01 01:19:31 -07:00
Chris Lu
ab606dec2a
filer: add path-specific option to enforce readonly
2021-06-04 01:03:41 -07:00
Chris Lu
c5de97ddbe
keep alive for streaming connections
...
fix https://github.com/chrislusf/seaweedfs/issues/2096
related to https://github.com/chrislusf/seaweedfs/issues/1277
2021-05-31 01:26:24 -07:00
Chris Lu
1456616a77
recreate grpc connections if too many errors
...
address https://github.com/chrislusf/seaweedfs/issues/2098
2021-05-30 00:07:43 -07:00
Chris Lu
3d3fa43542
filer: re-create grpc connections if having transport error
...
fix https://github.com/chrislusf/seaweedfs/issues/2070
2021-05-22 17:46:53 -07:00
Chris Lu
dc1309f084
FUSE mount: support multiple filers
...
fix https://github.com/chrislusf/seaweedfs/issues/2015
fix https://github.com/chrislusf/seaweedfs/issues/1531
2021-05-21 01:28:00 -07:00
Chris Lu
f0ad172e80
shell: show which server holds the lock
...
fix https://github.com/chrislusf/seaweedfs/issues/1983
2021-04-22 23:56:35 -07:00
Chris Lu
b465095db1
shell: add volume.check.disk to fix inconsistency for replicated volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1923
2021-03-22 00:03:16 -07:00
Chris Lu
b1a86cf808
s3: copy object to itself
...
fix https://github.com/chrislusf/seaweedfs/issues/1922
2021-03-19 01:31:49 -07:00
Chris Lu
35d939a1d2
go fmt
2021-03-14 13:21:02 -07:00
wuh-fnst
4c1d945e46
make List correctly judge whether it is the last file
2021-03-12 11:22:26 +08:00
Chris Lu
828f6e9f4d
volume: auto add missing vif files
...
fix https://github.com/chrislusf/seaweedfs/issues/1878
2021-03-09 12:09:32 -08:00
Chris Lu
9d402ebe9f
refactoring
2021-03-06 14:26:27 -08:00
Chris Lu
bd727b7b53
go fmt
2021-03-02 20:59:56 -08:00
Chris Lu
e982b06bcd
support IPv6
2021-03-02 20:59:39 -08:00
Chris Lu
f465d63b5d
grpc should fail when heart beating to master
...
fix https://github.com/chrislusf/seaweedfs/issues/1820
2021-02-19 03:37:29 -08:00
Chris Lu
f8446b42ab
this can compile now!!!
2021-02-16 02:47:02 -08:00
Chris Lu
487e435679
adjust http max idle connections per host
...
related to https://github.com/chrislusf/seaweedfs/issues/1802
2021-02-12 03:47:15 -08:00
Chris Lu
770393a48c
volume: add capability to change disk type when moving a volume
2021-02-09 23:58:08 -08:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
Chris Lu
15c60cbb26
close the grpc connection after 10 hours
...
related to https://github.com/chrislusf/seaweedfs/issues/1782
2021-02-07 03:50:01 -08:00
Chris Lu
990fa69bfe
add back AdjustedUrl() related code
2021-01-28 14:36:29 -08:00
Chris Lu
00707ec00f
mount: outsideContainerClusterMode proxy through filer
...
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
2021-01-24 19:01:58 -08:00
Chris Lu
6ca10725b8
Revert "mount: when outside cluster network, use filer as proxy to access volume servers"
...
This reverts commit 096e088d7b
.
2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b
mount: when outside cluster network, use filer as proxy to access volume servers
2021-01-24 01:41:38 -08:00
Chris Lu
80b8692688
filer.sync: replicate outside of either cluster, only need to see filers
2021-01-24 00:01:44 -08:00
Chris Lu
2e8dba571b
adjust volume server UI
2020-12-14 00:51:57 -08:00
Chris Lu
94525aa0fd
allocate volume by disk type
2020-12-13 23:08:21 -08:00
Chris Lu
0d2ec832e2
rename from volumeType to diskType
2020-12-13 11:59:32 -08:00
Chris Lu
d156c74ec0
volume server set volume type and heartbeat to the master
2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
316d1b4e69
refactor APIs
2020-12-12 12:42:53 -08:00
Chris Lu
97c942b5ee
paginate through large directories
2020-12-09 23:23:38 -08:00
Chris Lu
9acda432fe
fix import cycle
2020-12-06 20:12:52 -08:00
Chris Lu
ae5eb85a06
refactoring
2020-12-06 20:05:06 -08:00
Chris Lu
f4abd01adf
filer: cache small file to filer store
2020-11-30 04:34:04 -08:00
Chris Lu
965413c21b
shell: add volume.vacuum command
2020-11-28 23:18:02 -08:00
Chris Lu
dc304342b2
fs.configure: configurable volume growth
2020-11-17 01:00:02 -08:00
Chris Lu
5f19e81dab
filer confi: support hierachical configuration
2020-11-16 16:50:12 -08:00
Chris Lu
ee2fa14dbe
filer conf: delete location specific configuration
2020-11-15 20:15:47 -08:00
Chris Lu
0ea5c087ce
go fmt
2020-11-15 16:59:28 -08:00
Chris Lu
95c0de285d
refactoring
2020-11-15 16:58:48 -08:00
Chris Lu
0a406f652e
load filer conf and match by prefix
2020-11-15 00:26:05 -08:00
Chris Lu
68043cfcac
add reference implementation to detect create/update/delete/rename events
2020-11-14 21:21:58 -08:00
Chris Lu
a4dfa00030
adjust filer conf
2020-11-12 00:39:59 -08:00
Chris Lu
2251a8e472
minor
2020-11-12 00:38:58 -08:00
Chris Lu
5a16f17e47
remove unused message type
2020-11-12 00:38:23 -08:00
Chris Lu
c32b6f4d9d
prepare for path based configuration
2020-11-11 23:46:45 -08:00
Konstantin Lebedev
1eec5c8d5d
gen pb
2020-11-12 04:10:06 +05:00
Konstantin Lebedev
dc26012a3b
initial
2020-11-11 15:03:47 +05:00
Konstantin Lebedev
918546bbdb
add WaitForReady to grpc DialOption
2020-10-29 08:29:04 +05:00
Chris Lu
f375b93aef
renaming
2020-10-25 15:32:43 -07:00
Chris Lu
6c92fa349d
filer: support default rack
...
fix https://github.com/chrislusf/seaweedfs/issues/1546
2020-10-20 17:41:39 -07:00
Chris Lu
ace0ea3d28
s3: avoid duplicated bucket
2020-10-15 10:52:20 -07:00
Chris Lu
723ae11db4
refactoring in order to adjust volume server url later
2020-10-11 20:15:10 -07:00
Chris Lu
da4edf3651
master: check peers for existing leader before starting a leader election
...
fix https://github.com/chrislusf/seaweedfs/issues/1509
2020-10-07 01:25:39 -07:00
Chris Lu
55cb68c09c
filer: replication follows master config if not specified
2020-09-30 09:15:55 -07:00
Chris Lu
1012df7bb5
switch hardlink id from int64 to bytes
2020-09-24 11:11:42 -07:00
Chris Lu
5e239afdfc
hardlink works now
2020-09-24 03:06:48 -07:00
Chris Lu
cb427d48fa
filer report metrics configuration
2020-09-17 06:46:51 -07:00
Chris Lu
c9202c4b3d
add storage backend to GetMasterConfigurationResponse
2020-09-16 01:12:22 -07:00
Chris Lu
200fe5c83e
go fmt
2020-09-13 21:26:30 -07:00
Chris Lu
5d6753fb98
shell: add volumeServer.leave command
2020-09-13 21:25:51 -07:00
Chris Lu
daf0a449f7
properly cancel context for streaming grpc
2020-09-09 12:07:15 -07:00
Chris Lu
387ab6796f
filer: cross cluster synchronization
2020-09-09 11:21:23 -07:00
Chris Lu
bba90ff3c8
read filer signature
2020-09-05 22:52:15 -07:00
Chris Lu
a41588279a
change log level 5 to 4
2020-08-30 20:12:04 -07:00
Chris Lu
ca658a97c5
add signatures to messages to avoid double processing
2020-08-28 23:48:48 -07:00
James Hartig
3ccfa4c6ad
Added VolumeMarkWritable and VolumeStatus grpc methods
...
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
2020-08-19 11:42:56 -04:00
Chris Lu
208849702d
logs
2020-08-18 12:52:54 -07:00
Chris Lu
003d48da21
adjust logs
2020-08-15 19:55:28 -07:00
Chris Lu
eb493283dd
deletion error report
2020-08-14 00:23:01 -07:00
Chris Lu
edfa73782f
adjust log level
2020-08-14 00:22:21 -07:00
James Hartig
229f11c660
Added VolumeNeedleStatus volume server grpc method
...
This is needed for the diffing tool to get the cookie for a needle
2020-07-22 15:02:21 -04:00
Chris Lu
2286d27730
rename
2020-07-15 14:19:15 -07:00
Chris Lu
0db4204c81
add is_file_chunks to filer.proto
2020-07-14 22:42:52 -07:00
Chris Lu
70d8a3a1d3
add SubscribeLocalMetadata without checking persisted meta logs
2020-07-05 15:50:07 -07:00
Chris Lu
353bea8ddb
refactoring
2020-07-03 22:25:35 -07:00
Chris Lu
37d5b3ba12
replication: pass isFromOtherCluster also to EventNotification
...
EventNotification is consistent with message queue and metadata logs.
2020-07-01 08:06:20 -07:00
Chris Lu
31e23e9783
filer: support active<=>active filer replication
2020-06-30 22:53:57 -07:00
Chris Lu
de5ca9b258
remove fixJpgOrientation
2020-06-20 12:50:40 -07:00
Chris Lu
97239ce6f1
rename filechunk is_gzipped to is_compressed
2020-06-20 08:15:49 -07:00
Chris Lu
ca3516ac6d
adjust protoc
2020-06-20 08:00:25 -07:00
Chris Lu
d4235afe4d
support multiple locks
2020-05-25 18:39:53 -07:00
Chris Lu
1ea9bc66d9
avoid leaking grpc.NewClientStream
2020-05-17 08:56:33 -07:00
Chris Lu
dfccc3c263
able to read chan and write chan
2020-05-08 02:47:22 -07:00
Chris Lu
1e3e4b3072
add broker connects to filer
2020-05-05 02:05:28 -07:00
Chris Lu
9e72e9e4b8
able to subscribe any topic from any point of time
2020-04-29 17:40:08 -07:00
Chris Lu
bdc337a719
make lock/unlock optional
2020-04-23 03:50:05 -07:00
Chris Lu
ff0a7c1d18
clean up
2020-04-23 03:10:47 -07:00
Chris Lu
30ee4f3291
add exclusive lock library on shell
2020-04-23 02:31:04 -07:00
Chris Lu
77873b832b
add master side code for cluster wise exclusive lock
2020-04-23 01:55:44 -07:00
Chris Lu
aebe39a803
avoid repeated grpc connection creation
...
fix https://github.com/chrislusf/seaweedfs/issues/1277
2020-04-22 18:26:24 -07:00
Chris Lu
9f9826e95a
refactor
2020-04-21 21:28:47 -07:00
Chris Lu
e24b25de78
async meta caching: can stream updates now
2020-04-21 21:16:13 -07:00
Chris Lu
5c348087dc
messaging: able to pub sub multiple partitions
2020-04-21 00:59:55 -07:00
Chris Lu
978da73cdb
revert this grpc related change
...
maybe related to https://github.com/chrislusf/seaweedfs/issues/1277
2020-04-20 02:53:22 -07:00
Chris Lu
ce3cb25cfb
working for in memory single log buffer
2020-04-19 23:37:50 -07:00
Chris Lu
71ffb98475
broker: add profiling
2020-04-19 03:03:40 -07:00
Chris Lu
076c8bd3bc
filer master start up with default ip address instead of just localhost
2020-04-18 15:17:27 -07:00
Chris Lu
5af49c7aa5
adjust API
2020-04-18 11:45:57 -07:00
Chris Lu
3f3dba5a68
broker: append message logs
2020-04-17 02:29:38 -07:00
Chris Lu
bda82f61bc
filer: able to append to a file
2020-04-17 02:28:09 -07:00
Chris Lu
508f3490a0
update messaging proto
2020-04-16 03:29:57 -07:00
Chris Lu
ce4b369be2
scaffolding messaging
2020-04-16 02:21:23 -07:00
Chris Lu
7764e0465c
refactoring
2020-04-12 21:00:55 -07:00
Chris Lu
8e23dc078b
refactoring
2020-04-12 20:48:21 -07:00
Chris Lu
6f948e4887
remove configurable topics folder location
2020-04-12 13:07:59 -07:00
Chris Lu
e909b55633
refactoring
2020-04-11 12:39:46 -07:00
Chris Lu
ec2eb8bc48
add If-None-Match and If-Modified-Since
...
fix https://github.com/chrislusf/seaweedfs/issues/1269
2020-04-08 08:12:00 -07:00
Chris Lu
5a5908407d
filer: support larger file size
...
fix https://github.com/chrislusf/seaweedfs/issues/1257
2020-04-05 21:48:45 -07:00
Chris Lu
2d43f85577
watch entries with common path prefix
2020-04-05 12:51:21 -07:00
Chris Lu
dcc6b6a3ed
go fmt
2020-04-05 00:52:31 -07:00
Chris Lu
bf270d9e8c
filer: able to tail meta data changes
2020-04-05 00:51:16 -07:00
Chris Lu
50a5018b7f
writing meta logs is working
2020-03-30 01:19:33 -07:00
Chris Lu
a75d50bbb8
FUSE: add configurable in memory chunk cache size
2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3
FUSE: add chunk cache for recently accessed file chunks
2020-03-28 13:43:31 -07:00
Chris Lu
712720864d
error checking for remove
2020-03-26 02:21:30 -07:00
Chris Lu
dd5b582d05
go fmt
2020-03-26 00:09:01 -07:00
Chris Lu
6d87a0c5f5
batch delete orphan data by file keys
2020-03-25 02:21:15 -07:00
Chris Lu
7f1e3c843d
refactoring
2020-03-23 01:14:21 -07:00
Chris Lu
c07bcd5065
refactoring
2020-03-23 00:30:02 -07:00
Chris Lu
654a69ff52
refactoring
2020-03-23 00:06:24 -07:00
Chris Lu
c0f0fdb3ba
refactoring
2020-03-23 00:01:34 -07:00
Chris Lu
c4bea45099
S3 API: fix DeleteMultipleObjectsHandler
...
fix https://github.com/chrislusf/seaweedfs/issues/1241
2020-03-20 14:17:31 -07:00
Chris Lu
bec6ec7db6
go fmt
2020-03-17 10:01:55 -07:00
Chris Lu
c3cb6fa1d7
volume: compaction can cause readonly volumes
...
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-17 09:43:57 -07:00
Chris Lu
2e3f6ad3a9
filer: remember content is gzipped or not
2020-03-08 21:39:33 -07:00
Chris Lu
afb20de14c
breaks dependency loop
2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b
fuse mount: avoid lookup nil entry
...
fix https://github.com/chrislusf/seaweedfs/issues/1221
2020-03-07 16:51:46 -08:00
Chris Lu
1ae83c2938
go fmt
2020-03-07 06:12:57 -08:00
Chris Lu
13e215ee5c
filer: option to encrypt data on volume server
2020-03-06 00:49:47 -08:00
Chris Lu
f90c43635d
refactoring
2020-03-04 00:39:47 -08:00
Chris Lu
410bce3925
go fmt
2020-03-01 22:39:08 -08:00
Chris Lu
6a8484b4ae
master able to list all master clients by type
2020-03-01 22:13:47 -08:00
Chris Lu
0ca68a2a6d
WIP
2020-03-01 22:13:15 -08:00
Chris Lu
555413d9fc
weed queue starts
2020-02-27 00:07:13 -08:00
Chris Lu
892e726eb9
avoid reusing context object
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-02-25 21:50:12 -08:00
Chris Lu
bd3254b53f
adjust logging
2020-02-25 17:24:08 -08:00
Chris Lu
0841bedb15
move filer assign volume grpc errror to response
2020-02-25 17:15:09 -08:00
Chris Lu
bc38b72a20
s3: implemented DeleteMultipleObjects
2020-02-25 14:38:36 -08:00
Chris Lu
6ab7368ef2
filer: dynamically create bucket under /buckets folder
2020-02-24 22:28:45 -08:00
Chris Lu
d8dec2323b
s3: move buckets folder configuration to filer
2020-02-24 14:34:14 -08:00
Chris Lu
e96f85ed8a
make pb files
2020-02-23 18:06:34 -08:00
Chris Lu
346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server
...
Update stats and status volume server
2020-02-23 18:05:25 -08:00
Chris Lu
bbb163c5ad
add for https://github.com/chrislusf/seaweedfs/pull/1207
2020-02-23 18:04:31 -08:00
LazyDBA247-Anyvision
ec8619176a
volume server protobuf: add precentFree & percentUsage to the DiskStatus
2020-02-23 23:24:24 +02:00
Chris Lu
f55f49970f
move volume server stats to grpc
2020-02-21 21:45:03 -08:00
Chris Lu
0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl
...
Add s3 ACL
2020-02-17 12:53:30 -08:00
Chris Lu
4cdde5f569
configuration stores the identity list
2020-02-17 12:31:59 -08:00
Chris Lu
9b6296e77a
volume: add grpc file read operation
...
This is added more for performance benchmarking.
2020-02-14 00:37:32 -08:00
Chris Lu
9ed364f053
support acl
2020-02-09 14:30:02 -08:00
Chris Lu
40ae533fa3
shell: add volume.configure.replication to change replication for a volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1192
2020-02-02 15:37:23 -08:00
Chris Lu
c48fc8b4de
grpc send error via response instead of grpc error
2020-01-25 09:17:19 -08:00
Chris Lu
d4e75a0d18
filer: option to create file only if it is new, O_EXCL
2020-01-22 11:42:40 -08:00
Chris Lu
03ab0b1b9a
correct .vif file creation
2019-12-28 21:40:12 -08:00
Chris Lu
0cc8347a3a
go fmt
2019-12-28 21:13:10 -08:00
Chris Lu
37b64a50b4
ec: generate and copy .vif file
2019-12-28 12:44:59 -08:00
Chris Lu
c06f7eb48a
load volume info from .vif file, use superblock as a backup
2019-12-28 12:28:58 -08:00
Chris Lu
2000284435
rename volume tier info to volume info
2019-12-28 11:21:49 -08:00
Chris Lu
b3c83f9404
fix spelling
2019-12-28 11:16:10 -08:00
Chris Lu
48d28d3eb2
tier: support remote file attributes and remember the file extension
2019-12-25 21:37:24 -08:00
Chris Lu
d960b3474a
tier storage: support downloading the remote dat files
2019-12-25 09:53:13 -08:00
Chris Lu
09ca936c78
shell: add ec.decode command
2019-12-23 12:48:20 -08:00
Chris Lu
0fa1269bc7
filer: streaming file listing
2019-12-13 00:22:37 -08:00
Chris Lu
e426bd541e
grow volumes on volume servers with slots freed by cloud storage
2019-12-03 21:36:42 -08:00
Chris Lu
1becbce657
display remote volumes on volume server ui page
2019-12-02 23:23:54 -08:00
Chris Lu
70648d35ad
go fmt
2019-12-02 20:49:58 -08:00
Chris Lu
126fb492d2
fix test
2019-12-02 15:52:33 -08:00
Chris Lu
ec8de250e2
tiered storage: can copy to s3, read from s3
...
master not aware tiered volume yet, file assigning is not working yet
2019-12-02 15:08:31 -08:00
Chris Lu
228231f3d7
minor
2019-12-02 15:08:31 -08:00
Chris Lu
0da7b894cc
pass backend config from master to volume servers
2019-11-29 01:05:09 -08:00
Chris Lu
586798ecc0
go fmt
2019-11-27 12:34:57 -08:00
Chris Lu
0f9ba84274
s3
2019-11-27 03:09:45 -08:00
Chris Lu
471902cd51
change parameter name
2019-11-10 12:05:45 -08:00
zhangsong
61fa485700
add volume number param in assign operation
2019-11-10 20:11:03 +08:00
Chris Lu
faec9076a4
adjust parameter names
2019-10-21 22:57:01 -07:00
joeslay
d53aee179b
Merge pull request #10 from chrislusf/master
...
merge seaweed master
2019-10-14 16:03:40 +01:00
Chris Lu
f8d4b7d1c0
support basic json filtering and selection
2019-10-06 22:35:08 -07:00
Chris Lu
cf47f657af
scaffold for volume server query feature
2019-10-06 22:35:08 -07:00
j.laycock
6fc6322c90
Change joeslay paths to chrislusf paths
2019-09-12 14:18:21 +01:00
joeslay
d8c34b032f
Merge pull request #6 from chrislusf/master
...
merge seaweed master
2019-09-12 11:44:09 +01:00
Chris Lu
ae53f63680
filer: recursive deletion optionally ignoring any errors
...
fix https://github.com/chrislusf/seaweedfs/issues/1062
2019-09-11 20:26:20 -07:00
j.laycock
84f2dc3b53
Remove test code
2019-09-05 11:14:02 +01:00
Tom Maxwell
4a878c0006
Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve
2019-09-04 15:27:14 +01:00
j.laycock
9aa0859697
Remove inmemory from vaccum stuff
2019-09-03 18:20:45 +01:00
j.laycock
cbd0a98fa1
Add InMemory to Volume Grow structure
2019-09-03 18:19:02 +01:00
Tom Maxwell
d637d86d22
Changes to try and pass the URL parameters through - in memory flag not working still
2019-09-03 15:41:28 +01:00
j.laycock
595a1beff0
Swap imports to use joeslay
2019-09-02 11:28:40 +01:00
Chris Lu
19360ab10a
master: redirect clients to the new leader
2019-07-31 01:54:42 -07:00
Chris Lu
6883f9e322
mark volume readonly before ec encoding
2019-06-26 23:02:22 -07:00
Chris Lu
d5560f2705
simplify metrics settings
2019-06-23 15:29:49 -07:00
Chris Lu
8da5d5b094
filer.copy: use filer settings, avoid unnecessary command line options
...
fix https://github.com/chrislusf/seaweedfs/issues/968
2019-06-23 01:57:35 -07:00
Chris Lu
1babec00e7
check deleted chunks faster
2019-06-22 13:22:22 -07:00
Chris Lu
cd45ab072a
fix compilation error
2019-06-22 12:30:08 -07:00
Chris Lu
a111f26fe6
avoid nil
...
fix https://github.com/chrislusf/seaweedfs/issues/988
2019-06-21 20:56:27 -07:00