gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9250 commits 28 branches 278 tags 135 MiB
Commit graph

460 commits

Author SHA1 Message Date
Chris Lu 4909bd9684 gRpc connection error on filer when no volume left #2243 
fix https://github.com/chrislusf/seaweedfs/issues/2243

grpc do not cache connections only when connection problem happens.
Normal error results should not close the shared grpc connection.
 2021-08-14 05:06:44 -07:00
Chris Lu 5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Chris Lu 5571f4f70a master: add master.follower to handle read file id lookup requests 2021-08-12 18:10:59 -07:00
Chris Lu 69655ba8e5 mount: cache on reading remote storage 2021-08-09 22:11:57 -07:00
Chris Lu 713c035a6e shell: remote.cache remote.uncache 2021-08-09 14:35:18 -07:00
Chris Lu c0b12da4ef shell: add filer.remote.unmount 2021-08-08 22:26:37 -07:00
Chris Lu eed26af266 Merge branch 'master' into add_remote_storage 2021-08-08 15:48:04 -07:00
Chris Lu 4370a4db63 use int64 for volume count in case of negative overflow 2021-08-08 15:19:39 -07:00
Chris Lu 270770d7d7 refactor 2021-08-07 14:18:53 -07:00
Chris Lu 6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu 9df7d16791 read <- remote_storage 2021-07-31 22:39:38 -07:00
Chris Lu 899963ac20 remote storage location changed to struct 2021-07-29 02:08:55 -07:00
Chris Lu 1752eeb538 remote.mount saves the mapping 2021-07-27 03:26:35 -07:00
Chris Lu 4b94b03d90 directory to remote storage mapping 2021-07-27 01:16:28 -07:00
Chris Lu 99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00
Chris Lu 70effac0d3 configure and store remote configurations 2021-07-21 02:24:34 -07:00
Chris Lu 450222dd64 add remote to filer.Entry and filer_pb entry, add RemoteConf 2021-07-19 02:47:27 -07:00
Chris Lu 8dc5def435 proto: add remote 2021-07-19 01:12:31 -07:00
Chris Lu b194f91f47 add version to filer configuration response 2021-07-12 01:23:20 -07:00
lyg 1a4db87e19 fix s3 metadata error with multipart upload 2021-07-02 11:00:42 +08:00
Chris Lu 215b169562 mount: recursively rename locally 2021-07-01 01:19:31 -07:00
Chris Lu ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu c5de97ddbe keep alive for streaming connections 
fix https://github.com/chrislusf/seaweedfs/issues/2096

related to https://github.com/chrislusf/seaweedfs/issues/1277
 2021-05-31 01:26:24 -07:00
Chris Lu 1456616a77 recreate grpc connections if too many errors 
address https://github.com/chrislusf/seaweedfs/issues/2098
 2021-05-30 00:07:43 -07:00
Chris Lu 3d3fa43542 filer: re-create grpc connections if having transport error 
fix https://github.com/chrislusf/seaweedfs/issues/2070
 2021-05-22 17:46:53 -07:00
Chris Lu dc1309f084 FUSE mount: support multiple filers 
fix https://github.com/chrislusf/seaweedfs/issues/2015
fix https://github.com/chrislusf/seaweedfs/issues/1531
 2021-05-21 01:28:00 -07:00
Chris Lu f0ad172e80 shell: show which server holds the lock 
fix https://github.com/chrislusf/seaweedfs/issues/1983
 2021-04-22 23:56:35 -07:00
Chris Lu b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Chris Lu 35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
wuh-fnst 4c1d945e46 make List correctly judge whether it is the last file 2021-03-12 11:22:26 +08:00
Chris Lu 828f6e9f4d volume: auto add missing vif files 
fix https://github.com/chrislusf/seaweedfs/issues/1878
 2021-03-09 12:09:32 -08:00
Chris Lu 9d402ebe9f refactoring 2021-03-06 14:26:27 -08:00
Chris Lu bd727b7b53 go fmt 2021-03-02 20:59:56 -08:00
Chris Lu e982b06bcd support IPv6 2021-03-02 20:59:39 -08:00
Chris Lu f465d63b5d grpc should fail when heart beating to master 
fix https://github.com/chrislusf/seaweedfs/issues/1820
 2021-02-19 03:37:29 -08:00
Chris Lu f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu 487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu 770393a48c volume: add capability to change disk type when moving a volume 2021-02-09 23:58:08 -08:00
Chris Lu 821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu 15c60cbb26 close the grpc connection after 10 hours 
related to https://github.com/chrislusf/seaweedfs/issues/1782
 2021-02-07 03:50:01 -08:00
Chris Lu 990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu 00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu 6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu 096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu 80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu 2e8dba571b adjust volume server UI 2020-12-14 00:51:57 -08:00
Chris Lu 94525aa0fd allocate volume by disk type 2020-12-13 23:08:21 -08:00
Chris Lu 0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu 316d1b4e69 refactor APIs 2020-12-12 12:42:53 -08:00
Chris Lu 97c942b5ee paginate through large directories 2020-12-09 23:23:38 -08:00
Chris Lu 9acda432fe fix import cycle 2020-12-06 20:12:52 -08:00
Chris Lu ae5eb85a06 refactoring 2020-12-06 20:05:06 -08:00
Chris Lu f4abd01adf filer: cache small file to filer store 2020-11-30 04:34:04 -08:00
Chris Lu 965413c21b shell: add volume.vacuum command 2020-11-28 23:18:02 -08:00
Chris Lu dc304342b2 fs.configure: configurable volume growth 2020-11-17 01:00:02 -08:00
Chris Lu 5f19e81dab filer confi: support hierachical configuration 2020-11-16 16:50:12 -08:00
Chris Lu ee2fa14dbe filer conf: delete location specific configuration 2020-11-15 20:15:47 -08:00
Chris Lu 0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
Chris Lu 95c0de285d refactoring 2020-11-15 16:58:48 -08:00
Chris Lu 0a406f652e load filer conf and match by prefix 2020-11-15 00:26:05 -08:00
Chris Lu 68043cfcac add reference implementation to detect create/update/delete/rename events 2020-11-14 21:21:58 -08:00
Chris Lu a4dfa00030 adjust filer conf 2020-11-12 00:39:59 -08:00
Chris Lu 2251a8e472 minor 2020-11-12 00:38:58 -08:00
Chris Lu 5a16f17e47 remove unused message type 2020-11-12 00:38:23 -08:00
Chris Lu c32b6f4d9d prepare for path based configuration 2020-11-11 23:46:45 -08:00
Konstantin Lebedev 1eec5c8d5d gen pb 2020-11-12 04:10:06 +05:00
Konstantin Lebedev dc26012a3b initial 2020-11-11 15:03:47 +05:00
Konstantin Lebedev 918546bbdb add WaitForReady to grpc DialOption 2020-10-29 08:29:04 +05:00
Chris Lu f375b93aef renaming 2020-10-25 15:32:43 -07:00
Chris Lu 6c92fa349d filer: support default rack 
fix https://github.com/chrislusf/seaweedfs/issues/1546
 2020-10-20 17:41:39 -07:00
Chris Lu ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Chris Lu 723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu da4edf3651 master: check peers for existing leader before starting a leader election 
fix https://github.com/chrislusf/seaweedfs/issues/1509
 2020-10-07 01:25:39 -07:00
Chris Lu 55cb68c09c filer: replication follows master config if not specified 2020-09-30 09:15:55 -07:00
Chris Lu 1012df7bb5 switch hardlink id from int64 to bytes 2020-09-24 11:11:42 -07:00
Chris Lu 5e239afdfc hardlink works now 2020-09-24 03:06:48 -07:00
Chris Lu cb427d48fa filer report metrics configuration 2020-09-17 06:46:51 -07:00
Chris Lu c9202c4b3d add storage backend to GetMasterConfigurationResponse 2020-09-16 01:12:22 -07:00
Chris Lu 200fe5c83e go fmt 2020-09-13 21:26:30 -07:00
Chris Lu 5d6753fb98 shell: add volumeServer.leave command 2020-09-13 21:25:51 -07:00
Chris Lu daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu 387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu bba90ff3c8 read filer signature 2020-09-05 22:52:15 -07:00
Chris Lu a41588279a change log level 5 to 4 2020-08-30 20:12:04 -07:00
Chris Lu ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
James Hartig 3ccfa4c6ad Added VolumeMarkWritable and VolumeStatus grpc methods 
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
 2020-08-19 11:42:56 -04:00
Chris Lu 208849702d logs 2020-08-18 12:52:54 -07:00
Chris Lu 003d48da21 adjust logs 2020-08-15 19:55:28 -07:00
Chris Lu eb493283dd deletion error report 2020-08-14 00:23:01 -07:00
Chris Lu edfa73782f adjust log level 2020-08-14 00:22:21 -07:00
James Hartig 229f11c660 Added VolumeNeedleStatus volume server grpc method 
This is needed for the diffing tool to get the cookie for a needle
 2020-07-22 15:02:21 -04:00
Chris Lu 2286d27730 rename 2020-07-15 14:19:15 -07:00
Chris Lu 0db4204c81 add is_file_chunks to filer.proto 2020-07-14 22:42:52 -07:00
Chris Lu 70d8a3a1d3 add SubscribeLocalMetadata without checking persisted meta logs 2020-07-05 15:50:07 -07:00
Chris Lu 353bea8ddb refactoring 2020-07-03 22:25:35 -07:00
Chris Lu 37d5b3ba12 replication: pass isFromOtherCluster also to EventNotification 
EventNotification is consistent with message queue and metadata logs.
 2020-07-01 08:06:20 -07:00
Chris Lu 31e23e9783 filer: support active<=>active filer replication 2020-06-30 22:53:57 -07:00
Chris Lu de5ca9b258 remove fixJpgOrientation 2020-06-20 12:50:40 -07:00
Chris Lu 97239ce6f1 rename filechunk is_gzipped to is_compressed 2020-06-20 08:15:49 -07:00
Chris Lu ca3516ac6d adjust protoc 2020-06-20 08:00:25 -07:00
Chris Lu d4235afe4d support multiple locks 2020-05-25 18:39:53 -07:00
Chris Lu 1ea9bc66d9 avoid leaking grpc.NewClientStream 2020-05-17 08:56:33 -07:00
Chris Lu dfccc3c263 able to read chan and write chan 2020-05-08 02:47:22 -07:00
Chris Lu 1e3e4b3072 add broker connects to filer 2020-05-05 02:05:28 -07:00
Chris Lu 9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu bdc337a719 make lock/unlock optional 2020-04-23 03:50:05 -07:00
Chris Lu ff0a7c1d18 clean up 2020-04-23 03:10:47 -07:00
Chris Lu 30ee4f3291 add exclusive lock library on shell 2020-04-23 02:31:04 -07:00
Chris Lu 77873b832b add master side code for cluster wise exclusive lock 2020-04-23 01:55:44 -07:00
Chris Lu aebe39a803 avoid repeated grpc connection creation 
fix https://github.com/chrislusf/seaweedfs/issues/1277
 2020-04-22 18:26:24 -07:00
Chris Lu 9f9826e95a refactor 2020-04-21 21:28:47 -07:00
Chris Lu e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu 5c348087dc messaging: able to pub sub multiple partitions 2020-04-21 00:59:55 -07:00
Chris Lu 978da73cdb revert this grpc related change 
maybe related to https://github.com/chrislusf/seaweedfs/issues/1277
 2020-04-20 02:53:22 -07:00
Chris Lu ce3cb25cfb working for in memory single log buffer 2020-04-19 23:37:50 -07:00
Chris Lu 71ffb98475 broker: add profiling 2020-04-19 03:03:40 -07:00
Chris Lu 076c8bd3bc filer master start up with default ip address instead of just localhost 2020-04-18 15:17:27 -07:00
Chris Lu 5af49c7aa5 adjust API 2020-04-18 11:45:57 -07:00
Chris Lu 3f3dba5a68 broker: append message logs 2020-04-17 02:29:38 -07:00
Chris Lu bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu 508f3490a0 update messaging proto 2020-04-16 03:29:57 -07:00
Chris Lu ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Chris Lu 7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu 8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu 6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu 5a5908407d filer: support larger file size 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-05 21:48:45 -07:00
Chris Lu 2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu dcc6b6a3ed go fmt 2020-04-05 00:52:31 -07:00
Chris Lu bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu 50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu 826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu 712720864d error checking for remove 2020-03-26 02:21:30 -07:00
Chris Lu dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu 6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu 7f1e3c843d refactoring 2020-03-23 01:14:21 -07:00
Chris Lu c07bcd5065 refactoring 2020-03-23 00:30:02 -07:00
Chris Lu 654a69ff52 refactoring 2020-03-23 00:06:24 -07:00
Chris Lu c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu bec6ec7db6 go fmt 2020-03-17 10:01:55 -07:00
Chris Lu c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu 2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu 8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu 1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu 13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu 410bce3925 go fmt 2020-03-01 22:39:08 -08:00
Chris Lu 6a8484b4ae master able to list all master clients by type 2020-03-01 22:13:47 -08:00
Chris Lu 0ca68a2a6d WIP 2020-03-01 22:13:15 -08:00
Chris Lu 555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu 892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu bd3254b53f adjust logging 2020-02-25 17:24:08 -08:00
Chris Lu 0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu 6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu e96f85ed8a make pb files 2020-02-23 18:06:34 -08:00
Chris Lu 346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server 
Update stats and status volume server
 2020-02-23 18:05:25 -08:00
Chris Lu bbb163c5ad add for https://github.com/chrislusf/seaweedfs/pull/1207 2020-02-23 18:04:31 -08:00
LazyDBA247-Anyvision ec8619176a volume server protobuf: add precentFree & percentUsage to the DiskStatus 2020-02-23 23:24:24 +02:00
Chris Lu f55f49970f move volume server stats to grpc 2020-02-21 21:45:03 -08:00
Chris Lu 0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl 
Add s3 ACL
 2020-02-17 12:53:30 -08:00
Chris Lu 4cdde5f569 configuration stores the identity list 2020-02-17 12:31:59 -08:00
Chris Lu 9b6296e77a volume: add grpc file read operation 
This is added more for performance benchmarking.
 2020-02-14 00:37:32 -08:00
Chris Lu 9ed364f053 support acl 2020-02-09 14:30:02 -08:00
Chris Lu 40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu 03ab0b1b9a correct .vif file creation 2019-12-28 21:40:12 -08:00
Chris Lu 0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu 37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu c06f7eb48a load volume info from .vif file, use superblock as a backup 2019-12-28 12:28:58 -08:00
Chris Lu 2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu b3c83f9404 fix spelling 2019-12-28 11:16:10 -08:00
Chris Lu 48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
Chris Lu d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
Chris Lu 09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu 0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu 1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu 70648d35ad go fmt 2019-12-02 20:49:58 -08:00
Chris Lu 126fb492d2 fix test 2019-12-02 15:52:33 -08:00
Chris Lu ec8de250e2 tiered storage: can copy to s3, read from s3 
master not aware tiered volume yet, file assigning is not working yet
 2019-12-02 15:08:31 -08:00
Chris Lu 228231f3d7 minor 2019-12-02 15:08:31 -08:00
Chris Lu 0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu 586798ecc0 go fmt 2019-11-27 12:34:57 -08:00
Chris Lu 0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Chris Lu 471902cd51 change parameter name 2019-11-10 12:05:45 -08:00
zhangsong 61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00
Chris Lu faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
joeslay d53aee179b
Merge pull request #10 from chrislusf/master 
merge seaweed master
 2019-10-14 16:03:40 +01:00
Chris Lu f8d4b7d1c0 support basic json filtering and selection 2019-10-06 22:35:08 -07:00
Chris Lu cf47f657af scaffold for volume server query feature 2019-10-06 22:35:08 -07:00