gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
8853 commits 28 branches 278 tags 135 MiB
Commit graph

666 commits

Author SHA1 Message Date
Konstantin Lebedev f5246b748d Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	weed/pb/master_pb/master.pb.go
 2022-04-07 18:50:27 +05:00
Konstantin Lebedev 85d80fd36d fix removing old raft server 2022-04-07 15:31:37 +05:00
Konstantin Lebedev 357aa818fe add raft shell cmds 2022-04-06 15:23:53 +05:00
chrislu bc888226fc erasure coding: tracking encoded/decoded volumes 
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
 2022-04-05 19:03:02 -07:00
chrislu 4aae87f405 check missing hdd disk type 2022-04-04 14:48:00 -07:00
chrislu 6a2bcd03aa configure mount quota 2022-04-02 21:34:26 -07:00
chrislu 105578a2f2 skip pinging self for master and volume server 2022-04-01 20:25:35 -07:00
chrislu 4ecba915f3 add check between peers 2022-04-01 17:40:25 -07:00
chrislu 4b5c0e3fa9 check cluster connectivities 2022-04-01 17:27:49 -07:00
Konstantin Lebedev 7f1383a41e findExtraChunksInVolumeServers in consideration of replication 2022-04-01 14:45:41 +05:00
Konstantin Lebedev 3817e05dd0 fix collect filer files 2022-04-01 10:17:09 +05:00
Konstantin Lebedev 3cedb21bb7 skip new entities 2022-03-31 21:36:10 +05:00
Konstantin Lebedev 1d9e30d8c0 fsck replicas 2022-03-31 19:10:06 +05:00
Konstantin Lebedev 4f5b018a02 collecting only bucket volumes 2022-03-31 13:35:58 +05:00
chrislu 21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
banjiaojuhao f7f2a597dd minor 2022-03-08 16:22:55 +08:00
chrislu ede6ce44c6 fix test 2022-03-06 17:09:55 -08:00
chrislu 6e49e75a5b use logical number of files and sizes for statistics and quota 2022-03-04 18:47:44 -08:00
chrislu 95717d1006 simpler output 2022-02-24 13:50:08 -08:00
Chris Lu 6834df77a0
Merge pull request #2701 from guo-sj/fix_bugs_in_return_value 2022-02-23 19:23:37 -08:00
guosj 26f3ab8d4b fix a return bug in func (c *commandVacuum) Do 2022-02-24 09:04:38 +08:00
Chris Lu 38dcaaa76e
Merge pull request #2683 from guo-sj/fix_fsconfigure_bug 
fix minor bug in commandFsConfigure.Do
 2022-02-16 18:16:47 -08:00
guosj ca121ecd66 fix minor bug in commandFsConfigure.Do 2022-02-17 09:57:52 +08:00
Chris Lu 64afbc5235
Merge pull request #2640 from Radtoo/fsck_with_delete 
Added basic deletion capabilities to fsck.
 2022-02-14 23:00:01 -08:00
Radtoo 724ce04b1c We picked the second flag's name. 2022-02-13 00:53:35 +01:00
Radtoo fbe7ed7927 Revert the the flag to "reallyDeleteFromVolume". 2022-02-10 20:01:48 +01:00
chrislu 21aaa4c1f1 ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount) 
fix https://github.com/chrislusf/seaweedfs/issues/2642
 2022-02-08 01:51:13 -08:00
chrislu 13f6ec1c4e test checking ec distribution 2022-02-08 01:50:05 -08:00
chrislu f34c2ff7c5 use embed txt 2022-02-08 01:14:27 -08:00
chrislu c8c7c10c3f volume.tier.move: avoid double counting 
related to https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:57:35 -08:00
chrislu f18803424a volume.balance: add delay during tight loop 
fix https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:53:55 -08:00
chrislu 85c1615b43 filer read empty file may cause OOM in some cases 
fix https://github.com/chrislusf/seaweedfs/issues/2641
 2022-02-07 23:08:54 -08:00
Radtoo fbb14e0ea8 Adding separate toggle to purge absent vols 
While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected.

Also renamed forcePurge to just purge.
 2022-02-06 23:46:52 +01:00
Radtoo 6b17f45da2 Also delete paths for which a volume is entirely absent. 2022-02-06 23:22:04 +01:00
Radtoo fa0cfdfb7f Added basic deletion capabilities to fsck. 2022-02-06 17:05:58 +01:00
chrislu 6bee1e9714 [volume.check.disk] fix wrong logic to compare 
fix https://github.com/chrislusf/seaweedfs/issues/2626

minuend - subtrahend
 2022-02-01 07:48:28 -08:00
chrislu b8490fe427 adjust volume count even when not applying the changes 2022-01-28 19:11:46 -08:00
divanikus 67e3fe996a async volumeTierMove 2022-01-26 18:22:31 +03:00
chrislu 9d0f58c329 skip printing fs configuration 2022-01-21 13:29:47 -08:00
chrislu ce2049cdb6 refactoring, move genFn before saveFn 2022-01-21 12:08:58 -08:00
chrislu e47f63d159 enforce bucket quota 2022-01-21 02:34:42 -08:00
chrislu 6e57d8d0de s3: check bucket usage and adjust read only according to quota 2022-01-21 02:15:27 -08:00
chrislu 606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
chrislu b1063162b6 display bucket quota 2022-01-21 00:55:04 -08:00
chrislu f103491912 s3: list bucket size from weed shell 2022-01-21 00:26:49 -08:00
chrislu ea57654e34 refactoring 2022-01-21 00:16:50 -08:00
chrislu 15c01d8b7f add some notes 2022-01-12 15:04:48 -08:00
chrislu 107a4884a8 shell: tighter memory allocation 2022-01-12 14:59:29 -08:00
chrislu adfd54e7c4 fix compilation 2022-01-12 01:24:24 -08:00
chrislu 826a7b307e master: remove hard coded filer settings in master.toml 
fix https://github.com/chrislusf/seaweedfs/issues/2529
 2022-01-12 01:11:25 -08:00
chrislu 60dc450091 skip fixing read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2562
 2022-01-06 09:52:28 -08:00
chrislu 9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
banjiaojuhao 083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
chrislu 94e5c0a454 skip purging from ec volumes 2021-12-12 22:55:27 -08:00
chrislu a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
chrislu 991a3dca0d rename file 2021-12-10 13:11:00 -08:00
chrislu 7f0a97c7b6 shell: volume.list supports different verbosity level 2021-12-05 21:54:40 -08:00
chrislu e6c026db65 volume.fix.replication: fix misplaced volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2416
 2021-12-05 16:56:25 -08:00
chrislu 53e2dee177 [volume.check.disk] was using the wrong source and target locations 
fix https://github.com/chrislusf/seaweedfs/issues/2268
 2021-12-05 14:32:04 -08:00
Chris Lu f3a334965d Update command_remote_uncache.go 
minor
 2021-11-28 23:10:32 -08:00
Chris Lu 9ccfc1cfee Update command_volume_fsck.go 
minor
 2021-11-28 22:39:24 -08:00
Chris Lu 71a94267c9 Update command_volume_fsck.go 
remove unused code
 2021-11-28 22:37:11 -08:00
user 9668b15f38 Filtering by volume id is prioritized. 2021-11-11 18:18:56 +09:00
user 563a74a9eb Volume filter by collection pattern added. 2021-11-11 18:01:47 +09:00
user dbb8003ce3 Volume filter function added. 2021-11-11 17:45:17 +09:00
user c387fe957b Unused parameter removed. 2021-11-11 16:08:41 +09:00
Chris Lu 4729a57cc0 use constants 2021-11-08 17:47:56 -08:00
Chris Lu a8b0f8864d fix help message 2021-11-07 12:38:35 -08:00
Chris Lu 5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu 314c32514b add command to list filers 2021-11-05 17:39:41 -07:00
Chris Lu 00ae965d8d randomize a bit for ec shards distribution 2021-11-04 09:23:40 -07:00
Chris Lu af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu ab97b17e62 better printout 2021-11-02 23:45:47 -07:00
Chris Lu 5160eb08f7 shell: optionally read filer address from master 2021-11-02 23:38:45 -07:00
Chris Lu 18bfbf62fc add one unit test 2021-11-02 23:36:43 -07:00
Chris Lu 794375ca0a adjust help message since both fullPercent and quietFor are needed. 2021-11-01 17:22:47 -07:00
Chris Lu 24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu d774fa6c9a rename variable 2021-10-25 14:39:20 -07:00
Chris Lu 2539ba0b62 fix compilation 2021-10-25 14:38:48 -07:00
Chris Lu 5f2d7c1589 erasure coding: skip erasure coding if less than recommended 4 nodes 2021-10-25 14:38:11 -07:00
Chris Lu 5435027ff0 volume copy: stream out copying progress and avoid grpc request timeout 
fix https://github.com/chrislusf/seaweedfs/issues/2386
 2021-10-24 02:52:56 -07:00
Eng Zer Jun a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Konstantin Lebedev be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu e862b2529a refactor 2021-10-01 12:10:11 -07:00
Konstantin Lebedev 5e64b22b45 check that the topology has been updated 2021-10-01 18:51:22 +05:00
Konstantin Lebedev 2cecde89c3 rename opt volumesPerStep 2021-10-01 00:17:54 +05:00
Konstantin Lebedev fc51ffce2b https://github.com/chrislusf/seaweedfs/issues/1846 2021-09-30 20:24:24 +05:00
Chris Lu ad5099e570 refactor 2021-09-19 12:02:23 -07:00
Chris Lu 2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu 119d5908dd shell: do not need to lock to see volume -h 2021-09-13 22:13:34 -07:00
Chris Lu 6cd1ce8b74 erasure coding: add cleanup step if anything goes wrong 2021-09-13 01:55:49 -07:00
Chris Lu e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu 0b4269b6a8 remember commands even if failed 2021-09-08 15:55:19 -07:00
Chris Lu 0128239c0f handle ipv6 addresses 2021-09-07 16:43:54 -07:00
Chris Lu 8e4b43a017 minor 2021-09-06 15:13:38 -07:00
Chris Lu 57a95887d2 remote.cache remote.uncache supports all mounted directories 2021-09-05 14:47:06 -07:00
Chris Lu c735608685 obfuscate secret keys on display 2021-09-05 14:23:49 -07:00
Chris Lu 32e94de86a default auto trimming suffix to true 2021-09-05 13:27:35 -07:00
Chris Lu 60573fd3e2 option to map remote bucket to trimmed bucket name 2021-09-05 11:55:52 -07:00
Chris Lu d57d4c5f8f shell: add remote.mount.buckets 2021-09-04 21:37:25 -07:00
Chris Lu d983aa4c7d correct filtering 2021-09-04 13:58:14 -07:00
Chris Lu 49952ba905 fix cache/uncache filters 2021-09-04 13:45:39 -07:00
Chris Lu df29281536 rename 2021-09-04 05:11:06 -07:00
Chris Lu c3db389e42 refactor 2021-09-04 05:09:38 -07:00
Chris Lu 8707ef00a6 refactor 2021-09-04 05:02:20 -07:00
Chris Lu 63c84584bb refactor 2021-09-04 04:53:36 -07:00
Chris Lu da49d25950 auto list of storage types 2021-09-04 00:18:21 -07:00
Chris Lu 38f73c8324 add gcs project id 2021-09-03 23:38:33 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu 3bd48c4f29 filer.remote.sync: exit when directory is unmounted 
this will not propagate the deletions back to the cloud
 2021-09-01 01:29:22 -07:00
Chris Lu cca588429b shell: fs.meta.save skip saving system logs 2021-08-31 23:05:56 -07:00
Chris Lu 0ee208c1a4 adjust example 2021-08-31 02:04:32 -07:00
Chris Lu 97b7948cd9 adjust error message 2021-08-31 01:07:34 -07:00
Chris Lu c9205458a1 cloud drive: create mount directory if not exists 2021-08-31 01:07:19 -07:00
Chris Lu 1dfcdc53b7 cloud drive: add storj, filebase 2021-08-30 17:43:14 -07:00
Chris Lu 701cc133f3 cloud drive: s3 add option for v4 signature 2021-08-30 17:28:33 -07:00
Chris Lu f117f7bf54 adjust comment 2021-08-30 15:56:14 -07:00
Chris Lu 44d0ac95f6 add cluster id if already set 2021-08-29 22:19:46 -07:00
Chris Lu 18bf556d93 fix comments 2021-08-29 22:19:25 -07:00
Chris Lu 21f37a0a75 remove unused code 2021-08-29 22:19:11 -07:00
Chris Lu 001a472057 cloud mount: remote storage support hdfs 2021-08-29 18:41:29 -07:00
Chris Lu 975fbc9ee3 add wasabi example 2021-08-28 22:49:38 -07:00
Chris Lu 65a29ad42a add warning on unmount a folder 2021-08-28 02:25:48 -07:00
Chris Lu a22f37b01c remove tencent region 2021-08-28 02:23:03 -07:00
Chris Lu 0d8936f6cd add aliyun example 2021-08-28 01:09:48 -07:00
Chris Lu 49a8dfb976 adjust default concurrent level 2021-08-26 17:05:56 -07:00
Chris Lu 6a0bb7106b cloud drive: parallelize remote storage downloading 2021-08-26 16:16:26 -07:00
Chris Lu 05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
Chris Lu c08ac536ed cloud drive: add support for Wasabi 
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
 2021-08-25 17:34:29 -07:00
Chris Lu a7a914f120 cloud drive: add support for Baidu BOS 2021-08-24 23:46:33 -07:00
Chris Lu 2d805beed4 add help mesages 2021-08-24 23:28:35 -07:00
Chris Lu c6c97bd83d add default env variables 2021-08-24 23:25:36 -07:00
Chris Lu a19c728034 cloud drive: add support for Tencent COS 2021-08-24 23:19:45 -07:00
Chris Lu 19a81d25af cloud drive: add support for Aliyun OSS 2021-08-24 23:14:24 -07:00
Chris Lu 47d775cf68 cloud drive: add support for BackBlaze 2021-08-24 22:30:06 -07:00
Chris Lu 6bab20d862 cat entry content 2021-08-24 02:38:32 -07:00
Chris Lu e9ebe24f2e cloud drive: add support for Azure 2021-08-24 01:18:30 -07:00
Chris Lu 00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu 2836a58d87 cloud drive: S3 supports storage class 2021-08-23 02:18:59 -07:00
Chris Lu ca35a77e85 adjust help message 2021-08-23 00:37:55 -07:00
Chris Lu 258063de26 cloud drive: add google cloud storage 2021-08-23 00:29:27 -07:00
Chris Lu 2158d4fe4d adjust help message 2021-08-21 02:17:10 -07:00
Chris Lu ccb3df41f8 elide secret key 2021-08-17 11:27:08 -07:00
Chris Lu 49b5e47bd1 retry forever with filer.remote.sync, and some refactoring 2021-08-15 12:38:26 -07:00
Chris Lu a539d64896 refactor 2021-08-15 12:09:54 -07:00
Chris Lu ec989b0377 remove println 2021-08-15 02:10:27 -07:00
Chris Lu 9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu c34747c79d rename, fix wrong logic. 2021-08-14 21:46:34 -07:00
Chris Lu cb53802752 adjust help message 2021-08-14 15:55:53 -07:00
Chris Lu 889b143fa7 adjust modification detection logic 2021-08-14 15:44:47 -07:00