gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
7559 commits 28 branches 278 tags 135 MiB
Commit graph

569 commits

Author SHA1 Message Date
Chris Lu 32e94de86a default auto trimming suffix to true 2021-09-05 13:27:35 -07:00
Chris Lu 60573fd3e2 option to map remote bucket to trimmed bucket name 2021-09-05 11:55:52 -07:00
Chris Lu d57d4c5f8f shell: add remote.mount.buckets 2021-09-04 21:37:25 -07:00
Chris Lu d983aa4c7d correct filtering 2021-09-04 13:58:14 -07:00
Chris Lu 49952ba905 fix cache/uncache filters 2021-09-04 13:45:39 -07:00
Chris Lu df29281536 rename 2021-09-04 05:11:06 -07:00
Chris Lu c3db389e42 refactor 2021-09-04 05:09:38 -07:00
Chris Lu 8707ef00a6 refactor 2021-09-04 05:02:20 -07:00
Chris Lu 63c84584bb refactor 2021-09-04 04:53:36 -07:00
Chris Lu da49d25950 auto list of storage types 2021-09-04 00:18:21 -07:00
Chris Lu 38f73c8324 add gcs project id 2021-09-03 23:38:33 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu 3bd48c4f29 filer.remote.sync: exit when directory is unmounted 
this will not propagate the deletions back to the cloud
 2021-09-01 01:29:22 -07:00
Chris Lu cca588429b shell: fs.meta.save skip saving system logs 2021-08-31 23:05:56 -07:00
Chris Lu 0ee208c1a4 adjust example 2021-08-31 02:04:32 -07:00
Chris Lu 97b7948cd9 adjust error message 2021-08-31 01:07:34 -07:00
Chris Lu c9205458a1 cloud drive: create mount directory if not exists 2021-08-31 01:07:19 -07:00
Chris Lu 1dfcdc53b7 cloud drive: add storj, filebase 2021-08-30 17:43:14 -07:00
Chris Lu 701cc133f3 cloud drive: s3 add option for v4 signature 2021-08-30 17:28:33 -07:00
Chris Lu f117f7bf54 adjust comment 2021-08-30 15:56:14 -07:00
Chris Lu 44d0ac95f6 add cluster id if already set 2021-08-29 22:19:46 -07:00
Chris Lu 18bf556d93 fix comments 2021-08-29 22:19:25 -07:00
Chris Lu 21f37a0a75 remove unused code 2021-08-29 22:19:11 -07:00
Chris Lu 001a472057 cloud mount: remote storage support hdfs 2021-08-29 18:41:29 -07:00
Chris Lu 975fbc9ee3 add wasabi example 2021-08-28 22:49:38 -07:00
Chris Lu 65a29ad42a add warning on unmount a folder 2021-08-28 02:25:48 -07:00
Chris Lu a22f37b01c remove tencent region 2021-08-28 02:23:03 -07:00
Chris Lu 0d8936f6cd add aliyun example 2021-08-28 01:09:48 -07:00
Chris Lu 49a8dfb976 adjust default concurrent level 2021-08-26 17:05:56 -07:00
Chris Lu 6a0bb7106b cloud drive: parallelize remote storage downloading 2021-08-26 16:16:26 -07:00
Chris Lu 05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
Chris Lu c08ac536ed cloud drive: add support for Wasabi 
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
 2021-08-25 17:34:29 -07:00
Chris Lu a7a914f120 cloud drive: add support for Baidu BOS 2021-08-24 23:46:33 -07:00
Chris Lu 2d805beed4 add help mesages 2021-08-24 23:28:35 -07:00
Chris Lu c6c97bd83d add default env variables 2021-08-24 23:25:36 -07:00
Chris Lu a19c728034 cloud drive: add support for Tencent COS 2021-08-24 23:19:45 -07:00
Chris Lu 19a81d25af cloud drive: add support for Aliyun OSS 2021-08-24 23:14:24 -07:00
Chris Lu 47d775cf68 cloud drive: add support for BackBlaze 2021-08-24 22:30:06 -07:00
Chris Lu 6bab20d862 cat entry content 2021-08-24 02:38:32 -07:00
Chris Lu e9ebe24f2e cloud drive: add support for Azure 2021-08-24 01:18:30 -07:00
Chris Lu 00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu 2836a58d87 cloud drive: S3 supports storage class 2021-08-23 02:18:59 -07:00
Chris Lu ca35a77e85 adjust help message 2021-08-23 00:37:55 -07:00
Chris Lu 258063de26 cloud drive: add google cloud storage 2021-08-23 00:29:27 -07:00
Chris Lu 2158d4fe4d adjust help message 2021-08-21 02:17:10 -07:00
Chris Lu ccb3df41f8 elide secret key 2021-08-17 11:27:08 -07:00
Chris Lu 49b5e47bd1 retry forever with filer.remote.sync, and some refactoring 2021-08-15 12:38:26 -07:00
Chris Lu a539d64896 refactor 2021-08-15 12:09:54 -07:00
Chris Lu ec989b0377 remove println 2021-08-15 02:10:27 -07:00
Chris Lu 9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu c34747c79d rename, fix wrong logic. 2021-08-14 21:46:34 -07:00
Chris Lu cb53802752 adjust help message 2021-08-14 15:55:53 -07:00
Chris Lu 889b143fa7 adjust modification detection logic 2021-08-14 15:44:47 -07:00
Chris Lu 708debca14 remote.cache and uncache: more flexible options to select files to cache or uncache 2021-08-14 15:11:55 -07:00
Chris Lu 0db2517994 go fmt 2021-08-14 02:55:44 -07:00
Chris Lu 0f7d4556d8 shell: volume.tier.move makes up changes if volume move failed 2021-08-13 03:09:28 -07:00
Chris Lu 333cdce485 add verbose message 2021-08-13 02:57:14 -07:00
Chris Lu 057ef429ac format 2021-08-10 12:33:29 -07:00
Chris Lu b63b042afc dedup keeps the largest replica 2021-08-10 12:30:41 -07:00
Chris Lu 0526db12e2 do not treat read only volumes differently 2021-08-10 11:37:12 -07:00
Chris Lu e50a5b8e28 minor: print disk type 2021-08-10 11:10:09 -07:00
Chris Lu db6275a0c8 print out balance ratio 2021-08-10 10:43:42 -07:00
Chris Lu 85832d02c0 wait for goroutines 2021-08-10 04:13:12 -07:00
Chris Lu 8ff6c9a0c6 output format 2021-08-10 03:25:18 -07:00
Chris Lu 48f448ee09 parallelize tier move 2021-08-10 03:08:29 -07:00
Chris Lu 69a6da7969 avoid fail on tail error 2021-08-10 02:50:28 -07:00
Chris Lu 18228f3044 fix help message 2021-08-10 02:48:41 -07:00
Chris Lu 02f728cb54 fix bug with remote.uncache 2021-08-09 15:24:21 -07:00
Chris Lu 402315f117 go fmt 2021-08-09 14:37:34 -07:00
Chris Lu 713c035a6e shell: remote.cache remote.uncache 2021-08-09 14:35:18 -07:00
Chris Lu c5f38c365d go fmt 2021-08-08 22:30:36 -07:00
Chris Lu df85f7a1eb adjust help message 2021-08-08 22:30:12 -07:00
Chris Lu c0b12da4ef shell: add filer.remote.unmount 2021-08-08 22:26:37 -07:00
Chris Lu 882a93dacd fix tests 2021-08-08 17:56:26 -07:00
Chris Lu eed26af266 Merge branch 'master' into add_remote_storage 2021-08-08 15:48:04 -07:00
Chris Lu 26c222f596 shell: volume.tier.move avoid moving all volumes to one destination 2021-08-08 15:12:39 -07:00
Chris Lu 13e45e1605 filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
Chris Lu 679f800caa Merge branch 'master' into add_remote_storage 2021-08-06 20:27:12 -07:00
Chris Lu 0c0f77e2ae skip not found error on deletion 2021-08-06 19:35:47 -07:00
Chris Lu 8828f485c0 print volume deletion error 2021-08-06 19:30:22 -07:00
Chris Lu e23ca3845b
Merge pull request #2237 from combineads/add_mtime 
Add crtime and mtime
 2021-08-05 21:27:13 -07:00
Chris Lu 286e5dd375 Merge branch 'master' into add_remote_storage 2021-08-05 21:07:04 -07:00
Chris Lu 1e22166939 adjust error message 2021-08-05 21:06:55 -07:00
byunghwa.yun e84fad9acf Add crtime and mtime 2021-08-06 12:13:37 +09:00
Chris Lu 1a5d29520c add tests 2021-08-05 14:49:24 -07:00
Chris Lu 6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu b9ecf1e3a8 refacotring 2021-08-04 14:56:13 -07:00
Chris Lu d84c311699 refactoring 2021-08-04 12:30:18 -07:00
Chris Lu f6a9ad8001 fix tests 2021-08-04 00:31:06 -07:00
Chris Lu 42969c9c62 Merge branch 'master' into add_remote_storage 2021-08-03 21:20:05 -07:00
Chris Lu 8cf0c515bf shell: volume.fix.replication retries even when there is no error #2235 
fix https://github.com/chrislusf/seaweedfs/issues/2235
 2021-08-03 09:32:55 -07:00
Chris Lu 2ae9705442 adjust text 2021-08-01 22:55:19 -07:00
Chris Lu 3739717092 Revert "adds a test" 
This reverts commit f690643b47.
 2021-08-01 22:54:45 -07:00
Chris Lu f690643b47 adds a test 2021-08-01 22:53:50 -07:00
Chris Lu 6de786185d volume.balance: balance read only volumes first 2021-08-01 22:32:50 -07:00
byunghwa.yun 9e839cb5cc Add force option in volume.deleteEmpty command 2021-08-02 11:37:20 +09:00
Chris Lu 81b255df8b Merge branch 'master' into add_remote_storage 2021-08-01 15:44:21 -07:00
Chris Lu 2ca1839d77 shell: add volume.deleteEmpty command 2021-08-01 15:33:45 -07:00
Chris Lu 9df7d16791 read <- remote_storage 2021-07-31 22:39:38 -07:00
Chris Lu f5a69a0e44 Merge branch 'master' into add_remote_storage 2021-07-31 09:39:48 -07:00
byunghwa.yun 5d4438a72f Fix typo 2021-07-31 22:23:52 +09:00
Chris Lu a3290faf17 shell command to calculate size / number of volumes in a collection 
fix https://github.com/chrislusf/seaweedfs/issues/2224
 2021-07-30 15:49:46 -07:00
Chris Lu c6f992b2a3 remove dead code 2021-07-30 15:18:01 -07:00
Chris Lu 899963ac20 remote storage location changed to struct 2021-07-29 02:08:55 -07:00
Chris Lu 4deac06da0 log fix 2021-07-28 22:25:04 -07:00
divanikus 5c6270a93a shell: ability to use wildcards for collections, all collections if ommited 2021-07-27 23:53:01 +03:00
Chris Lu 1752eeb538 remote.mount saves the mapping 2021-07-27 03:26:35 -07:00
Chris Lu 4b94b03d90 directory to remote storage mapping 2021-07-27 01:16:28 -07:00
Chris Lu 99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00
Chris Lu 35f70c51b0 refactor 2021-07-25 03:37:37 -07:00
Chris Lu 72eec84167 shell: add fs.mkdir 2021-07-25 02:37:39 -07:00
Chris Lu 092932af8f shell: faster bucket delete 
avoid waiting for large object metadata
 2021-07-23 11:25:46 -07:00
Chris Lu 7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu 67537ff05b
Merge pull request #2208 from combineads/add_rm_options 2021-07-21 11:08:43 -07:00
byunghwa.yun 41d559f54e Add fs rm options 2021-07-21 21:10:36 +09:00
Chris Lu 70effac0d3 configure and store remote configurations 2021-07-21 02:24:34 -07:00
Chris Lu 3138805b33
Merge pull request #2200 from combineads/add_fs_rm 
weed shell: fs.rm remove file or directory
 2021-07-21 00:35:54 -07:00
Chris Lu 7ab389e7ec optimization: improve random range query for large files 2021-07-19 23:07:22 -07:00
byunghwa.yun b23b307e08 Add fs rm 2021-07-20 08:49:28 +09:00
Chris Lu 99155e4880 add -retry option for volume.fix.replication 
fix https://github.com/chrislusf/seaweedfs/issues/2191
 2021-07-16 12:13:46 -07:00
Chris Lu 2f209675ab Added -retry option for volumeServer.evacuate 
related to https://github.com/chrislusf/seaweedfs/issues/2191
 2021-07-16 12:08:21 -07:00
Chris Lu fb7a1be1c4 refactor 2021-07-16 11:47:37 -07:00
Chris Lu 4be5b4ff5c [volume.check.disk] skip errors and check next volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2129
 2021-07-16 02:03:32 -07:00
Chris Lu 5a838dbe53 fix compilation 2021-07-13 11:40:21 -07:00
Chris Lu 6103649ffb shell: volume.check.disk adds retries in case the volumes are just moved 
related to https://github.com/chrislusf/seaweedfs/issues/2194
 2021-07-13 11:19:56 -07:00
Chris Lu 01adc567aa shell: volume.fsck deletes a volume is the volume has only orphaned data. 
fix https://github.com/chrislusf/seaweedfs/issues/2190
 2021-07-12 11:22:00 -07:00
bingoohuang 9e48bff8ee fix typo in fs.configure help message text 2021-07-12 13:51:21 +08:00
Chris Lu d013d6d968 shell: volume.fsck "reallyDeleteFromVolume" should send padded file ids to delete 
fix https://github.com/chrislusf/seaweedfs/issues/2188
 2021-07-10 23:16:06 -07:00
Chris Lu 3d624d1e16 rename 2021-07-09 03:19:21 -07:00
Chris Lu da7bd62822 /etc files are stored inside metadata store 2021-07-09 02:33:14 -07:00
Chris Lu 08377fecb8 fixes 2021-06-27 23:32:57 -07:00
Chris Lu 1cac2f2278 shell: volume.fsck checks missing chunks in filer 
fix https://github.com/chrislusf/seaweedfs/issues/2154
 2021-06-24 23:56:24 -07:00
Chris Lu 9dd09bbb33 refactor 2021-06-24 17:22:53 -07:00
Daniel Nagy 7ce3bee0c0
Check for history file errors as well 2021-06-08 11:14:49 +02:00
Chris Lu ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu 615cb24ba6 shell: use tab print style for completion 2021-05-31 03:29:29 -07:00
Chris Lu b70aeb7585 bucket list only directories 2021-05-23 10:36:22 -07:00
Chris Lu 30c67e3652 minor 2021-05-19 23:59:39 -07:00
Chris Lu 76c48ffe27 optional parallel copy ec shards 
fix https://github.com/chrislusf/seaweedfs/issues/2048
 2021-05-06 01:53:35 -07:00
Chris Lu ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu f0ad172e80 shell: show which server holds the lock 
fix https://github.com/chrislusf/seaweedfs/issues/1983
 2021-04-22 23:56:35 -07:00
Chris Lu 46ef1811a1 correct help message 2021-04-22 22:26:38 -07:00
Chris Lu b971317a16 avoid possible corrupted file names 2021-04-15 11:41:34 -07:00
Chris Lu 609e228578 avoid forward slash in file names 2021-04-15 10:53:04 -07:00
Chris Lu ba92f2e714 add node.selectedVolumes 
fix https://github.com/chrislusf/seaweedfs/issues/1990
 2021-04-15 03:19:28 -07:00
Chris Lu fc0cbf565f add option to obfuscate the file names 2021-04-14 15:37:24 -07:00
Chris Lu e75633c64f volume.check.disk: break loop for read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2002
 2021-04-14 12:40:13 -07:00
Chris Lu 9d50867d08 volume.tier.move: avoid data loss when destination volume server already has the volume 
fix https://github.com/chrislusf/seaweedfs/issues/2001
 2021-04-14 10:26:26 -07:00
Chris Lu 90677e1097 ensure to call line.Close() 
fix https://github.com/chrislusf/seaweedfs/issues/1995

similar to https://github.com/peterh/liner/issues/104
 2021-04-14 01:33:21 -07:00
Konstantin Lebedev 020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00