gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6511 commits 28 branches 278 tags 135 MiB
Commit graph

4270 commits

Author SHA1 Message Date
Chris Lu 94a01fcfcb filer.remote.gateway: add options to include or exclude new bucket names to mirror 2021-09-15 23:04:16 -07:00
Chris Lu b5f4910412 adjust help messages 2021-09-15 22:53:10 -07:00
Chris Lu f0907eb83c add command filre.remote.gateway 2021-09-15 22:48:04 -07:00
Chris Lu 63da4bbb54 separate filer.remote.gateway command to avoid confusion 2021-09-15 22:47:17 -07:00
Chris Lu 2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu 119d5908dd shell: do not need to lock to see volume -h 2021-09-13 22:13:34 -07:00
Chris Lu 7504be58f9 Avoid xattr printed out as HTTP headers 
fix https://github.com/chrislusf/seaweedfs/issues/2336
 2021-09-13 04:00:57 -07:00
Chris Lu 96514f0f00 fix tests 2021-09-13 02:19:48 -07:00
Chris Lu 20ac710ceb 2.68 2021-09-13 02:16:09 -07:00
Chris Lu 6cd1ce8b74 erasure coding: add cleanup step if anything goes wrong 2021-09-13 01:55:49 -07:00
Chris Lu f74b29416a better etag matching 2021-09-13 00:31:46 -07:00
Chris Lu e9760f261e minor 2021-09-13 00:31:06 -07:00
Chris Lu e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu 2c9d4c8f43 custom grpc port: filer 2021-09-12 02:28:37 -07:00
Chris Lu e690a2be16 custom grpc port: volume server 2021-09-12 02:25:15 -07:00
Chris Lu 232ad2fe65 custom grpc port: master follower 2021-09-12 02:25:00 -07:00
Chris Lu bd1f800efe custom grpc port: master 2021-09-12 02:19:10 -07:00
Chris Lu 7591336a22 log format 2021-09-11 14:27:57 -07:00
Chris Lu 8c6ff55226 add volume not found error type, to reduce error log 2021-09-11 14:26:41 -07:00
Chris Lu 7e600bff5f adjust log level 2021-09-11 12:59:15 -07:00
Chris Lu a243d7e047 EC: clean up if failed in the middle 2021-09-11 02:07:14 -07:00
Chris Lu f7f01ac209 return if error 2021-09-11 02:06:43 -07:00
Chris Lu 59cb6146a3 refactor 2021-09-11 02:06:30 -07:00
Chris Lu 5496d68f6a increase counter only if not early terminated 2021-09-11 02:05:55 -07:00
Chris Lu 2b8ab1534a mysql table DDL: minor, not necessary, but a little more strict 
the directory is already hashed into dirhash. So this change is not strictly required.
 2021-09-08 23:55:18 -07:00
joshuafc 5654d0d60d CompactionTableSizeMultiplier of leveldb use default value. #2325 
To improve performance of leveldb find key in condition of large directory(millions of files) which use uuid as filename.
 2021-09-09 10:42:34 +08:00
Chris Lu 0b4269b6a8 remember commands even if failed 2021-09-08 15:55:19 -07:00
Chris Lu 0207f5fe9b replicated remote.cache 2021-09-08 15:54:55 -07:00
Chris Lu 00edcf71fc fix compilation 2021-09-08 07:27:03 -07:00
Chris Lu 574485ec69 better IP v6 support 2021-09-07 19:29:42 -07:00
Chris Lu 0128239c0f handle ipv6 addresses 2021-09-07 16:43:54 -07:00
zhoub 6a7ed1bd0e add bloom filter to leveldb_store to improve fuse performance. 2021-09-07 21:09:10 +08:00
Chris Lu 9fdf02bcda remove detecting ipv6 
Got this error on my local:

transport: Error while dialing dial tcp [fe80::1]:19333: connect: no route to host

related to https://github.com/chrislusf/seaweedfs/pull/2310
 2021-09-07 02:31:34 -07:00
Chris Lu 889a39a24f ipv6 2021-09-07 02:30:25 -07:00
Chris Lu 8c6d706328 2.67 2021-09-07 00:08:03 -07:00
Chris Lu 2b1feb732c remote.cache supports replication 2021-09-06 18:30:44 -07:00
Chris Lu 3adc3da291 refactor 2021-09-06 16:35:55 -07:00
Chris Lu 6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Chris Lu 64f6532fbe update help message 2021-09-06 16:16:22 -07:00
Chris Lu 308d4b2dae filer.remote.sync: createBucketWithRandomSuffix defaults to true 2021-09-06 15:14:26 -07:00
Chris Lu 8e4b43a017 minor 2021-09-06 15:13:38 -07:00
Chris Lu c218ef20c7 filer.remote.sync: automatically detect the primary remote storage 2021-09-06 15:10:55 -07:00
Chris Lu 1702ce5395 remove unused code 2021-09-06 12:01:44 -07:00
Chris Lu 0cda61d539 return error if not found 2021-09-05 23:43:44 -07:00
Chris Lu f62df7f695 fix wrong srcPath checking 2021-09-05 23:37:40 -07:00
Chris Lu e93d4935e3 add other replica locations when assigning volumes 2021-09-05 23:32:25 -07:00
Chris Lu 7a13816e94 refactor 2021-09-05 23:17:15 -07:00
Chris Lu 6022db6d6a 2.66 2021-09-05 16:21:14 -07:00
Chris Lu 006c01a519 fix format 2021-09-05 16:18:50 -07:00
Chris Lu 65af3cf4df master: disconnect only the phantom volume server 
fix https://github.com/chrislusf/seaweedfs/issues/2311
 2021-09-05 15:20:03 -07:00
Chris Lu 57a95887d2 remote.cache remote.uncache supports all mounted directories 2021-09-05 14:47:06 -07:00
Chris Lu 28f45f8fa6 re-org 2021-09-05 14:38:31 -07:00
Chris Lu c735608685 obfuscate secret keys on display 2021-09-05 14:23:49 -07:00
Chris Lu 32e94de86a default auto trimming suffix to true 2021-09-05 13:27:35 -07:00
Chris Lu 60573fd3e2 option to map remote bucket to trimmed bucket name 2021-09-05 11:55:52 -07:00
Chris Lu 2348e8d8da
Merge pull request #2310 from nivekuil/ipv6 
Detect ipv6 addresses
 2021-09-05 10:56:44 -07:00
nivekuil 0fe9d2997b Detect ipv6 addresses 2021-09-05 06:21:40 -07:00
Chris Lu bdefdee4e6 filer.remote.sync: add option to add randomized suffix to buckets to avoid conflicts 2021-09-04 22:46:28 -07:00
Chris Lu d57d4c5f8f shell: add remote.mount.buckets 2021-09-04 21:37:25 -07:00
Chris Lu 98c68d7a79 filer.remote.sync fix upload logic 2021-09-04 18:46:28 -07:00
Chris Lu d983aa4c7d correct filtering 2021-09-04 13:58:14 -07:00
Chris Lu c5ee03d6af format 2021-09-04 13:57:55 -07:00
Chris Lu cc278a5928 Update Makefile 2021-09-04 13:46:49 -07:00
Chris Lu 16a342eb8a re-ordering 2021-09-04 13:46:44 -07:00
Chris Lu 796b2eb929 remember sync time 2021-09-04 13:46:22 -07:00
Chris Lu 53b9b521c9 adjust error message 2021-09-04 13:46:06 -07:00
Chris Lu 49952ba905 fix cache/uncache filters 2021-09-04 13:45:39 -07:00
Chris Lu 99d4594ee8 update remote storage mount mapping 2021-09-04 05:22:34 -07:00
Chris Lu df29281536 rename 2021-09-04 05:11:06 -07:00
Chris Lu c3db389e42 refactor 2021-09-04 05:09:38 -07:00
Chris Lu e9ce099453 rename 2021-09-04 05:03:51 -07:00
Chris Lu 8707ef00a6 refactor 2021-09-04 05:02:20 -07:00
Chris Lu 13cc2ef723 when creating bucket, use remote storage client for default storage 2021-09-04 04:54:55 -07:00
Chris Lu 63c84584bb refactor 2021-09-04 04:53:36 -07:00
Chris Lu 4b28c5f6c3 filer.remote.sync: split into buckets mode and single directory mode 2021-09-04 04:35:46 -07:00
Chris Lu da49d25950 auto list of storage types 2021-09-04 00:18:21 -07:00
Chris Lu 38f73c8324 add gcs project id 2021-09-03 23:38:33 -07:00
Chris Lu 8ec357b3d3 go mod 2021-09-03 23:25:33 -07:00
Chris Lu bbc77f7af4 fix compilation 2021-09-03 22:56:59 -07:00
Chris Lu 0652805236 cloud drive: add createBucket() deleteBucket() 2021-09-03 22:30:55 -07:00
Chris Lu 03a31587ce go fmt 2021-09-03 20:42:28 -07:00
Chris Lu 83cd0fc739 cloud drive: add list buckets 2021-09-03 20:42:02 -07:00
Chris Lu fbfc90fd1e adjust formatting remote location 2021-09-03 18:52:37 -07:00
Chris Lu 491f7636f8 s3: CopyObject return http Status 400 Bad Request for non-existing source 
fix https://github.com/chrislusf/seaweedfs/issues/2306
 2021-09-03 17:38:39 -07:00
Chris Lu bca4a9de78 simplify 2021-09-02 23:09:24 -07:00
Chris Lu 958125bd02 conforming to http user agent common practice 2021-09-02 22:55:35 -07:00
Chris Lu 11a496404b reset wait time 2021-09-02 19:55:01 -07:00
Chris Lu 7f2a8246a9 minor 2021-09-02 12:32:59 -07:00
Chris Lu 1b942dfa3c add error message 2021-09-02 12:32:46 -07:00
Chris Lu 9a73b0e3c9 refactor 2021-09-02 07:07:16 -07:00
Chris Lu e281f0fa82 refactor 2021-09-02 06:53:21 -07:00
Chris Lu 24fda6f47f remove QR code for mobile upload 2021-09-01 16:59:37 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu d1a4e19a3f volume: copy file also copies modification time 
to ensure ttl can work well
 2021-09-01 02:42:57 -07:00
Chris Lu 3bd48c4f29 filer.remote.sync: exit when directory is unmounted 
this will not propagate the deletions back to the cloud
 2021-09-01 01:29:22 -07:00
Chris Lu 3faaa6e360 ensure cached client with updated storage conf 2021-09-01 01:27:45 -07:00
Chris Lu 8e125339d5 line wrap 2021-08-31 23:30:28 -07:00
Chris Lu 68ea99190f add additionalPathPrefixes 2021-08-31 23:26:26 -07:00
Chris Lu 43fd11278e support follow additional path prefixes 2021-08-31 23:23:08 -07:00
Chris Lu cca588429b shell: fs.meta.save skip saving system logs 2021-08-31 23:05:56 -07:00
Chris Lu 0ee208c1a4 adjust example 2021-08-31 02:04:32 -07:00
Chris Lu 97b7948cd9 adjust error message 2021-08-31 01:07:34 -07:00
Chris Lu c9205458a1 cloud drive: create mount directory if not exists 2021-08-31 01:07:19 -07:00
Chris Lu 1dfcdc53b7 cloud drive: add storj, filebase 2021-08-30 17:43:14 -07:00
Chris Lu 701cc133f3 cloud drive: s3 add option for v4 signature 2021-08-30 17:28:33 -07:00
Chris Lu f117f7bf54 adjust comment 2021-08-30 15:56:14 -07:00
Chris Lu 72d4ff6b28
Merge pull request #2280 from blacktear23/filerstore-tikv 
Use TiKV as Filerstore
 2021-08-30 02:21:14 -07:00
Chris Lu f811fd0903 fix build 2021-08-30 02:03:08 -07:00
yulai.li b17b81529e Add build tags for TiKV filerstore 2021-08-30 15:59:25 +08:00
Rain Li e5f9ff983d
Merge branch 'master' into filerstore-tikv 2021-08-30 15:52:54 +08:00
Chris Lu 44d0ac95f6 add cluster id if already set 2021-08-29 22:19:46 -07:00
Chris Lu 18bf556d93 fix comments 2021-08-29 22:19:25 -07:00
Chris Lu 21f37a0a75 remove unused code 2021-08-29 22:19:11 -07:00
Chris Lu 5adfdd7982 change to a new bloom filter library 2021-08-29 21:26:38 -07:00
Chris Lu 05f32376eb add cluster id in filer configuration response 2021-08-29 21:02:10 -07:00
Chris Lu eacaa44dc2 refactor 2021-08-29 19:13:48 -07:00
Chris Lu 6deee4c0b9 import hdfs 2021-08-29 18:49:43 -07:00
Chris Lu a31f2907f0 cloud drive: filer.remote.sync supports remove folder 2021-08-29 18:46:28 -07:00
Chris Lu 001a472057 cloud mount: remote storage support hdfs 2021-08-29 18:41:29 -07:00
Chris Lu 05d2774117 refactoring 2021-08-28 22:49:38 -07:00
Chris Lu 975fbc9ee3 add wasabi example 2021-08-28 22:49:38 -07:00
Chris Lu bec3f63298 2.65 2021-08-28 05:27:33 -07:00
Chris Lu 223c1b3571 adjust invalidation logic 2021-08-28 05:21:01 -07:00
nivekuil 8c523d50ad Invalidate mount meta cache more aggressively 2021-08-28 04:26:37 -07:00
Chris Lu 65a29ad42a add warning on unmount a folder 2021-08-28 02:25:48 -07:00
Chris Lu a22f37b01c remove tencent region 2021-08-28 02:23:03 -07:00
Chris Lu a9b41900d8 handle cases when mounted directory does not exist 2021-08-28 01:56:56 -07:00
Chris Lu 0d8936f6cd add aliyun example 2021-08-28 01:09:48 -07:00
Chris Lu dc481c081c remove unused function 2021-08-27 22:18:54 -07:00
Chris Lu 9242f3aaeb fix build 2021-08-26 17:33:57 -07:00
Chris Lu 49a8dfb976 adjust default concurrent level 2021-08-26 17:05:56 -07:00
Chris Lu 6a0bb7106b cloud drive: parallelize remote storage downloading 2021-08-26 16:16:26 -07:00
Chris Lu 05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
yulai.li c1dc5ab4ac Add deleterange_concurrency to filer configuration file 2021-08-26 18:25:08 +08:00
yulai.li 318757ef8c Change DeleteFolderChildren to DeleteRange api 2021-08-26 17:49:56 +08:00
yulai.li de8ef28460 Update filer.toml scaffold to support tikv store 2021-08-26 16:26:27 +08:00
yulai.li 546efeba8f Fix build bug 2021-08-26 16:20:35 +08:00
yulai.li 2088f28424 init post 2021-08-26 15:20:18 +08:00
Chris Lu c08ac536ed cloud drive: add support for Wasabi 
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
 2021-08-25 17:34:29 -07:00
Chris Lu 9bcf94b2b1 ensure multi-threaded correctness 2021-08-25 17:28:50 -07:00
Chris Lu a7a914f120 cloud drive: add support for Baidu BOS 2021-08-24 23:46:33 -07:00
Chris Lu 2d805beed4 add help mesages 2021-08-24 23:28:35 -07:00
Chris Lu c6c97bd83d add default env variables 2021-08-24 23:25:36 -07:00
Chris Lu a19c728034 cloud drive: add support for Tencent COS 2021-08-24 23:19:45 -07:00
Chris Lu 19a81d25af cloud drive: add support for Aliyun OSS 2021-08-24 23:14:24 -07:00
Chris Lu 47d775cf68 cloud drive: add support for BackBlaze 2021-08-24 22:30:06 -07:00
Chris Lu 6bab20d862 cat entry content 2021-08-24 02:38:32 -07:00
Chris Lu e9ebe24f2e cloud drive: add support for Azure 2021-08-24 01:18:30 -07:00
Chris Lu 7c39a18ba5 update azure library 2021-08-24 00:32:35 -07:00
Chris Lu a78d0227cd adjust package name 2021-08-23 23:19:31 -07:00