gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
5901 commits 28 branches 278 tags 135 MiB
Commit graph

3816 commits

Author SHA1 Message Date
Chris Lu 19caeb7b02 fix writing the small file 2021-06-06 20:57:03 -07:00
Chris Lu bb45dea15a filer: parallel data upload 2021-06-06 20:23:36 -07:00
Chris Lu e00443a940 mount: adjust starting order 
avoid possible nil wfs.Server
 2021-06-06 20:22:42 -07:00
Chris Lu 44f1ba6894 refactor 2021-06-06 18:43:04 -07:00
Chris Lu 6c82326575 use bytes.Buffer to reduce memory allocation and gc 2021-06-06 13:42:36 -07:00
Chris Lu 9cba5cca0b optionally disable concurrent upload limit 2021-06-06 13:13:33 -07:00
Chris Lu 21ad9a4ac2 filer: mongodb avoids E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key 2021-06-06 13:12:01 -07:00
Chris Lu e6ba2f9c37 verbose but cleaner logic to invalidate fuse cache 2021-06-05 13:09:37 -07:00
Chris Lu fadc1febdb FUSE: invalidate FUSE cached entries 
fix https://github.com/chrislusf/seaweedfs/issues/2108
 2021-06-05 02:23:07 -07:00
Chris Lu ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu 0a5388744c
Merge pull request #2105 from Woellchen/current_and_parent_dot_directories 
Return artificial . and .. directories
 2021-06-02 12:54:44 -07:00
Patrick Schmidt 77100754e6 Return artificial . and .. directories 2021-06-02 21:28:02 +02:00
Chris Lu 62142ff1d2
Merge pull request #2104 from kmlebedev/sync_volume_on_close 
avoid data loss after restarting a container with a volum server
 2021-06-02 11:47:57 -07:00
Chris Lu cc34475012 remove file handle locking for setattr 2021-06-02 11:44:12 -07:00
Konstantin Lebedev 99ef280c7c avoid data loss after restarting a container with a volum server 2021-06-02 17:07:19 +05:00
Chris Lu 3db1642392 mount: skip persisting metadata if file is open 2021-06-02 03:40:08 -07:00
Chris Lu 556cc3a4ca mount: avoid exception if disk cache is not initialized 
related to https://github.com/chrislusf/seaweedfs/issues/2102
 2021-05-31 16:42:55 -07:00
Chris Lu 615cb24ba6 shell: use tab print style for completion 2021-05-31 03:29:29 -07:00
Chris Lu c5de97ddbe keep alive for streaming connections 
fix https://github.com/chrislusf/seaweedfs/issues/2096

related to https://github.com/chrislusf/seaweedfs/issues/1277
 2021-05-31 01:26:24 -07:00
Chris Lu 2b60e2abb1 only disallow streaming signed when no auth enabled 
fix https://github.com/chrislusf/seaweedfs/issues/2101
 2021-05-31 01:03:04 -07:00
Chris Lu 4233ad3f07 2.50 2021-05-30 20:40:30 -07:00
Chris Lu 1456616a77 recreate grpc connections if too many errors 
address https://github.com/chrislusf/seaweedfs/issues/2098
 2021-05-30 00:07:43 -07:00
Chris Lu fb8036385a s3: save metadata during put-object 
fix https://github.com/chrislusf/seaweedfs/issues/2092
 2021-05-29 14:14:30 -07:00
Chris Lu 2d7b4e5bb6 filer.backup: escape colon from path on windows 
fix https://github.com/chrislusf/seaweedfs/issues/2084
 2021-05-29 06:45:27 -07:00
Chris Lu ae185b997f
Merge pull request #2099 from danielflira/mount-helper 
fix parameter multiple values
 2021-05-28 21:57:15 -07:00
danielflira 849f36c1ac fix parameter multiple values 2021-05-29 00:37:25 -03:00
Eugeniy Kozlov 6e8bd16819 freespace params fix 2021-05-28 14:19:24 +03:00
Chris Lu 4d55132c7d scaffold add comments 2021-05-27 14:40:25 -07:00
Chris Lu 45bffc92a8 filer.backup: fix cloud sinks when updating entry 2021-05-26 14:53:11 -07:00
Chris Lu 921e0d5008 remove verbose log 2021-05-26 14:43:34 -07:00
Chris Lu 1a70cb9b63 local sink: write and update files 
fix issues with https://github.com/chrislusf/seaweedfs/issues/2084
 2021-05-26 14:42:21 -07:00
Chris Lu e699d16c85
Merge pull request #2085 from danielflira/mount-helper 
create fuse subcommand to use weed with mount
 2021-05-26 12:19:40 -07:00
danielflira 84488ebb33 replace filer.remote and parts[1] 2021-05-26 12:07:36 -03:00
Chris Lu ac9bf71544 use jquery 3.6.0 2021-05-26 00:21:11 -07:00
danielflira 877c192af2 create fuse subcommand to use weed with mount 2021-05-25 23:32:35 -03:00
Chris Lu 67dd094b35 sqlite: does not support non-linux/darwin/windows 2021-05-25 17:36:15 -07:00
Chris Lu dce1f02c9e filer.backup: backup to local directory optionally is incremental 
fixed one issue with https://github.com/chrislusf/seaweedfs/issues/2084
 2021-05-25 17:19:20 -07:00
Konstantin Lebedev b612d5aebd s3 test get w/ If-Match: bogus ETag 2021-05-24 16:59:44 +05:00
Konstantin Lebedev 69c768870b - object write cache control 
- object write expires
 2021-05-24 15:43:55 +05:00
Konstantin Lebedev 84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Konstantin Lebedev 0e404cd84b Chunk download stats 2021-05-24 12:14:50 +05:00
Konstantin Lebedev bb42633522 rm chunk Download stats 2021-05-24 12:08:12 +05:00
Chris Lu 064269bb57 filer: Support sqlite as filer meta store 2021-05-23 23:58:28 -07:00
Chris Lu b70aeb7585 bucket list only directories 2021-05-23 10:36:22 -07:00
Chris Lu 42fb03a66e 2.49 2021-05-23 00:51:47 -07:00
Chris Lu 3d3fa43542 filer: re-create grpc connections if having transport error 
fix https://github.com/chrislusf/seaweedfs/issues/2070
 2021-05-22 17:46:53 -07:00
Chris Lu 2d0a0733ec filer: bootstrap from peer filer 
fix https://github.com/chrislusf/seaweedfs/issues/1861
 2021-05-22 03:17:02 -07:00
Chris Lu e5a2bf1287 s3: deprecating filer.options.buckets_fsync 2021-05-22 00:24:23 -07:00
Chris Lu 431684798b s3: add errors if requests are signed by no authentication is setup 
fix https://github.com/chrislusf/seaweedfs/issues/2075
 2021-05-21 14:08:47 -07:00
Chris Lu 2f136a04a1 organize a bit better for temp file directory 2021-05-21 13:02:18 -07:00
Konstantin Lebedev 3325b850cc rm func CheckAllChunkViews 2021-05-21 16:05:59 +05:00
Konstantin Lebedev 26a4f34a57 del checks 
url err logging
stats chunk fetch
 2021-05-21 15:59:12 +05:00
Chris Lu e8b7d4ad64 simplify a bit 2021-05-21 02:10:44 -07:00
Chris Lu 0b00edfc3b waits in case master lost connection 2021-05-21 02:09:22 -07:00
Chris Lu f37a4cbd0f randomize initial filer 2021-05-21 01:41:34 -07:00
Chris Lu 5d77840cff adjust help message 2021-05-21 01:38:57 -07:00
Chris Lu dc1309f084 FUSE mount: support multiple filers 
fix https://github.com/chrislusf/seaweedfs/issues/2015
fix https://github.com/chrislusf/seaweedfs/issues/1531
 2021-05-21 01:28:00 -07:00
Konstantin Lebedev 42f631f549 fix https://github.com/chrislusf/seaweedfs/issues/2046 2021-05-20 13:19:35 +05:00
Chris Lu 30c67e3652 minor 2021-05-19 23:59:39 -07:00
Konstantin Lebedev 03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" 
This reverts commit ac71117e
 2021-05-20 11:45:21 +05:00
Chris Lu 87a32bfef4 avoid possible nil when node is disconnected from its parent 
fix https://github.com/chrislusf/seaweedfs/issues/2073
 2021-05-19 10:02:01 -07:00
Chris Lu 1aa7e99a89 skip file not found error when deleting 2021-05-15 09:37:39 -07:00
Chris Lu 45a762223a 2.48 2021-05-14 10:26:42 -07:00
Chris Lu 789465d99e set renamed item to new directory 
related to https://github.com/chrislusf/seaweedfs/issues/2064
 2021-05-13 22:52:19 -07:00
Chris Lu 83c037e093 fix logs 2021-05-13 22:41:19 -07:00
Chris Lu 5a29d284fa ensure proper directory name 
a fix related to https://github.com/chrislusf/seaweedfs/issues/2064
 2021-05-13 10:18:35 -07:00
Chris Lu 3ff307e842 reduce logs 2021-05-13 00:32:37 -07:00
Chris Lu a48ebd7c73 mount: read file when file is still being written 
a possible fix for https://github.com/chrislusf/seaweedfs/issues/2065
 2021-05-13 00:32:23 -07:00
Chris Lu 26a55bbb5c Adjust error message when bucket name conflicts with existing collections 
fix https://github.com/chrislusf/seaweedfs/issues/2069
 2021-05-12 22:30:39 -07:00
Chris Lu ceb620a30a directory rename: change directory name after renaming 
fix https://github.com/chrislusf/seaweedfs/issues/2068
 2021-05-12 22:04:47 -07:00
Chris Lu b430d1b6ee filer.copy: "check.size" before copying files 
fix https://github.com/chrislusf/seaweedfs/issues/2067
 2021-05-12 21:45:39 -07:00
Chris Lu 0f7b43af99 2.47 2021-05-11 10:12:15 -07:00
Chris Lu d2d36a3f9d master: avoid creating too many volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2062
 2021-05-11 10:05:31 -07:00
Chris Lu 4596e64710 2.46 2021-05-10 21:58:37 -07:00
Chris Lu e55aa41690 Merge branch 'upload_via_temp_file' 2021-05-10 21:48:18 -07:00
Chris Lu 1737af480a adjust logs 2021-05-10 21:47:51 -07:00
Chris Lu d06ecc2649 working properly 2021-05-10 21:47:07 -07:00
Chris Lu fe2edd4b50 2.45 2021-05-10 13:53:03 -07:00
Chris Lu 55d547afec switch back to in memory uploading for now 2021-05-10 13:50:10 -07:00
Chris Lu 9a6aa00e9d avoid nil locations 
fix https://github.com/chrislusf/seaweedfs/issues/2059
 2021-05-10 02:39:52 -07:00
Chris Lu da0a4e775b Revert "Revert "Merge pull request #2027 from bingoohuang/master"" 
This reverts commit d74cdf0115.
 2021-05-10 00:13:55 -07:00
Chris Lu aa6949ef25 2.44 2021-05-09 23:28:48 -07:00
Chris Lu ba77833343 EOF handling 2021-05-09 23:04:24 -07:00
Chris Lu f4cf30b497 mount: write to tmp file before uploading 2021-05-09 22:56:10 -07:00
Chris Lu 280e7cf289 mount: in case the set attribute is called before persisting the file 2021-05-09 22:55:30 -07:00
Chris Lu 74052064b6 add one missing optimization 2021-05-09 17:25:42 -07:00
Chris Lu 59ace54925 refactor 2021-05-09 17:22:30 -07:00
Chris Lu 50be19d23e refactor 2021-05-09 15:33:01 -07:00
Chris Lu 55e060cf61 refactor 2021-05-09 15:28:54 -07:00
Chris Lu 735e65be17 refactor 2021-05-09 15:22:38 -07:00
Chris Lu 93e84a12f2 refactor 2021-05-09 15:15:18 -07:00
Chris Lu 3942e3b2ef a better fix 2021-05-09 01:42:19 -07:00
Chris Lu 957e1a1bc1 fuse: important: if filer -filer.path is not root, directory listing will fail 2021-05-09 00:32:21 -07:00
Chris Lu 7ca75347ec minor 2021-05-07 21:56:45 -07:00
Chris Lu 8f8738867f add retry to assign volume 
fix https://github.com/chrislusf/seaweedfs/issues/2056
 2021-05-07 07:29:26 -07:00
Chris Lu 007401f3a0 remove duplicated code 2021-05-07 07:14:24 -07:00
Chris Lu aaad4b578b
Merge pull request #2049 from qieqieplus/async-assign 
ahead of time volume assignment
 2021-05-06 10:00:25 -07:00
qieqieplus c4d32f6937 ahead of time volume assignment 2021-05-06 18:55:44 +08:00
Chris Lu 55a8f57381 go fmt 2021-05-06 03:37:51 -07:00
Chris Lu 5753749c90 remove verbose logs 2021-05-06 03:34:34 -07:00
Chris Lu 38f411219a mount: skip local chunk cache if opened write only 2021-05-06 03:31:40 -07:00
Chris Lu c899bdf063 a little optimization 2021-05-06 03:03:00 -07:00
Chris Lu 3eb336e0b0 report error only for the first multipart upload 
the glog.Errorf would always print for s3 multipart uploads
 2021-05-06 01:57:54 -07:00
Chris Lu 76c48ffe27 optional parallel copy ec shards 
fix https://github.com/chrislusf/seaweedfs/issues/2048
 2021-05-06 01:53:35 -07:00
Chris Lu ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu 24efa31e49
Merge pull request #2045 from qieqieplus/fix-vacuum-commit 2021-05-05 07:41:38 -07:00
qieqieplus ac26080bd2 fix concurrent vacuum & delete panic 2021-05-05 17:54:50 +08:00
Chris Lu e24ba2aadc filer: delete specific tags 
fix https://github.com/chrislusf/seaweedfs/issues/2041
 2021-05-02 21:53:43 -07:00
Chris Lu e87e6ef33c s3: return 404 if bucket does not exist 
fix https://github.com/chrislusf/seaweedfs/issues/2039
 2021-05-02 21:30:37 -07:00
Chris Lu a4bb37a5fe add debug info 2021-05-01 16:10:26 -07:00
Chris Lu 2e56407c6b fix visited checking 2021-05-01 16:09:29 -07:00
Chris Lu c48ef78670 2.43 2021-05-01 00:39:04 -07:00
Chris Lu 3a86d4dbfd mount: fix directory invalidation 
fix https://github.com/chrislusf/seaweedfs/issues/2038
 2021-04-30 22:51:06 -07:00
Chris Lu d74cdf0115 Revert "Merge pull request #2027 from bingoohuang/master" 
Need to revert because docker image build failed. The docker apk package only has go 1.15.
 2021-04-30 03:36:15 -07:00
Chris Lu 84312e6799 2.42 2021-04-30 03:14:07 -07:00
Chris Lu fa0d973113
Merge pull request #2037 from utsl42/reader_at 
make reader_at handle random reads more efficiently for FUSE
 2021-04-28 17:31:46 -07:00
Nathan Hawkins 042de9359c make reader_at handle random reads more efficiently for FUSE 2021-04-28 19:13:37 -04:00
Chris Lu a26a37dfa3 fix compilation 
fix related to #2032
 2021-04-28 13:36:53 -07:00
Chris Lu ef94ff6837
Merge pull request #2035 from kmlebedev/fix_chunks_etag 
fix aws style Etag for chunks
 2021-04-28 10:37:18 -07:00
Chris Lu 8ae3ea4b1d
Merge pull request #2034 from kmlebedev/iam_listkeys_filter 
iam ListAccessKeys filtred by username
 2021-04-28 10:34:40 -07:00
Chris Lu 9dca75aea8
Merge pull request #2032 from tobiasmuehl/patch-2 
Detect rar archives by mime type
 2021-04-28 10:31:19 -07:00
Konstantin Lebedev c2269123d3 fix aws style Etag for chunks 2021-04-28 22:28:05 +05:00
Konstantin Lebedev 39f636c682 iam ListAccessKeys filtred by username 2021-04-28 17:15:22 +05:00
Tobias Mühl a8864e2abd
Detect rar archives by mime type 
RAR archives might not have .rar extension, see [Wikipedia](https://en.wikipedia.org/wiki/RAR_(file_format))
 2021-04-28 15:54:19 +07:00
Tobias Mühl 12a7e87007
Do not compress brotli archives 2021-04-28 15:51:49 +07:00
Konstantin Lebedev a48785c7df auth use bucket wild cards 2021-04-27 21:45:40 +05:00
bingoohuang 7a9d27fce8 promote to go:embed instead of github.com/rakyll/statik 2021-04-27 17:22:24 +08:00
bingoohuang cf552417a7 minFreeSpace refactored 2021-04-27 10:37:24 +08:00
bingoohuang 31f1cdeac2 minFreeSpace argument allows size like 10GiB 2021-04-26 18:48:34 +08:00
Chris Lu 86185262bb 2.41 2021-04-24 16:54:36 -07:00
Chris Lu 79f2e780c1 ensure name pattern checking is case sensitive 2021-04-24 11:51:23 -07:00
Chris Lu ddc8643ee0 filer: directory listing adds namePatternExclude 
fix https://github.com/chrislusf/seaweedfs/issues/2023
 2021-04-24 11:49:03 -07:00
Chris Lu f0ad172e80 shell: show which server holds the lock 
fix https://github.com/chrislusf/seaweedfs/issues/1983
 2021-04-22 23:56:35 -07:00
Chris Lu 89eb9f6e70 clean up .uploads directory in mysql and postgres tables 
fix https://github.com/chrislusf/seaweedfs/issues/1957

When no uploads are running, you can run this SQL to clean up.
delete from <bucket_name> where directory like '/.uploads/%'
 2021-04-22 23:23:23 -07:00
Chris Lu 46ef1811a1 correct help message 2021-04-22 22:26:38 -07:00
Chris Lu 11c120c040 master UI adds volume size limit 2021-04-22 14:22:48 -07:00
Chris Lu 6e5df901e4 adjust package names 2021-04-22 14:22:48 -07:00
Chris Lu ca998328c2 do not add new volumes when below minFreeSpacePercent 
fix https://github.com/chrislusf/seaweedfs/issues/2017
 2021-04-21 23:11:11 -07:00
Chris Lu cd7bf1a72a filer.copy copy empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/2016
 2021-04-21 11:17:43 -07:00
Chris Lu a8114da02d avoid thundering herd effect 
transient errors may cause thundering herd effect to all trying to recover from remove ec shards
 2021-04-21 10:17:12 -07:00
Chris Lu ae74d8f02a fix error message 
related to https://github.com/chrislusf/seaweedfs/issues/2012
 2021-04-21 01:40:16 -07:00
Chris Lu 11c405fc85 ensure file handles are released 2021-04-20 19:56:55 -07:00
liuxiaobo c31c5e829c
fix path-specific filer store comment error 2021-04-20 10:08:58 +08:00
Chris Lu 83cf94ad2d delay new file creation unless file is opened exclusively 2021-04-19 10:58:25 -07:00
Chris Lu e983f91b03 2.40 2021-04-18 13:58:01 -07:00
Chris Lu d1c813c470 let the fuse library manage directory id 
otherwise, on mac, during large directory deletion, if some ReaDirAll happens, the lib seems confused about the directories, and some child directories are not deleted.
 2021-04-18 13:07:28 -07:00
Chris Lu 372872ebbf set root node inode number 2021-04-18 13:07:28 -07:00
Chris Lu e332da4837 set inode value 2021-04-18 13:07:28 -07:00
Chris Lu 2acf6be24e resend the http request if connection is stale 2021-04-18 13:07:28 -07:00
Chris Lu 6cbd786db9 correctly runs git clone 2021-04-18 13:07:28 -07:00