gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
7419 commits 28 branches 278 tags 135 MiB
Commit graph

4931 commits

Author SHA1 Message Date
chrislu 3aeee3d748 ensure releasing file handle 2022-03-07 14:01:24 -08:00
chrislu 8136384473 remove debug message 2022-03-07 11:22:26 -08:00
banjiaojuhao b9ff7723dd [bugfix] filer: nil pointer dereference 2022-03-07 23:26:25 +08:00
chrislu da3d330616 s3 and filer transport using unix domain socket instead of tcp 2022-03-07 02:00:14 -08:00
chrislu 0cb17b45b1 refactoring 2022-03-07 01:59:01 -08:00
chrislu 6d3db4445b buffer for all range requests 2022-03-07 01:56:47 -08:00
chrislu f3bcbeb60a a little optimization 2022-03-07 00:24:59 -08:00
chrislu f7ee60996c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-07 00:08:09 -08:00
chrislu 6f3ec989a7 fix manifest batch size 2022-03-07 00:07:53 -08:00
Chris Lu 1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest 
filer: support get metadata with resolved manifest chunk
 2022-03-07 00:05:46 -08:00
chrislu bb0b784544 minor 2022-03-07 00:04:59 -08:00
banjiaojuhao bfcc9ca808 filer: support metadata with resolved manifest chunk 2022-03-07 15:47:51 +08:00
banjiaojuhao 71f3046841 filer: add back isAppend function 2022-03-07 15:41:07 +08:00
chrislu 0ba4e4cd23 2.93 2022-03-06 18:54:12 -08:00
chrislu 46a28b8819 mount: adjust disk space based on quota 2022-03-06 17:22:49 -08:00
chrislu ede6ce44c6 fix test 2022-03-06 17:09:55 -08:00
chrislu 21ef152423 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-06 17:04:24 -08:00
chrislu f3442e36e6 mount: quota adjust error type to be syscall.ENOSPC 2022-03-06 17:04:21 -08:00
Chris Lu 89cd0c27a8
Merge pull request #2728 from kmlebedev/set_default_leveldb2 2022-03-06 07:26:30 -08:00
Konstantin Lebedev cf444ebd07 Set default leveldb2 enabled 
avoid Filer store is enabled for both leveldb2 and mysql
 2022-03-06 18:27:25 +05:00
banjiaojuhao a07c93f4dd filer: reset bytesBuffer before use 2022-03-06 21:07:36 +08:00
chrislu c7e8ac18f0 mount: quota for one mounted collection 
related to https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48
 2022-03-06 02:44:40 -08:00
chrislu b7c992f410 add flag to enforce quota 2022-03-05 22:10:43 -08:00
chrislu f1713c96ae avoid possible runtime error: index out of range [0] with length 0 2022-03-05 21:14:31 -08:00
chrislu da76af187f mount: avoid possible index out of bounds error 2022-03-04 22:36:01 -08:00
chrislu 6e49e75a5b use logical number of files and sizes for statistics and quota 2022-03-04 18:47:44 -08:00
chrislu f51e20028a mount: avoid comma in mount options 
fix https://github.com/chrislusf/seaweedfs/issues/2719
 2022-03-03 03:42:29 -08:00
chrislu dc0f48682c unused 2022-03-03 03:41:35 -08:00
chrislu 28b8974a3a mount: fix directory pagination when using midnight commander 2022-03-03 02:59:31 -08:00
chrislu a96d4254e9 filer, s3, volume server: a bit memory optimization 2022-03-02 20:15:28 -08:00
chrislu 6fbbc78574 stream reading a whole chunk 2022-03-02 13:50:46 -08:00
chrislu 784583afc6 avoid pool memory allocation if too large 2022-03-02 13:50:28 -08:00
chrislu ba14307319 2.92 2022-02-28 15:22:19 -08:00
chrislu fcf3714443 mount: add back support for filer.path 2022-02-28 12:16:53 -08:00
chrislu 80c017907b filer.backup: fix backing up encrypted chunks 
I have done filer.backup test:
replication.toml:
[sink.local]
enabled = true
directory = "/srv/test"
___
system@dat1:/srv/test$ weed filer.backup -filer=app1:8888 -filerProxy
I0228 12:39:28 19571 filer_replication.go:129] Configure sink to local
I0228 12:39:28 19571 filer_backup.go:98] resuming from 2022-02-28 12:04:20.210984693 +0100 CET
I0228 12:39:29 19571 filer_backup.go:113] backup app1:8888 progressed to 2022-02-28 12:04:20.211726749 +0100 CET 0.33/sec

system@dat1:/srv/test$ ls -l
total 16
drwxr-xr-x 2 system system 4096 Feb 28 12:39 a
-rw-r--r-- 1 system system   48 Feb 28 12:39 fu.txt
-rw-r--r-- 1 system system   32 Feb 28 12:39 _index.html
-rw-r--r-- 1 system system   68 Feb 28 12:39 index.php
system@dat1:/srv/test$ cat fu.txt
?	?=?^??`?f^};?{4?Z%?X0=??rV????|"?1??踪~??
system@dat1:/srv/test$
On the active mount on the target server it's:
system@app1:/srv/app$ ls -l
total 2
drwxrwxr-x 1 system system  0 Feb 28 12:04 a
-rw-r--r-- 1 system system 20 Feb 28 12:04 fu.txt
-rw-r--r-- 1 system system  4 Feb 28 12:04 _index.html
-rw-r--r-- 1 system system 40 Feb 28 12:04 index.php
system@app1:/srv/app$ cat fu.txt
This is static boy!
Filer was started with: weed filer master="app1:9333,app2:9333,app3:9333" -encryptVolumeData
It seems like it's still encrypted?
 2022-02-28 10:07:06 -08:00
chrislu aad62ee148 mount: mark directory uncached if forgotten 2022-02-28 02:08:24 -08:00
chrislu 554e239097 release file handle 2022-02-28 01:23:14 -08:00
chrislu 3639fedd01 mount: fix fsync opened and renamed files 2022-02-28 00:34:17 -08:00
chrislu 63a9d8f01d ensure inodes are not duplicating unless hardlinked 2022-02-27 23:13:49 -08:00
chrislu de77d00c81 correctly clean up for a file 2022-02-27 23:12:28 -08:00
chrislu 18543c6e8b minor 2022-02-27 23:11:09 -08:00
chrislu 09cd00f356 2.91 2022-02-27 04:03:39 -08:00
chrislu 941ced60a4 download 2 chunks if at the beginning of a file 2022-02-27 03:57:24 -08:00
chrislu f9d9eed0c9 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-27 03:41:35 -08:00
chrislu d602d68fd1 remove dead code 2022-02-27 03:41:32 -08:00
Chris Lu 6c07af6014
Merge pull request #2715 from gfx-labs/acl-pr 2022-02-27 03:27:32 -08:00
chrislu c3792c8352 remove dead code 2022-02-27 03:03:19 -08:00
chrislu aa9eef81e6 retire mount v1 2022-02-27 02:57:27 -08:00
eddy-gfx fb940dd807
Merge branch 'chrislusf:master' into acl-pr 2022-02-27 04:52:08 -06:00
elee 881a0fe806 ensure compatibility 2022-02-27 04:50:59 -06:00
elee 954ad98e0d set canned acl on replication create 2022-02-27 04:49:31 -06:00
chrislu 6e1ab97988 use debug option to see operations 2022-02-27 02:02:30 -08:00
chrislu 2112d99140 mount2: add back readonly mode 2022-02-27 01:13:32 -08:00
chrislu 9ef5bb20f6 mount2: invalidate fuse cache for replaced inode 2022-02-27 00:00:23 -08:00
chrislu 4ddcbaab57 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-26 23:20:47 -08:00
chrislu 551d00d51a prefetch other chunks when stream reading 2022-02-26 23:20:45 -08:00
Chris Lu e1d3dd5e18
Merge pull request #2712 from guo-sj/correct_comments 2022-02-26 09:16:12 -08:00
guosj 82cad5e330
correct comments 2022-02-26 22:44:26 +08:00
chrislu 7b1a713d2a remove dead code 2022-02-26 03:23:15 -08:00
chrislu 3a58b7bac6 a little safer 2022-02-26 03:23:06 -08:00
chrislu 708e14fcfa avoid possible too big memory allocation 2022-02-26 03:22:41 -08:00
chrislu 3345a50d9b prefetch 2 chunks 2022-02-26 03:06:17 -08:00
chrislu 86ce69f709 remove logs 2022-02-26 03:00:20 -08:00
chrislu b2a148cb4c use file size as max range 2022-02-26 03:00:08 -08:00
chrislu 2ab0ad24a3 use memory pool 2022-02-26 02:59:19 -08:00
chrislu 28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu 3ad5fa6f6f chunk cache adds function ReadChunkAt 2022-02-25 21:55:04 -08:00
chrislu fc7a4957ea fix mount2 options 2022-02-25 21:22:44 -08:00
chrislu 72c0233938 less logs 2022-02-25 15:34:24 -08:00
chrislu 101e6d80b0 mount2: listXattr return ok if xattr is empty 2022-02-25 14:38:56 -08:00
chrislu 03466f955e rename: delete source entry metadata only, skipping hard links 2022-02-25 02:57:54 -08:00
chrislu 8080fe4cc1 logs 2022-02-25 02:56:23 -08:00
chrislu e423548673 rename: pass along entry metadata 2022-02-25 02:53:37 -08:00
chrislu e8110bb54c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-25 01:17:28 -08:00
chrislu 202a29d014 refactoring 2022-02-25 01:17:26 -08:00
chrislu 91d6785cf3 define metadata action types 2022-02-25 00:54:16 -08:00
chrislu be3fc77391 mount2: use consistent inode 2022-02-25 00:53:27 -08:00
Chris Lu 9873bae115
Merge pull request #2706 from guo-sj/fix_log_info 2022-02-24 22:48:19 -08:00
guosj 121b31f750 fix incorrect log information 2022-02-25 13:41:20 +08:00
chrislu ceaf993a27 mount2: add rdev 2022-02-24 14:51:25 -08:00
chrislu 95717d1006 simpler output 2022-02-24 13:50:08 -08:00
chrislu b05962b90e rename: handle hard links 2022-02-24 02:59:00 -08:00
chrislu e31ec04f4f mount2: POSIX deleted opened file nlink should be 0 2022-02-24 01:59:37 -08:00
chrislu 419e355e9e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-24 01:31:07 -08:00
chrislu 91f0481f4e mount2: SetAttr set mode correctly 2022-02-24 01:31:04 -08:00
Chris Lu 97a4b66df7
Merge pull request #2704 from guo-sj/fix_bugs_in_return_value 
fix return value in storage/volume_vacuum.go:444
 2022-02-24 00:49:29 -08:00
chrislu abe7214c1f Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-24 00:37:59 -08:00
chrislu b93d57da31 mount2: dir read opened file 2022-02-24 00:37:57 -08:00
guosj 3e7aa1caf5 fix return value in storage/volume_vacuum.go:444 2022-02-24 15:54:36 +08:00
Chris Lu 6834df77a0
Merge pull request #2701 from guo-sj/fix_bugs_in_return_value 2022-02-23 19:23:37 -08:00
guosj 26f3ab8d4b fix a return bug in func (c *commandVacuum) Do 2022-02-24 09:04:38 +08:00
chrislu 320637dc7a use "mv.from" for moving files 2022-02-23 15:34:42 -08:00
chrislu c29bc9a367 fix error handling 2022-02-23 15:34:25 -08:00
Chris Lu da58c748bc
Merge pull request #2698 from guo-sj/fix_bugs_in_return_value 2022-02-23 00:41:30 -08:00
Chris Lu bd092d8318
Merge pull request #2697 from guo-sj/fix_bugs_in_return_value 2022-02-23 00:30:52 -08:00
guosj d68c27f82d fix another return value bug 2022-02-23 16:21:25 +08:00
guosj 8f9aa0cddd fix bugs in return value 2022-02-23 16:17:48 +08:00
Tuan Vuong d2ec62656d initialize master address in iam options 2022-02-23 12:01:54 +07:00
banjiaojuhao 6ab09e9071 filer_http: support uploading file with offset 2022-02-22 00:15:00 +08:00
chrislu 497ebbbd45 2.90 2022-02-20 22:00:13 -08:00
chrislu 56da3494cb leveldb3: adjust memory allocation for each bucket 2022-02-20 13:40:13 -08:00
banjiaojuhao e6126cef62 filer_web: support moving entry 2022-02-20 23:56:23 +08:00
chrislu 7bc67399e4 listing for filer stores without prefixed query: break if no more progress 2022-02-19 10:12:51 -08:00
chrislu 248c0c8087 minor 2022-02-19 09:54:09 -08:00
chrislu 4ee0a6f47b filer store: reduce one possible listing operation 2022-02-19 00:43:42 -08:00
chrislu b3594278c9 optimize a bit 2022-02-18 23:36:10 -08:00
chrislu 61811dc2f1 comments 2022-02-18 22:14:40 -08:00
chrislu daa27b2119 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-18 20:55:10 -08:00
chrislu 9014d00fd0 Revert "s3: listObjectParts return ErrNoSuchUpload if does not exist" 
This reverts commit 6cf2e7d493.
 2022-02-18 20:54:54 -08:00
banjiaojuhao 4c30934cd9 filer: support get file entry 2022-02-18 22:52:26 +08:00
chrislu 63062ed7f0 mount2: fix unlink 2022-02-18 01:10:53 -08:00
chrislu 93e12d5f30 fix build 2022-02-18 00:48:00 -08:00
chrislu d62370d4e0 add todo 2022-02-18 00:47:15 -08:00
chrislu e8ce30fdc5 mount2: adjust file mode 2022-02-18 00:47:02 -08:00
chrislu f9d33f70b0 return fuse.Status when looking up by inode 2022-02-18 00:45:43 -08:00
chrislu b9cf4f12fc Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-16 21:33:51 -08:00
chrislu 49b84b6e2a list entries while reading from remote 2022-02-16 21:32:15 -08:00
Chris Lu 38dcaaa76e
Merge pull request #2683 from guo-sj/fix_fsconfigure_bug 
fix minor bug in commandFsConfigure.Do
 2022-02-16 18:16:47 -08:00
guosj ca121ecd66 fix minor bug in commandFsConfigure.Do 2022-02-17 09:57:52 +08:00
chrislu 65a19e3abc fix listing with correct inode 2022-02-16 17:01:39 -08:00
chrislu 6ac066d1dc count lookup or not 2022-02-16 16:49:03 -08:00
chrislu a129bda7d9 sync data first before stopping 2022-02-16 09:11:34 -08:00
chrislu 118d0e01a8 less logs 2022-02-16 08:45:07 -08:00
chrislu a6bc67c34c less logs 2022-02-16 08:38:51 -08:00
chrislu 2facd65998 fix second listing 2022-02-16 08:16:27 -08:00
chrislu 1013fc90d8 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-16 07:57:11 -08:00
chrislu 22739c653a clean up 2022-02-16 07:57:08 -08:00
Chris Lu 3cb19cf6db
Merge pull request #2680 from kmlebedev/volume_healthz 2022-02-16 04:43:44 -08:00
Konstantin Lebedev 9ea09cc41c healthz check to avoid drain pod with last replicas 2022-02-16 14:18:36 +05:00
chrislu 65bfeafb5a add back mkdir 2022-02-16 01:09:21 -08:00
chrislu 37e8fce841 clean up cache on exit 2022-02-16 00:39:21 -08:00
chrislu 1560ec7e26 remove unused code 2022-02-16 00:37:24 -08:00
chrislu 3cbce878f2 mount2: fix directory pagination 2022-02-15 22:42:10 -08:00
chrislu df51e0c042 mongodb: remove ErrNoDocuments checking since it is only for creating single record, not for querying 2022-02-15 16:18:36 -08:00
chrislu 98cce6a150 mongodb: ensure closing cursor 2022-02-15 16:15:44 -08:00
Chris Lu aa7fc299b8
Merge pull request #2676 from banjiaojuhao/add_filer_store-redis_lua 
FilerStore: add redis_lua
 2022-02-15 12:10:34 -08:00
banjiaojuhao fc3b75f2f8 weed/operation/delete_content.go: nil pointer dereference 2022-02-15 23:07:03 +08:00
banjiaojuhao b5ec346700 FilerStore: add redis_lua 2022-02-15 20:54:57 +08:00
chrislu 17ac5244c3 mount2: avoid double listing directories 2022-02-15 01:44:17 -08:00
guosj db7766323e complete project code, remain test code 2022-02-15 17:35:33 +08:00
chrislu 222798d926 mount2: fix for read dir plus on linux 2022-02-15 00:29:17 -08:00
chrislu 4e72863ba5 mount2 add debug mode 2022-02-15 00:26:30 -08:00
Chris Lu 64afbc5235
Merge pull request #2640 from Radtoo/fsck_with_delete 
Added basic deletion capabilities to fsck.
 2022-02-14 23:00:01 -08:00
chrislu 4e181db21a mount: default disable cache 
* Prevent cases as https://github.com/seaweedfs/seaweedfs-csi-driver/issues/43
* Improve read write benchmarks
* Improve AI training performance. Most of the files are just read once.
 2022-02-14 20:42:33 -08:00
chrislu ebc22625b4 less noisy on mac 2022-02-14 16:38:01 -08:00
chrislu 377bf31445 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-14 13:48:51 -08:00
chrislu e8420aaed7 fix building for windows freebsd 2022-02-14 13:48:48 -08:00
Chris Lu d3ee621fce
Merge pull request #2661 from garenchan/ck-dev1 2022-02-14 10:08:37 -08:00
Konstantin Lebedev 526094d2da StopTimeout 30 sec 2022-02-14 21:42:27 +05:00
Konstantin Lebedev 275e9a4e86 reduce to default http server KillTimeout and StopTimeout 2022-02-14 21:38:24 +05:00