gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
7483 commits 28 branches 278 tags 135 MiB
Commit graph

243 commits

Author SHA1 Message Date
chrislu 706a86077d remove dead code 2022-02-27 03:41:22 -08:00
chrislu f87da798a4 to be re-written following fuse virtual file system 2022-02-11 03:09:30 -08:00
chrislu 7a0c35674c clean up previously mounted folder 2022-02-10 20:46:53 -08:00
chrislu c3f9d9fa2e initial setup 2022-02-10 20:32:13 -08:00
chrislu ee03cb0aa1 go mod tidy 2022-01-19 21:46:03 -08:00
chrislu 01ccd6778e sync with fuse 1.2.2 2022-01-11 12:24:54 -08:00
chrislu d351541757 upgrade fuse version 2021-12-28 18:18:54 -08:00
Konstantin Lebedev 791ad774a2 CVE-2020-13949 
CVE-2020-27813
CVE-2019-19794
CVE-2021-38561
 2021-12-23 13:57:41 +05:00
Chris Lu 2ba08afed1
Merge pull request #2498 from kmlebedev/s3_audit_log 2021-12-07 09:35:48 -08:00
Konstantin Lebedev 4ec8715f20 audit log 2021-12-07 12:15:48 +05:00
chrislu f7229084a9 upgrade go.mongodb.org/mongo-driver 
go.mongodb.org/mongo-driver@v1.7.0: retracted by module author: Contains data race bug in background connection establishment.
go: to switch to the latest unretracted version
 2021-12-04 23:48:55 -08:00
Chris Lu 3eeaffeadd update gjson 
resolve github.com/tidwall/gjson vulnerability found
 2021-10-25 16:13:31 -07:00
Chris Lu 3e2acf677c removing tikv to resolve "go mod tidy" problem 
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back.

go mod tidy
go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker
go: finding module for package cloud.google.com/go/kms/apiv1
go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/coreos/etcd/clientv3/credentials
go: finding module for package github.com/coreos/etcd/clientv3/balancer
go: finding module for package github.com/d4l3k/messagediff
go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible
go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0
go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1
go: finding module for package google.golang.org/grpc/naming
github.com/chrislusf/seaweedfs/weed/filer/tikv imports
	github.com/tikv/client-go/v2/tikv imports
	go.etcd.io/etcd/clientv3 tested by
	go.etcd.io/etcd/clientv3.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
 2021-10-10 19:27:02 -07:00
Chris Lu 893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu 10d6392101 clean up 2021-09-30 21:29:58 -07:00
Chris Lu 41efc6e629 Update go.mod 
CVE-2020-28483
high severity
Vulnerable versions: < 1.7.0
Patched version: 1.7.0
This affects all versions of package github.com/gin-gonic/gin under 1.7.0. When gin is exposed directly to the internet, a client's IP can be spoofed by setting the X-Forwarded-For header.
 2021-09-30 21:23:05 -07:00
Chris Lu 92d65c917c use grocksdb 2021-09-30 20:19:48 -07:00
chrislusf 3ffbaaa071 use github.com/linxGnu/grocksdb 2021-10-01 03:16:03 +00:00
Konstantin Lebedev 4c7a3cf2fd trivy: security fix 2021-09-30 14:06:32 +05:00
Chris Lu 43d6678d6d remove dep 2021-09-03 23:36:37 -07:00
Chris Lu 8ec357b3d3 go mod 2021-09-03 23:25:33 -07:00
Chris Lu cc36f5b01f go get -u github.com/Azure/azure-storage-blob-go/azblob 2021-09-03 22:31:29 -07:00
Chris Lu 24fda6f47f remove QR code for mobile upload 2021-09-01 16:59:37 -07:00
Chris Lu 8ecd609b48 go tidy 2021-08-30 17:28:14 -07:00
Rain Li e5f9ff983d
Merge branch 'master' into filerstore-tikv 2021-08-30 15:52:54 +08:00
Chris Lu 5adfdd7982 change to a new bloom filter library 2021-08-29 21:26:38 -07:00
Chris Lu 001a472057 cloud mount: remote storage support hdfs 2021-08-29 18:41:29 -07:00
yulai.li 546efeba8f Fix build bug 2021-08-26 16:20:35 +08:00
yulai.li 2088f28424 init post 2021-08-26 15:20:18 +08:00
Chris Lu 7c39a18ba5 update azure library 2021-08-24 00:32:35 -07:00
Chris Lu e64339e8c0 update go mod to go 1.17 2021-08-17 11:20:50 -07:00
Chris Lu 223b344c79 update fuse version 2021-08-17 11:18:22 -07:00
Konstantin Lebedev 1f301e4ba4 gccgo build 2021-08-17 20:27:47 +05:00
Chris Lu 8461c025ef mount: use pooled goroutine instead of round-robin 
fix https://github.com/chrislusf/seaweedfs/issues/2263
 2021-08-17 03:37:06 -07:00
Konstantin Lebedev 865b06b7fe The IEEE assembler optimizations has been submitted and will be part of the Go 1.6 standard library. 
https://github.com/klauspost/crc32
 2021-08-17 13:06:48 +05:00
Chris Lu 7667d26ebf gocql update 
related to https://github.com/chrislusf/seaweedfs/issues/2246
 2021-08-12 00:20:11 -07:00
byunghwa.yun bdc7730fdb Add autocomplete 2021-08-03 02:56:52 +09:00
Chris Lu 60f5c0a2f5 fix security alert on github.com/dgrijalva/jwt-go 
resolve https://github.com/chrislusf/seaweedfs/security/dependabot/go.mod/github.com%2Fdgrijalva%2Fjwt-go/open
 2021-07-22 10:22:09 -07:00
Chris Lu f3746d350f resolve gjson security alert 
resolve https://github.com/chrislusf/seaweedfs/security/dependabot/go.mod/github.com%2Ftidwall%2Fgjson/open
 2021-07-22 09:57:49 -07:00
Chris Lu a70e772a03 update mongodb lib due to security warning 2021-07-22 09:49:40 -07:00
Chris Lu 215b169562 mount: recursively rename locally 2021-07-01 01:19:31 -07:00
Chris Lu 78b1fb921c adjust log level 2021-06-23 20:59:54 -07:00
Jonas Falck 829b195084 Add process metrics of weed itself 2021-06-22 13:09:42 +02:00
Chris Lu f15d7a57f5 mount: support macfuse 4.x 2021-06-03 01:14:50 -07:00
Konstantin Lebedev 69c768870b - object write cache control 
- object write expires
 2021-05-24 15:43:55 +05:00
Chris Lu 064269bb57 filer: Support sqlite as filer meta store 2021-05-23 23:58:28 -07:00
Chris Lu fc5a8af09e fix possible nil 2021-05-11 22:06:41 -07:00
Chris Lu da0a4e775b Revert "Revert "Merge pull request #2027 from bingoohuang/master"" 
This reverts commit d74cdf0115.
 2021-05-10 00:13:55 -07:00
Chris Lu 930581cedc fuse: avoid nil when invalidating entry 
fix https://github.com/chrislusf/seaweedfs/issues/2055
 2021-05-07 07:09:04 -07:00
Chris Lu ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu d74cdf0115 Revert "Merge pull request #2027 from bingoohuang/master" 
Need to revert because docker image build failed. The docker apk package only has go 1.15.
 2021-04-30 03:36:15 -07:00
bingoohuang 7a9d27fce8 promote to go:embed instead of github.com/rakyll/statik 2021-04-27 17:22:24 +08:00
Chris Lu 4b3cc28cab fuse 1.1.4 2021-04-18 13:29:37 -07:00
Chris Lu d9a2a7f1c4 WIP 
no memory issue

if some directory is removed, it may have this error

$ rm -Rf ~/tmp/m2/s1
rm: fts_read: Device not configured
 2021-04-18 13:06:38 -07:00
Konstantin Lebedev 8e02e138ea Merge branch 'upstreamMaster' into iam_handlers 2021-04-12 11:22:51 +05:00
Konstantin Lebedev 5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Chris Lu 98c08a3dcd raft: fix possible nil 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1962c8a]

goroutine 68239935 [running]:
github.com/chrislusf/raft.(*LogEntry).Term(...)
 /home/travis/gopath/pkg/mod/github.com/chrislusf/raft@v1.0.4/log_entry.go:59
github.com/chrislusf/raft.(*server).TakeSnapshot(0xc00014f320, 0xc028b5a600, 0xc03995bc00)
 /home/travis/gopath/pkg/mod/github.com/chrislusf/raft@v1.0.4/server.go:1276 +0x50a
github.com/chrislusf/raft.(*server).maybeTakeSnapshot.func1(0xc00014f320)
 /home/travis/gopath/pkg/mod/github.com/chrislusf/raft@v1.0.4/server.go:1221 +0x5b
created by github.com/chrislusf/raft.(*server).maybeTakeSnapshot
 /home/travis/gopath/pkg/mod/github.com/chrislusf/raft@v1.0.4/server.go:1219 +0x98
 2021-04-10 02:36:53 -07:00
Chris Lu b06c5b9d99 upgrade raft to v1.0.5 
fix https://github.com/chrislusf/seaweedfs/issues/1974
 2021-04-07 20:59:48 -07:00
李海 69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Konstantin Lebedev 7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Chris Lu e19b6b2d0a upgrade fuse version, add batch forget 2021-03-12 02:55:06 -08:00
Chris Lu 6d729fa742 Update go.mod 2021-03-10 12:22:58 -08:00
Yoni Nakache 302dbb1b9c
update pg to latest version 
currently, it is an old 2+ years driver
 2021-03-08 07:16:06 +02:00
Chris Lu 7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Konstantin Lebedev 24ec17219b RabbitMQ delay retry with Dead Letter Exchange 
https://github.com/chrislusf/seaweedfs/issues/1773

https://github.com/google/go-cloud/issues/2952
 2021-02-03 21:32:30 +05:00
Chris Lu 22e167bc8a mount: skip daemon timeout on FreeBSD 2021-01-27 15:34:43 -08:00
Chris Lu 3a1d3d3413 mount: properly invalidate kernel node cache entry 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-26 02:50:53 -08:00
Chris Lu 84f05787f8 mount: limit background requests and congestion threshold 2021-01-20 23:30:14 -08:00
Chris Lu acc0d5cd2c Revert "Revert "filer store: redis module upgrade"" 
This reverts commit dd0171e35c.
 2021-01-11 02:30:19 -08:00
Chris Lu dd0171e35c Revert "filer store: redis module upgrade" 
This reverts commit 40161c9cec.
 2021-01-11 02:09:18 -08:00
Chris Lu 90b117acf1 update ccache version 2021-01-08 02:17:43 -08:00
Chris Lu 40161c9cec filer store: redis module upgrade 2021-01-08 01:08:44 -08:00
yuwang d253e0ea6e fix s3 upload "use of closed network connection" 2021-01-06 15:11:54 +08:00
Chris Lu c3f674ac1d address raft nil snapshot during flush 
fix https://github.com/chrislusf/seaweedfs/issues/1718
 2021-01-03 13:38:27 -08:00
Chris Lu f4d87f8b9d works 
performance seems about the same as leveldb for insert entries
 2021-01-02 23:32:58 -08:00
Chris Lu 32955b04b8 Update go.mod 2020-12-23 22:08:41 -08:00
Chris Lu 64c48c9724 seems compiling 2020-12-23 21:44:53 -08:00
Chris Lu 3fedfec1e7 check cross device rename error 2020-12-10 23:50:32 -08:00
Chris Lu 92f906b6fc remove zstd 
fix https://github.com/chrislusf/seaweedfs/issues/1629
 2020-11-21 13:06:45 -08:00
Chris Lu 0a406f652e load filer conf and match by prefix 2020-11-15 00:26:05 -08:00
Chris Lu 7cfa616843 raft: grpc wait for ready 
related to https://github.com/chrislusf/seaweedfs/issues/1618

https://github.com/grpc/grpc/blob/master/doc/wait-for-ready.md

7c64624a06
 2020-11-14 20:52:26 -08:00
Chris Lu 57c770a7b4 go get github.com/chrislusf/raft@v1.0.2 2020-11-14 20:39:12 -08:00
Chris Lu c923170d7c Revert "go mod tidy" 
This reverts commit 85e43b678f.
 2020-11-02 19:11:11 -08:00
Chris Lu 85e43b678f go mod tidy 2020-11-02 14:23:33 -08:00
Chris Lu be95f68ca7 scaffold for sftpd 2020-10-30 20:57:08 -07:00
Chris Lu e50eae9ad8 update fuse 2020-10-24 02:04:48 -07:00
Chris Lu 8eaa067da5 module upgrade 2020-10-23 23:09:23 -07:00
Chris Lu e0c8507d93 memory pool to reduce memory allocation 2020-10-23 00:41:40 -07:00
Chris Lu 9884dfc369 use bytebuffer for interval list 2020-10-21 19:29:51 -07:00
Chris Lu f64252023e Revert "a better byte buffer pool" 
This reverts commit 59c6422777.
 2020-10-20 23:21:54 -07:00
Chris Lu 59c6422777 a better byte buffer pool 2020-10-20 22:53:58 -07:00
Chris Lu 9b0e8ef026 filer: added QR code to transfer files with mobile devices 2020-10-10 18:00:21 -07:00
Chris Lu 8c8b8e2835 adjust go module 2020-10-04 02:27:24 -07:00
Chris Lu 666859f7cd mount: read prefetching 2020-10-04 01:31:04 -07:00
Устюжанин Антон Александрович dc31b19469 fix: restore raft state 2020-10-03 14:03:41 +05:00
ruitao.liu 83080b5e03 ES backended filer support kv ops. 2020-09-04 15:40:13 +08:00
Chris Lu 0ddcc2a8f5 go mod 2020-09-03 23:04:27 -07:00
ruitao.liu 798280e980 change filer2 to filer. 2020-09-03 17:05:26 +08:00
ruitao.liu a93d27d1e8 new filer option to es v7. 2020-09-03 16:34:58 +08:00
Chris Lu fcb0ff9890 git mod tidy 2020-08-06 22:22:14 -07:00
Chris Lu 37e964d4bd update gorilla mux 2020-07-26 14:08:22 -07:00
Chris Lu f4abe284ae adjust go.mod 
seems not useful any more
 2020-07-24 10:18:44 -07:00
李远军 8c318470dd Revert "remove fixJpgOrientation" 
This reverts commit de5ca9b2

minimise the revert changes
 2020-07-10 10:28:28 +08:00
Chris Lu 1b680c06c1 update to raft 1.0.1 2020-07-08 10:28:04 -07:00
Chris Lu a73fd7682a grpc upgrading 2020-06-21 09:24:21 -07:00
Chris Lu ca3516ac6d adjust protoc 2020-06-20 08:00:25 -07:00
Chris Lu 7aafb9e3f8 Revert "filer metrics upgrade prometheus client to 1.6.0" 
This reverts commit 09aef55839.
 2020-05-15 21:39:03 -07:00
Chris Lu f8aed8a7f5 Merge branch 'master' into msg_channel 2020-05-14 20:06:15 -07:00
Chris Lu 09aef55839 filer metrics upgrade prometheus client to 1.6.0 
fix metrics.go:123] could not push metrics to prometheus push gateway 192.168.xx:9091: unexpected status code 200 while pushing to http://192.168.xx:9091/metrics/job/filer/instance/m111p112%3A8888:
 2020-05-14 20:05:51 -07:00
Chris Lu dfccc3c263 able to read chan and write chan 2020-05-08 02:47:22 -07:00
wuyuxiang 6850d28d6b refacotr: move signal handling and pprof to grace package 2020-04-28 14:10:23 +08:00
bukton b9a71a5638 update package 2020-04-19 00:29:52 +07:00
bukton 290c6b7f01 Merge remote-tracking branch 'origin/master' into filer_mongodb 
# Conflicts:
#	go.mod
#	go.sum
#	weed/server/filer_server.go
 2020-04-19 00:21:45 +07:00
bukton 3c70163798 filer mongodb insert entry and find entry 2020-04-18 23:48:38 +07:00
bukton fd4576651b add new package mongodb and install lib mongodb 2020-04-18 22:26:57 +07:00
Chris Lu 65919d971d update statik 2020-04-12 17:25:54 -07:00
Chris Lu 295f00cdce update imaging lib 2020-04-12 17:04:16 -07:00
Chris Lu eb16bb0287 go mod cleanup 2020-04-12 16:44:58 -07:00
Chris Lu 1c2e920fa2 fix compilation error 2020-04-12 03:11:19 -07:00
Chris Lu 7f0d87b206 tree structured fs cache 
FsCache for FsNode, wrapping fs.Node
 2020-03-26 00:08:14 -07:00
Chris Lu 7213f446db update grpc 2020-03-13 16:31:40 -07:00
Chris Lu e85da50ad4 remove tikv, move its support to "tikv" branch 2020-03-11 20:59:00 -07:00
Chris Lu 9797420ac8 go mod tidy 2020-03-11 20:34:04 -07:00
Chris Lu ead756981b adjust go mod 2020-03-10 13:35:05 -07:00
Chris Lu 33b3bd467c Revert "HEAD operation changes to fasthttp" 
This reverts commit 58f126fd27.
 2020-02-10 13:43:53 -08:00
Chris Lu 58f126fd27 HEAD operation changes to fasthttp 2020-02-10 09:13:29 -08:00
Chris Lu d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
bs106 ce79a31448 Fix missing leading dot in extension 2019-12-17 09:47:12 +00:00
Chris Lu 0e79a44604 go module 2019-11-29 00:27:51 -08:00
Lei Liu 4c87b222f1
fuse: check mount point available before do runmount 
1. Use more readable mount point information
2. Fix some typos

eg:

$ df -Th
Filesystem              Type            Size  Used Avail Use% Mounted on
localhost:8888:/        fuse.seaweedfs  206G   512  206G   1% /mnt/weedfs

$ mount | grep weedfs
localhost:8888:/ on /mnt/weedfs type fuse.seaweedfs (rw,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-27 22:56:44 +08:00
Chris Lu ba73c053c3 change uuid package 
fix https://github.com/chrislusf/seaweedfs/issues/1126
 2019-11-18 19:24:37 -08:00
Chris Lu c0776aa40f trying to compile 2019-10-23 01:27:08 -07:00
Chris Lu faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
j.laycock baa813ee30 maybe prevent merge conflicts this time? 2019-10-14 16:02:55 +01:00
j.laycock 583dc3afdd maybe make changes merge? 2019-10-14 15:57:40 +01:00
j.laycock 6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
j.laycock 303234ebd5 change go.mod 2019-09-05 13:48:55 +01:00
Chris Lu cb299dfaa2 HCFS: use latest grpc versions, separate hadoop2 and hadoop3 2019-09-01 03:46:51 -07:00
Chris Lu 35d471a818
Revert "use std encoding lib" 2019-08-23 07:59:09 -07:00
divinerapier e798e4e9c5
use std encoding lib 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-08-19 18:07:54 +08:00
Feng Ye fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Feng Ye a435ba2b7f Use master branch of github.com/satori/go.uuid 2019-07-31 14:36:17 +08:00
divinerapier 68a234b3bb use go mod to manager dependencies 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-07-25 13:50:28 +08:00