Commit graph

9452 commits

Author SHA1 Message Date
zemul f352616b7d
Volume range read use sync.pool (#4422) 2023-04-24 07:00:48 -07:00
chrislu 18686b7375 3.47 2023-04-23 22:45:41 -07:00
zemul 68f1cc34a5
[mount]Fix copy file range (#4407)
* fix:mount deadlock

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-23 22:34:38 -07:00
chrislu e1b92bc701 reset local file content if remote file is updated
fix https://github.com/seaweedfs/seaweedfs/issues/4289
2023-04-22 22:32:21 -07:00
chrislu ff7b6d779e avoid overwriting variables
fix https://github.com/seaweedfs/seaweedfs/issues/4365
2023-04-22 11:49:59 -07:00
wusong 2e240704ab
Writables inconsistency (#4417)
fix: inconsistent read and write permissions between master and volume server

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-04-21 00:14:41 -07:00
wusong 19245dde50
mount: add retry for read only case (#4416)
* mount: add retry for read only case

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

* add new util retry function for mount

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

* change error list param

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

---------

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-04-21 00:09:31 -07:00
SmsS4 4131874fa8
use env for flags wtih dot in name (#4415) 2023-04-20 09:33:27 -07:00
Damiano Albani 9f55c7c90e
Rclone storage backend (#4402)
* Add Rclone storage backend

* Support templating the name of files stored via Rclone

* Enable Rclone accounting

* Remove redundant type conversion

* Provide progress information for Rclone download/upload operations

* Log error when Rclone can't instantiate filesystem

* Remove filename templating functionality for Rclone storage

To (maybe) be later reintroduced as a generic functionality for all
storage backends.

* Remove S3 specific check

* Move Rclone config initialisation to init() method
2023-04-18 14:12:12 -07:00
wusong a408b46d95
compilation fail (#4414)
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-04-18 00:36:55 -07:00
chrislu 0a22eea55d collect ec shard from multiple locations
fix https://github.com/seaweedfs/seaweedfs/issues/4365
2023-04-17 22:56:21 -07:00
chrislu d50d736e68 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-04-17 22:01:43 -07:00
chrislu ae9388723f adjust error message 2023-04-17 22:01:39 -07:00
dependabot[bot] 31db90942b
build(deps): bump github.com/aws/aws-sdk-go from 1.44.219 to 1.44.244 (#4412)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.219 to 1.44.244.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.219...v1.44.244)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-17 09:19:04 -07:00
dependabot[bot] 839dd438bc
build(deps): bump github.com/cenkalti/backoff/v4 from 4.2.0 to 4.2.1 (#4413)
Bumps [github.com/cenkalti/backoff/v4](https://github.com/cenkalti/backoff) from 4.2.0 to 4.2.1.
- [Release notes](https://github.com/cenkalti/backoff/releases)
- [Commits](https://github.com/cenkalti/backoff/compare/v4.2.0...v4.2.1)

---
updated-dependencies:
- dependency-name: github.com/cenkalti/backoff/v4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-17 09:18:48 -07:00
dependabot[bot] bf3d0c4850
build(deps): bump github.com/prometheus/client_golang from 1.14.0 to 1.15.0 (#4411)
build(deps): bump github.com/prometheus/client_golang

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.14.0...v1.15.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-17 09:18:34 -07:00
dependabot[bot] ea5bbf30bb
build(deps): bump github.com/lib/pq from 1.10.7 to 1.10.8 (#4410)
Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.10.7 to 1.10.8.
- [Release notes](https://github.com/lib/pq/releases)
- [Commits](https://github.com/lib/pq/compare/v1.10.7...v1.10.8)

---
updated-dependencies:
- dependency-name: github.com/lib/pq
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-17 09:18:24 -07:00
dependabot[bot] a9b4be50af
build(deps): bump github.com/tikv/client-go/v2 from 2.0.6 to 2.0.7 (#4409)
Bumps [github.com/tikv/client-go/v2](https://github.com/tikv/client-go) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/tikv/client-go/releases)
- [Commits](https://github.com/tikv/client-go/compare/v2.0.6...v2.0.7)

---
updated-dependencies:
- dependency-name: github.com/tikv/client-go/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-17 09:18:13 -07:00
chrislu 8ecdf958ab 3.46 2023-04-16 21:57:43 -07:00
Chengyu Liu 0f653d2153
update fuse client log which should output the fh of FileHandle. (#4404) 2023-04-15 22:15:16 -07:00
zemul 0122e022ea
Mount concurrent read (#4400)
* fix:mount deadlock

* feat: concurrent read

* fix

* Remove useless code

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-13 22:32:45 -07:00
Konstantin Lebedev 5614ad0000
fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter (#4399)
* fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter

* fix list with delimiter and start token

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-04-12 16:53:49 -07:00
samcday e60a7af644
fix(chart): use filer data PVC correctly (#4398) 2023-04-12 09:48:44 -07:00
Konstantin Lebedev 88b2bbadd1
try fix s3test test_bucket_listv2_delimiter_prefix (#4396) 2023-04-12 08:47:20 -07:00
Konstantin Lebedev 095cdb0c7f
fix test prefix is a directory (#4393)
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-04-12 00:08:13 -07:00
Konstantin Lebedev 44ad07276c
s3 fix get list of dir object key with slash suffix (#4391)
* s3 fix get list of dir object key with slash suffix
https://github.com/seaweedfs/seaweedfs/issues/3086

* list only entry dir eq prefix

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-04-11 23:52:35 -07:00
Kyle Petryszak ccc030b245
Fixing a service deployment bug (#4392) 2023-04-11 20:05:09 -07:00
Konstantin Lebedev 130bc3e668
s3 fix get fake dir object key (#4390) 2023-04-11 07:36:22 -07:00
dependabot[bot] d1ce89950e
build(deps): bump github.com/rabbitmq/amqp091-go from 1.7.0 to 1.8.0 (#4387)
Bumps [github.com/rabbitmq/amqp091-go](https://github.com/rabbitmq/amqp091-go) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/rabbitmq/amqp091-go/releases)
- [Changelog](https://github.com/rabbitmq/amqp091-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rabbitmq/amqp091-go/compare/v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: github.com/rabbitmq/amqp091-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-10 12:25:30 -07:00
Matt da1d3b5a33
adjusted regex to be from 1 to 255 for the value (#4377)
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation

* adjusted regex to be from 1 to 255 for the value

* added better error message

* fixed regex
2023-04-10 12:24:38 -07:00
dependabot[bot] b5b5c67cf6
build(deps): bump gocloud.dev/pubsub/natspubsub from 0.28.0 to 0.29.0 (#4389)
Bumps [gocloud.dev/pubsub/natspubsub](https://github.com/google/go-cloud) from 0.28.0 to 0.29.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.28.0...v0.29.0)

---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/natspubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-10 12:23:17 -07:00
dependabot[bot] 1d6a348d55
build(deps): bump github.com/schollz/progressbar/v3 from 3.13.0 to 3.13.1 (#4386)
build(deps): bump github.com/schollz/progressbar/v3

Bumps [github.com/schollz/progressbar/v3](https://github.com/schollz/progressbar) from 3.13.0 to 3.13.1.
- [Release notes](https://github.com/schollz/progressbar/releases)
- [Commits](https://github.com/schollz/progressbar/compare/v3.13.0...v3.13.1)

---
updated-dependencies:
- dependency-name: github.com/schollz/progressbar/v3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-10 12:22:59 -07:00
dependabot[bot] 04a9dc898f
build(deps): bump golang.org/x/tools from 0.7.0 to 0.8.0 (#4385)
Bumps [golang.org/x/tools](https://github.com/golang/tools) from 0.7.0 to 0.8.0.
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](https://github.com/golang/tools/compare/v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: golang.org/x/tools
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-10 12:22:46 -07:00
samcday adf80dc3f6
fix(chart): use hostpath filer persistence when deprecated enablePVC is unset (#4384) 2023-04-10 06:32:31 -07:00
Matt 2d213d87e1
File Path Configuration TTL Validation (#4376)
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation
2023-04-04 22:07:17 -07:00
chrislu efef6e94bf use UPSERT for postgres style databases 2023-04-04 11:52:36 -07:00
dependabot[bot] d4dec6c730
build(deps): bump google.golang.org/grpc from 1.53.0 to 1.54.0 (#4366)
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.53.0 to 1.54.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.53.0...v1.54.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-03 11:15:08 -07:00
dependabot[bot] 080bfa7052
build(deps): bump modernc.org/sqlite from 1.21.0 to 1.21.1 (#4370)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.21.0 to 1.21.1.
- [Release notes](https://gitlab.com/cznic/sqlite/tags)
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.21.0...v1.21.1)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-03 11:12:29 -07:00
dependabot[bot] 0f2a8b0af1
build(deps): bump cloud.google.com/go/pubsub from 1.28.0 to 1.30.0 (#4367)
Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.28.0 to 1.30.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.28.0...pubsub/v1.30.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/pubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-03 11:12:22 -07:00
dependabot[bot] 5aa89844b1
build(deps): bump google.golang.org/protobuf from 1.29.1 to 1.30.0 (#4369)
Bumps google.golang.org/protobuf from 1.29.1 to 1.30.0.

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-03 11:12:12 -07:00
dependabot[bot] 671ee6af6d
build(deps): bump github.com/linxGnu/grocksdb from 1.7.15 to 1.7.16 (#4368)
Bumps [github.com/linxGnu/grocksdb](https://github.com/linxGnu/grocksdb) from 1.7.15 to 1.7.16.
- [Release notes](https://github.com/linxGnu/grocksdb/releases)
- [Commits](https://github.com/linxGnu/grocksdb/compare/v1.7.15...v1.7.16)

---
updated-dependencies:
- dependency-name: github.com/linxGnu/grocksdb
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-03 11:12:05 -07:00
chrislu 422bfaed69 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-04-02 22:53:02 -07:00
chrislu b9ab65dba5 3.45 2023-04-02 22:52:58 -07:00
wusong 2b82d0f871
style: clerical error (#4361) 2023-03-31 07:36:21 -07:00
LHHDZ 8e80ce9476
fix key corrupt when fs.configure copy path trie (#4353)
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-29 11:29:44 -07:00
dependabot[bot] ff649d615d
build(deps): bump cloud.google.com/go/storage from 1.30.0 to 1.30.1 (#4342)
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.30.0 to 1.30.1.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.30.0...spanner/v1.30.1)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-27 10:46:30 -07:00
dependabot[bot] 3e69a3aac5
build(deps): bump github.com/golang/protobuf from 1.5.2 to 1.5.3 (#4344)
Bumps [github.com/golang/protobuf](https://github.com/golang/protobuf) from 1.5.2 to 1.5.3.
- [Release notes](https://github.com/golang/protobuf/releases)
- [Commits](https://github.com/golang/protobuf/compare/v1.5.2...v1.5.3)

---
updated-dependencies:
- dependency-name: github.com/golang/protobuf
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-27 10:32:27 -07:00
dependabot[bot] 04190f9951
build(deps): bump go.mongodb.org/mongo-driver from 1.11.1 to 1.11.3 (#4340)
Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.11.1 to 1.11.3.
- [Release notes](https://github.com/mongodb/mongo-go-driver/releases)
- [Commits](https://github.com/mongodb/mongo-go-driver/compare/v1.11.1...v1.11.3)

---
updated-dependencies:
- dependency-name: go.mongodb.org/mongo-driver
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-27 10:32:16 -07:00
dependabot[bot] d6ecde8a96
build(deps): bump google.golang.org/api from 0.112.0 to 0.114.0 (#4343)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.112.0 to 0.114.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.112.0...v0.114.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-27 10:32:03 -07:00
dependabot[bot] ddc5234f85
build(deps): bump github.com/gocql/gocql from 0.0.0-20210707082121-9a3953d1826d to 1.3.2 (#4341)
build(deps): bump github.com/gocql/gocql

Bumps [github.com/gocql/gocql](https://github.com/gocql/gocql) from 0.0.0-20210707082121-9a3953d1826d to 1.3.2.
- [Release notes](https://github.com/gocql/gocql/releases)
- [Commits](https://github.com/gocql/gocql/commits/v1.3.2)

---
updated-dependencies:
- dependency-name: github.com/gocql/gocql
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-27 10:31:54 -07:00