gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9107 commits 28 branches 278 tags 135 MiB
Commit graph

9107 commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
chrislu 5423790b2c add back previous chunk upload selection algo 2023-01-03 22:50:39 -08:00
chrislu 7bdae5172e batch delete EC needles 
fix https://github.com/seaweedfs/seaweedfs/issues/4107
 2023-01-03 22:05:26 -08:00
chrislu 8aec430df7 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-01-03 12:13:54 -08:00
chrislu c87a3ffce3 avoid hard coded versions 2023-01-03 12:13:52 -08:00
zemul 6b4c033431
add mount log (#4101) 
* filer.backup use replication.source.filer

* add mount log

* Revert "filer.backup use replication.source.filer"

This reverts commit 07bf6f956c.

* fix

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-01-03 00:00:45 -08:00
chrislu 48f2edc065 3.38 2023-01-02 23:23:02 -08:00
Chris Lu d4566d4aaa
more solid weed mount (#4089) 
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
 2023-01-02 23:20:45 -08:00
dependabot[bot] 367353b936
build(deps): bump github.com/aws/aws-sdk-go from 1.44.167 to 1.44.171 (#4100) 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.167 to 1.44.171.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.167...v1.44.171)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-02 12:24:46 -08:00
dependabot[bot] de787ce4ab
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.41.0 to 3.42.1 (#4099) 
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.41.0 to 3.42.1.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.41.0...v3.42.1)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-02 12:24:28 -08:00
dependabot[bot] 04bba1a6b2
build(deps): bump modernc.org/sqlite from 1.20.0 to 1.20.1 (#4098) 
---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-02 12:24:15 -08:00
CommanderRoot c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094) 2023-01-01 05:07:53 -08:00
CommanderRoot c2280e94cf
filer.store.mysql: Replace deprecated upsert syntax (#4096) 2023-01-01 05:06:57 -08:00
CommanderRoot 265a56630b
filer.store.mysql: Escape table columns in SQL query (#4095) 2023-01-01 05:06:41 -08:00
Guo Lei ed5f3f073b
add more help message, in case of misunderstanding (#4092) 2022-12-29 07:05:05 -08:00
Guo Lei d8cfa1552b
support enable/disable vacuum (#4087) 
* stop vacuum

* suspend/resume vacuum

* remove unused code

* rename

* rename param
 2022-12-28 01:36:44 -08:00
dependabot[bot] 84492976c9
build(deps): bump github.com/aws/aws-sdk-go from 1.44.162 to 1.44.167 (#4085) 2022-12-26 11:48:35 -08:00
dependabot[bot] bd176359cb
build(deps): bump github.com/prometheus/procfs from 0.8.0 to 0.9.0 (#4084) 2022-12-26 11:48:17 -08:00
dependabot[bot] b2aa1a23ff
build(deps): bump github.com/arangodb/go-driver from 1.4.0 to 1.4.1 (#4083) 2022-12-26 11:47:52 -08:00
Guo Lei 5f993a5f20
Fix ide warning (#4082) 
fix warning message from IDE
"redundant type from array, slice, or map composite literal"
 2022-12-26 00:03:47 -08:00
fibbanachi 3901f2da4d
Add logtostderr option in helm chart (#4080) 2022-12-23 07:36:15 -08:00
Konstantin Lebedev 569bc7b54d
fs.verify needles status of file (#4060) 
* fs.verify needles status of file

* fix parse args

* refactor logging

* set modifyTimeAgoAtSec
 2022-12-21 15:54:17 -08:00
chrislu 77b7c9f7cc lock and unlock chunks for read 2022-12-21 14:15:35 -08:00
chrislu d1797deccb remove duplicated logic of removing from writableChunks 2022-12-21 14:15:11 -08:00
chrislu 4878e0fd46 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-20 19:47:23 -08:00
chrislu 6ede19e825 add a simple file replication progress bar 2022-12-20 19:47:21 -08:00
Naden 27d187eed9
Fixes 4074: S3 Gateway fails to start when installed via Helm (#4075) 
Fixes #4074: S3 Gateway fails to start when installed via Helm

Co-authored-by: Naden <naden@harana.com>
 2022-12-20 17:24:54 -08:00
Naden 3b90eb1911
Fixes: 4076: Host path prefix should be configurable (#4077) 
Co-authored-by: Naden <naden@harana.com>
 2022-12-20 17:24:12 -08:00
chrislu f7beba8515 unlock before submitting the uploading jobs 2022-12-20 12:52:30 -08:00
chrislu 28d479e5c0 mount: adjust locking for upload pipeline 2022-12-19 15:07:22 -08:00
chrislu f9fb3cd1fc Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-19 11:32:00 -08:00
chrislu 6c7fe40305 filer sink retries reading file chunks, skipping missing chunks 
if the file chunk is not available during replication time, the file is skipped
 2022-12-19 11:31:58 -08:00
chrislu 3fc3e7083c metadata skip reading un-available logs 
this means the volumes for metadata histories have been lost.
 2022-12-19 11:30:50 -08:00
dependabot[bot] a1ffa03d75
build(deps): bump google.golang.org/api from 0.104.0 to 0.105.0 (#4067) 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.104.0 to 0.105.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.104.0...v0.105.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 10:21:20 -08:00
dependabot[bot] 3886c31ed8
build(deps): bump github.com/aws/aws-sdk-go from 1.44.146 to 1.44.162 (#4069) 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.146 to 1.44.162.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.146...v1.44.162)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 09:22:36 -08:00
dependabot[bot] 9fdf98b826
build(deps): bump cloud.google.com/go/pubsub from 1.27.1 to 1.28.0 (#4068) 
Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.27.1 to 1.28.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.27.1...pubsub/v1.28.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/pubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 09:22:23 -08:00
dependabot[bot] c7a28b20b5
build(deps): bump actions/checkout from 3.1.0 to 3.2.0 (#4066) 
Bumps [actions/checkout](https://github.com/actions/checkout) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](93ea575cb5...755da8c3cf)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 09:22:12 -08:00
dependabot[bot] 6c5d44a5c4
build(deps): bump actions/setup-go from 3.4.0 to 3.5.0 (#4065) 
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](d0a58c1c4d...6edd4406fa)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 09:21:49 -08:00
dependabot[bot] ae847c2780
build(deps): bump actions/dependency-review-action from 3.0.1 to 3.0.2 (#4064) 
Bumps [actions/dependency-review-action](https://github.com/actions/dependency-review-action) from 3.0.1 to 3.0.2.
- [Release notes](https://github.com/actions/dependency-review-action/releases)
- [Commits](11310527b4...0ff3da6f81)

---
updated-dependencies:
- dependency-name: actions/dependency-review-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 09:21:41 -08:00
dependabot[bot] 3b85dc5fbb
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.40.1 to 3.41.0 (#4070) 
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.40.1 to 3.41.0.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.40.1...v3.41.0)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 09:21:30 -08:00
dependabot[bot] b8a4ed455f
build(deps): bump golang.org/x/tools from 0.3.0 to 0.4.0 (#4071) 
Bumps [golang.org/x/tools](https://github.com/golang/tools) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](https://github.com/golang/tools/compare/v0.3.0...v0.4.0)

---
updated-dependencies:
- dependency-name: golang.org/x/tools
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 09:21:20 -08:00
lfhy 038bab3726
add -filer.path to webdav command (#4061) 2022-12-17 20:07:02 -08:00
chrislu 8a40fa8993 more detailed logs 2022-12-17 13:18:35 -08:00
LHHDZ f84c13dabd
fix response not allowed (#4057) 2022-12-13 05:48:50 -08:00
dependabot[bot] 693e07b509
build(deps): bump golang.org/x/net from 0.2.0 to 0.4.0 (#4055) 2022-12-12 08:57:22 -08:00
dependabot[bot] 1949944e18
build(deps): bump google.golang.org/api from 0.103.0 to 0.104.0 (#4054) 2022-12-12 08:57:11 -08:00
dependabot[bot] 6c8a3d57e4
build(deps): bump go.mongodb.org/mongo-driver from 1.11.0 to 1.11.1 (#4053) 2022-12-12 08:56:35 -08:00
dependabot[bot] d88199c8b0
build(deps): bump github.com/tikv/client-go/v2 from 2.0.2 to 2.0.3 (#4052) 2022-12-12 08:56:23 -08:00
dependabot[bot] 21f77384f0
build(deps): bump github.com/go-redsync/redsync/v4 from 4.6.0 to 4.7.1 (#4051) 2022-12-12 08:56:00 -08:00
LHHDZ bb289d7d69
sort tags by key (#4047) 
Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-12-11 22:49:57 -08:00
chrislu 438146249f 3.37 2022-12-11 21:08:06 -08:00