gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
7052 commits 28 branches 278 tags 135 MiB
Commit graph

4708 commits

Author SHA1 Message Date
Radtoo fa0cfdfb7f Added basic deletion capabilities to fsck. 2022-02-06 17:05:58 +01:00
chrislu 433fde4b18 move error to a separate file 
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
 2022-02-04 22:57:51 -08:00
Chris Lu a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics 
error metrics for filer and store
 2022-02-04 22:35:13 -08:00
Chris Lu 247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors 
add s3api error for copy in file, not directory
 2022-02-04 22:34:34 -08:00
chrislu ced3b89395 add util package 2022-02-04 21:34:58 -08:00
chrislu affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
chrislu 76e297d64f sync call to write file, avoid vif loading error 
fix https://github.com/chrislusf/seaweedfs/issues/2633
 2022-02-04 11:14:04 -08:00
zerospiel f3364fec99 weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting 
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).

Added generic comment rule of traversing methods
 2022-02-04 15:14:48 +03:00
Konstantin Lebedev 9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev 3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Lapshinn Vitaly 6bdc274d4d add s3api error for copy in file, not directory 2022-02-04 03:28:37 +03:00
chrislu 6cf2e7d493 s3: listObjectParts return ErrNoSuchUpload if does not exist 
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
    "Initiator": null,
    "Owner": null,
    "StorageClass": "STANDARD"
}

If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
 2022-02-03 12:34:16 -08:00
zerospiel b54a65ba5a weed/s3api: added new bucket handlers for more compatibility with AWS S3 
Protocol

Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.

Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment

Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
 2022-02-03 17:17:05 +03:00
chrislu 6bee1e9714 [volume.check.disk] fix wrong logic to compare 
fix https://github.com/chrislusf/seaweedfs/issues/2626

minuend - subtrahend
 2022-02-01 07:48:28 -08:00
chrislu 7270067289 2.88 2022-01-30 20:25:26 -08:00
chrislu 84c9bc4389 edge case: old entry was not replicated to remote storage 2022-01-30 20:23:24 -08:00
chrislu b8490fe427 adjust volume count even when not applying the changes 2022-01-28 19:11:46 -08:00
Chris Lu b3f0f170b4
Merge pull request #2618 from divanikus/master 2022-01-28 03:34:52 -08:00
chrislu b9b684194f remove max connection age 
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
 2022-01-27 02:28:22 -08:00
divanikus 67e3fe996a async volumeTierMove 2022-01-26 18:22:31 +03:00
chrislu 62d815d1ca use limited in memory buffer instead of swap file 2022-01-26 00:36:19 -08:00
Chris Lu 9596fce562
Merge pull request #2613 from kmlebedev/masterPromMetrics 
Master prom metrics
 2022-01-25 08:13:14 -08:00
Konstantin Lebedev 5c9259fa3c fix metrics master name 2022-01-25 14:42:47 +05:00
chrislu 7328962009 revert "POSIX: should not delete if a directory is not empty" 
revert 0c75f15062

Reported:

Hi, about commit: 0c75f15062
POSIX: should not delete if a directory is not empty
It should still delete with a command like rm -rf "${path}"/  because it is a forced delete, but now it gets fail to delete non-empty folder: [...]
Can you enable the delete if it is forced?
 2022-01-24 14:21:50 -08:00
Konstantin Lebedev c9952759c4 metrics master is leader 2022-01-24 20:13:07 +05:00
Konstantin Lebedev 28efe31524 new master metrics 2022-01-24 19:09:43 +05:00
chrislu 3bba2124ef use a sliding window of in-memory writable chunks 2022-01-23 23:02:05 -08:00
chrislu 520591e6ea reset swap file chunk after uploading 2022-01-23 18:30:53 -08:00
chrislu e185d90d24 2.87 2022-01-23 16:18:55 -08:00
chrislu 4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu bb6854b972 adjust log 2022-01-22 12:35:09 -08:00
chrislu 2bdd737971 rename file 2022-01-22 08:28:35 -08:00
chrislu 643bbbeb49 rename 2022-01-22 08:27:40 -08:00
chrislu 18ed06b420 use memory when under 16 chunks 2022-01-22 08:11:01 -08:00
chrislu 8e80f3cd65 move upload pipeline locking to a different file 2022-01-22 08:09:55 -08:00
chrislu dde34fa99d rename 2022-01-22 08:08:01 -08:00
chrislu a2aa542370 rename 2022-01-22 08:07:11 -08:00
chrislu c376ccc5a5 swap file based random write large file upload 2022-01-22 08:06:53 -08:00
chrislu b9ae16fbc5 fix memory allocation 2022-01-22 08:05:04 -08:00
chrislu 1ee828b768 refactor 
do not expose internal offset
 2022-01-22 06:34:29 -08:00
chrislu 02d0c12cdd rename 2022-01-22 06:00:10 -08:00
chrislu 5dea5c0449 refactor 2022-01-22 05:59:07 -08:00
chrislu 8aa6bf0bb9 refactoring 2022-01-22 05:40:10 -08:00
chrislu d97bd54e63 just refactoring 2022-01-22 04:18:54 -08:00
chrislu 482014f9da rename file 2022-01-22 03:50:18 -08:00
chrislu 3b4a9addaf rename 2022-01-22 01:46:10 -08:00
chrislu 4acfc098e9 re-order 2022-01-22 01:43:14 -08:00
chrislu e71dcfb3a6 add logging for memory allocation 2022-01-22 01:35:12 -08:00
chrislu 9d0f58c329 skip printing fs configuration 2022-01-21 13:29:47 -08:00
chrislu ce2049cdb6 refactoring, move genFn before saveFn 2022-01-21 12:08:58 -08:00
chrislu e47f63d159 enforce bucket quota 2022-01-21 02:34:42 -08:00
chrislu 6e57d8d0de s3: check bucket usage and adjust read only according to quota 2022-01-21 02:15:27 -08:00
chrislu 606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
chrislu b1063162b6 display bucket quota 2022-01-21 00:55:04 -08:00
chrislu f103491912 s3: list bucket size from weed shell 2022-01-21 00:26:49 -08:00
chrislu 6c7135d77e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-21 00:17:06 -08:00
chrislu ea57654e34 refactoring 2022-01-21 00:16:50 -08:00
Chris Lu 7c66f3b5fb
Merge pull request #2602 from kmlebedev/master_metrics 
master metricsHttpPort
 2022-01-20 09:26:25 -08:00
chrislu b3e526ba95 url should be always using forward slash 2022-01-19 22:16:26 -08:00
Konstantin Lebedev 77c98b657e master metricsHttpPort 2022-01-19 21:43:22 +05:00
chrislu 77362700e1 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix #2593
 2022-01-18 12:04:40 -08:00
chrislu 05c3c3f56b Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-17 23:38:13 -08:00
chrislu 9b77f0054e 2.86 2022-01-17 23:38:03 -08:00
Chris Lu ec254d8a89
Merge pull request #2597 from guol-fnst/gocql_to 
add gocql timeout setting
 2022-01-17 23:35:13 -08:00
chrislu 9274557552 keep dirty pages based on temp file 2022-01-17 23:23:49 -08:00
guol-fnst da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
chrislu c87b8f4c30 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix https://github.com/chrislusf/seaweedfs/issues/2593
 2022-01-17 23:09:37 -08:00
chrislu b2acfd75e9 ensure entry view cache is invalidated 2022-01-17 23:02:30 -08:00
chrislu f4ad63528a wait for reading threads to complete before dropping sealed chunks 2022-01-17 22:24:44 -08:00
chrislu 0a3f95ca01 more logs 2022-01-17 20:41:00 -08:00
chrislu b068bc291d testing with always resetting entry view cache 2022-01-17 20:07:01 -08:00
chrislu 047446d5ca remove extra async execution 2022-01-17 15:50:11 -08:00
chrislu 7bf7af971b more logs 2022-01-17 14:15:10 -08:00
chrislu fc22071a2f more logs 2022-01-17 14:02:37 -08:00
chrislu 381f4e73a0 delete actual reference first 2022-01-17 13:56:47 -08:00
chrislu 0ba88596e8 invalidate filehandle entry view cache 2022-01-17 13:53:30 -08:00
chrislu 1734017ba1 add test 2022-01-17 13:40:41 -08:00
chrislu da7f13e73e Revert "testing skip memory management" 
This reverts commit 6c908352cb.
 2022-01-17 03:21:31 -08:00
chrislu 6c908352cb testing skip memory management 2022-01-17 03:19:24 -08:00
chrislu 77d9993f38 remove unused variables 2022-01-17 03:19:11 -08:00
chrislu f710d5ffca a little speed up 2022-01-17 03:19:00 -08:00
chrislu fc0628c038 working 2022-01-17 01:53:56 -08:00
chrislu 1bd6d289d4 better locking on file handle 2022-01-15 05:45:29 -08:00
chrislu 2bfeb5d1c8 add filer to iam option 2022-01-15 03:37:52 -08:00
chrislu b17c426e99 weed server: optionally start IAM service 
related to https://github.com/chrislusf/seaweedfs/issues/2560
 2022-01-13 22:49:49 -08:00
chrislu 3c8b74318e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-13 13:03:07 -08:00
chrislu 8907e6a40a add more help messages 2022-01-13 13:03:04 -08:00
banjiaojuhao 45e9c83421 padding zero for sparse file 2022-01-13 22:21:22 +08:00
chrislu fe5b9e39cc POSIX: check permission when removing items 2022-01-13 02:07:39 -08:00
chrislu 1453263b63 remove dead code 2022-01-13 02:02:04 -08:00
chrislu e69c374956 minor 2022-01-13 02:01:53 -08:00
chrislu 9b954dc0d4 adjust make file 2022-01-13 01:33:13 -08:00
chrislu f2847f1266 POSIX: check deletion permission 2022-01-12 23:58:11 -08:00
chrislu 0c75f15062 POSIX: should not delete if a directory is not empty 2022-01-12 23:57:54 -08:00
chrislu de27058d0b POSIX: differentiate device and char device 2022-01-12 21:45:38 -08:00
chrislu d400a11832 POSIX: adjust source file ctime 
SeaweedFS uses mtime as ctime
 2022-01-12 21:45:18 -08:00
chrislu b44f05a2d0 POSIX: change timestamp on each attribute change 2022-01-12 19:31:25 -08:00
chrislu 15c01d8b7f add some notes 2022-01-12 15:04:48 -08:00
chrislu 107a4884a8 shell: tighter memory allocation 2022-01-12 14:59:29 -08:00
chrislu fec8428fd8 POSIX: different inode for same named different file types 2022-01-12 11:51:13 -08:00
chrislu e82ad60122 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-12 11:07:49 -08:00
chrislu caf0a3486b POSIX: adjust ctime for file truncate 2022-01-12 11:07:39 -08:00
Konstantin Lebedev edb753ab4d https://github.com/chrislusf/seaweedfs/issues/2583 2022-01-12 16:04:59 +05:00
chrislu adfd54e7c4 fix compilation 2022-01-12 01:24:24 -08:00
chrislu 6cc92817dc add logs for request mode 2022-01-12 01:13:19 -08:00
chrislu 826a7b307e master: remove hard coded filer settings in master.toml 
fix https://github.com/chrislusf/seaweedfs/issues/2529
 2022-01-12 01:11:25 -08:00
chrislu cd1ad88f30 POSIX: check name is too long ENAMETOOLONG 2022-01-12 00:16:00 -08:00
chrislu 2dcb8cb93b POSIX: ensure file and directory inodes are different 
this is just an in memory representation.

POSIX wants different inode numbers for the same named file or directory.
 2022-01-11 23:44:48 -08:00
chrislu 5bb37d5905 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 23:29:12 -08:00
chrislu 10ecf80ca1 add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
Kyle Sanderson 9e012001be
filer.copy: don't crash when volume creation fails 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
 2022-01-11 22:22:39 -08:00
chrislu 1a7d5b5b5e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 12:24:56 -08:00
chrislu 41daecfdca Update mount_std.go 2022-01-11 12:23:12 -08:00
chrislu 2d0ccc4d34 add logs 2022-01-11 12:23:01 -08:00
Chris Lu abe5da7d2c
Merge pull request #2575 from Radtoo/fix_paths2 
Fix paths2
 2022-01-11 12:04:30 -08:00
chrislu b8fbf19e9a mount: rename follow POSIX 2022-01-11 03:23:03 -08:00
chrislu 6a12520a96 fix logging 2022-01-10 01:00:11 -08:00
chrislu cbc055dc2b mount: file fsync 
fix https://github.com/chrislusf/seaweedfs/issues/2561
 2022-01-10 00:52:16 -08:00
chrislu 19555385f7 2.85 2022-01-09 19:30:23 -08:00
Radtoo 389002f195 Using positional arguments rather than option flag to enable better shell usage 2022-01-08 16:52:12 +01:00
Radtoo fba1efb77a Now works with a single file too 
Parsing removed from doFixOneVolume

Needle init removed from runFix
 2022-01-08 16:31:53 +01:00
chrislu 110d5a5233 support fixing a collection of volumes, or volumes under one directory 2022-01-07 14:52:16 -08:00
chrislu 60dc450091 skip fixing read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2562
 2022-01-06 09:52:28 -08:00
chrislu 3df8f96117 avoid changing inode 2022-01-06 01:36:11 -08:00
chrislu 67b0645808 mount: need to change entry name after renaming 2022-01-05 21:27:41 -08:00
chrislu 4de060daa6 mount: skip special character in the filenames 
fix https://github.com/chrislusf/seaweedfs/issues/2559
 2022-01-05 03:57:24 -08:00
chrislu e76105e2ab fix auth permission checking 2022-01-03 21:05:20 -08:00
chrislu a7887166cf wildcard prefix to restrict access to directories in s3 bucket 
https://github.com/chrislusf/seaweedfs/discussions/2551
 2022-01-03 15:39:36 -08:00
chrislu 5799a20f71 2.84 2022-01-02 17:05:19 -08:00
Chris Lu 42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158 
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
 2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst 1cd3b6b4e1 BUGFIX: security.toml contained wrong keys 2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
 2021-12-30 14:45:27 +01:00
chrislu 34742be029 remove duplicated metadata subscription in filer 
https://github.com/chrislusf/seaweedfs/issues/2545
 2021-12-30 01:51:52 -08:00
chrislu 5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
chrislu fb434318e3 dynamically adjust connection timeout 
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:44:39 -08:00
chrislu 5788bf2270 s3: increase timeout limit 
https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:21:02 -08:00
Sebastian Kurfuerst fcc09cef6f Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst d156d410ef rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst eda4c43a08 fix typo in error message 2021-12-29 12:38:14 +01:00
chrislu 498661e3bb mount: remove limits on number of parallel requests 2021-12-28 17:41:01 -08:00
chrislu 9a00c17555 reader: avoid wrong pattern detection due to lock waiting 2021-12-28 16:30:33 -08:00
chrislu f7a6f6b4c0 if this is enabled, there are some "bus error" with git clone 2021-12-28 13:44:52 -08:00
chrislu 0da2dfd640 fuse: change to direct io mode 
before and after:

chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.86s system 84% cpu 4.561 total
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.22s system 83% cpu 3.857 total
 2021-12-28 12:22:56 -08:00
chrislu 3fd4da34a4 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-28 01:10:38 -08:00
chrislu 2422556456 monitor write pattern: avoid timing due to locking 2021-12-28 01:10:35 -08:00
chenkai 47c30e3add filer list entries use context to break job 2021-12-28 15:03:41 +08:00
chrislu 80db8b13d8 bug: cleanup function was called twice 2021-12-27 20:53:02 -08:00
chrislu 67b723f74e Filer Server API support fsync 
fix https://github.com/chrislusf/seaweedfs/issues/2528
 2021-12-26 17:28:47 -08:00