Commit graph

6817 commits

Author SHA1 Message Date
chrislu 826a7b307e master: remove hard coded filer settings in master.toml
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
chrislu cd1ad88f30 POSIX: check name is too long ENAMETOOLONG 2022-01-12 00:16:00 -08:00
chrislu 2dcb8cb93b POSIX: ensure file and directory inodes are different
this is just an in memory representation.

POSIX wants different inode numbers for the same named file or directory.
2022-01-11 23:44:48 -08:00
chrislu 5bb37d5905 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 23:29:12 -08:00
chrislu 10ecf80ca1 add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
Chris Lu dbc6ed6832
Merge pull request #2579 from KyleSanderson/patch-1
filer.copy: don't crash when volume creation fails
2022-01-11 22:56:14 -08:00
Kyle Sanderson 9e012001be
filer.copy: don't crash when volume creation fails
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
2022-01-11 22:22:39 -08:00
chrislu 1a7d5b5b5e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 12:24:56 -08:00
chrislu 01ccd6778e sync with fuse 1.2.2 2022-01-11 12:24:54 -08:00
chrislu 41daecfdca Update mount_std.go 2022-01-11 12:23:12 -08:00
chrislu 2d0ccc4d34 add logs 2022-01-11 12:23:01 -08:00
Chris Lu abe5da7d2c
Merge pull request #2575 from Radtoo/fix_paths2
Fix paths2
2022-01-11 12:04:30 -08:00
chrislu b8fbf19e9a mount: rename follow POSIX 2022-01-11 03:23:03 -08:00
chrislu 6a12520a96 fix logging 2022-01-10 01:00:11 -08:00
chrislu cbc055dc2b mount: file fsync
fix https://github.com/chrislusf/seaweedfs/issues/2561
2022-01-10 00:52:16 -08:00
chrislu 19555385f7 2.85 2022-01-09 19:30:23 -08:00
Radtoo 389002f195 Using positional arguments rather than option flag to enable better shell usage 2022-01-08 16:52:12 +01:00
Radtoo fba1efb77a Now works with a single file too
Parsing removed from doFixOneVolume

Needle init removed from runFix
2022-01-08 16:31:53 +01:00
Chris Lu 3b1dc85c6f
Merge pull request #2564 from chrislusf/dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.16.1
Bump protobuf-java from 3.9.1 to 3.16.1 in /other/java/client
2022-01-07 15:16:21 -08:00
chrislu 110d5a5233 support fixing a collection of volumes, or volumes under one directory 2022-01-07 14:52:16 -08:00
dependabot[bot] 46237ea31a
Bump protobuf-java from 3.9.1 to 3.16.1 in /other/java/client
Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.9.1 to 3.16.1.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/master/generate_changelog.py)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.9.1...v3.16.1)

---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-07 22:37:10 +00:00
chrislu 60dc450091 skip fixing read only volumes
fix https://github.com/chrislusf/seaweedfs/issues/2562
2022-01-06 09:52:28 -08:00
chrislu 3df8f96117 avoid changing inode 2022-01-06 01:36:11 -08:00
chrislu 67b0645808 mount: need to change entry name after renaming 2022-01-05 21:27:41 -08:00
chrislu 4de060daa6 mount: skip special character in the filenames
fix https://github.com/chrislusf/seaweedfs/issues/2559
2022-01-05 03:57:24 -08:00
chrislu e76105e2ab fix auth permission checking 2022-01-03 21:05:20 -08:00
chrislu a7887166cf wildcard prefix to restrict access to directories in s3 bucket
https://github.com/chrislusf/seaweedfs/discussions/2551
2022-01-03 15:39:36 -08:00
chrislu 5799a20f71 2.84 2022-01-02 17:05:19 -08:00
Chris Lu 077f831749
Merge pull request #2550 from chrislusf/metadata_follow_with_client_id
add client id for all metadata listening clients
2022-01-02 01:11:27 -08:00
Chris Lu 42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst 99abddf376 FEATURE: add test setup for running the CephFS S3 compatibility suite 2021-12-31 22:07:49 +01:00
Sebastian Kurfuerst c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst 1cd3b6b4e1 BUGFIX: security.toml contained wrong keys 2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
chrislu 34742be029 remove duplicated metadata subscription in filer
https://github.com/chrislusf/seaweedfs/issues/2545
2021-12-30 01:51:52 -08:00
chrislu 5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
chrislu fb434318e3 dynamically adjust connection timeout
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:44:39 -08:00
chrislu 5788bf2270 s3: increase timeout limit
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
Sebastian Kurfuerst fcc09cef6f Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst d156d410ef rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst eda4c43a08 fix typo in error message 2021-12-29 12:38:14 +01:00
chrislu d351541757 upgrade fuse version 2021-12-28 18:18:54 -08:00
chrislu 498661e3bb mount: remove limits on number of parallel requests 2021-12-28 17:41:01 -08:00
chrislu 9a00c17555 reader: avoid wrong pattern detection due to lock waiting 2021-12-28 16:30:33 -08:00
chrislu f7a6f6b4c0 if this is enabled, there are some "bus error" with git clone 2021-12-28 13:44:52 -08:00
chrislu 0da2dfd640 fuse: change to direct io mode
before and after:

chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.86s system 84% cpu 4.561 total
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.22s system 83% cpu 3.857 total
2021-12-28 12:22:56 -08:00
chrislu 3fd4da34a4 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-28 01:10:38 -08:00
chrislu 2422556456 monitor write pattern: avoid timing due to locking 2021-12-28 01:10:35 -08:00
Chris Lu 6c8caea09c
Merge pull request #2539 from ck789987/filer-list-use-context
filer list entries use context to break job
2021-12-27 23:47:51 -08:00