Commit graph

6953 commits

Author SHA1 Message Date
chrislu cbc055dc2b mount: file fsync
fix https://github.com/chrislusf/seaweedfs/issues/2561
2022-01-10 00:52:16 -08:00
chrislu 19555385f7 2.85 2022-01-09 19:30:23 -08:00
Radtoo 389002f195 Using positional arguments rather than option flag to enable better shell usage 2022-01-08 16:52:12 +01:00
Radtoo fba1efb77a Now works with a single file too
Parsing removed from doFixOneVolume

Needle init removed from runFix
2022-01-08 16:31:53 +01:00
Chris Lu 3b1dc85c6f
Merge pull request #2564 from chrislusf/dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.16.1
Bump protobuf-java from 3.9.1 to 3.16.1 in /other/java/client
2022-01-07 15:16:21 -08:00
chrislu 110d5a5233 support fixing a collection of volumes, or volumes under one directory 2022-01-07 14:52:16 -08:00
dependabot[bot] 46237ea31a
Bump protobuf-java from 3.9.1 to 3.16.1 in /other/java/client
Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.9.1 to 3.16.1.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/master/generate_changelog.py)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.9.1...v3.16.1)

---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-07 22:37:10 +00:00
chrislu 60dc450091 skip fixing read only volumes
fix https://github.com/chrislusf/seaweedfs/issues/2562
2022-01-06 09:52:28 -08:00
chrislu 3df8f96117 avoid changing inode 2022-01-06 01:36:11 -08:00
chrislu 67b0645808 mount: need to change entry name after renaming 2022-01-05 21:27:41 -08:00
chrislu 4de060daa6 mount: skip special character in the filenames
fix https://github.com/chrislusf/seaweedfs/issues/2559
2022-01-05 03:57:24 -08:00
chrislu e76105e2ab fix auth permission checking 2022-01-03 21:05:20 -08:00
chrislu a7887166cf wildcard prefix to restrict access to directories in s3 bucket
https://github.com/chrislusf/seaweedfs/discussions/2551
2022-01-03 15:39:36 -08:00
chrislu 5799a20f71 2.84 2022-01-02 17:05:19 -08:00
Chris Lu 077f831749
Merge pull request #2550 from chrislusf/metadata_follow_with_client_id
add client id for all metadata listening clients
2022-01-02 01:11:27 -08:00
Chris Lu 42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst 99abddf376 FEATURE: add test setup for running the CephFS S3 compatibility suite 2021-12-31 22:07:49 +01:00
Sebastian Kurfuerst c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst 1cd3b6b4e1 BUGFIX: security.toml contained wrong keys 2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
chrislu 34742be029 remove duplicated metadata subscription in filer
https://github.com/chrislusf/seaweedfs/issues/2545
2021-12-30 01:51:52 -08:00
chrislu 5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
chrislu fb434318e3 dynamically adjust connection timeout
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:44:39 -08:00
chrislu 5788bf2270 s3: increase timeout limit
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
Sebastian Kurfuerst fcc09cef6f Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst d156d410ef rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst eda4c43a08 fix typo in error message 2021-12-29 12:38:14 +01:00
chrislu d351541757 upgrade fuse version 2021-12-28 18:18:54 -08:00
chrislu 498661e3bb mount: remove limits on number of parallel requests 2021-12-28 17:41:01 -08:00
chrislu 9a00c17555 reader: avoid wrong pattern detection due to lock waiting 2021-12-28 16:30:33 -08:00
chrislu f7a6f6b4c0 if this is enabled, there are some "bus error" with git clone 2021-12-28 13:44:52 -08:00
chrislu 0da2dfd640 fuse: change to direct io mode
before and after:

chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.86s system 84% cpu 4.561 total
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.22s system 83% cpu 3.857 total
2021-12-28 12:22:56 -08:00
chrislu 3fd4da34a4 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-28 01:10:38 -08:00
chrislu 2422556456 monitor write pattern: avoid timing due to locking 2021-12-28 01:10:35 -08:00
Chris Lu 6c8caea09c
Merge pull request #2539 from ck789987/filer-list-use-context
filer list entries use context to break job
2021-12-27 23:47:51 -08:00
chenkai 47c30e3add filer list entries use context to break job 2021-12-28 15:03:41 +08:00
chrislu 80db8b13d8 bug: cleanup function was called twice 2021-12-27 20:53:02 -08:00
chrislu 67b723f74e Filer Server API support fsync
fix https://github.com/chrislusf/seaweedfs/issues/2528
2021-12-26 17:28:47 -08:00
chrislu 9f9ef1340c use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
chrislu c935b9669e 2.83 2021-12-25 01:01:34 -08:00
chrislu eb4ad2546f use proper chunk size limit option 2021-12-24 22:52:18 -08:00
chrislu 41bbf320bb use 2MB chunk size. cache size is the wrong option 2021-12-24 22:50:19 -08:00
chrislu 982ea85d81 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-24 22:40:07 -08:00
chrislu 083d8e9ece add stream writer
this should improve streaming write performance, which is common in many cases, e.g., copying large files.

This is additional to improved random read write operations: 3e69d19380...19084d8791
2021-12-24 22:38:22 -08:00
Chris Lu e7a6a2733b
Merge pull request #2534 from skurfuerst/add-ui-access-to-security-toml 2021-12-24 09:57:00 -08:00
Sebastian Kurfürst 6db49100d6 BUGFIX: add access.ui setting to scaffolded security.toml
... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)
2021-12-24 13:59:04 +01:00
chrislu 255a1c7dcd refactor type names 2021-12-23 18:23:18 -08:00
chrislu f77ca41769 refactor 2021-12-23 17:48:34 -08:00
chrislu 1d36884845 rename files 2021-12-23 17:47:58 -08:00