gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
10040 commits 28 branches 278 tags 135 MiB
Commit graph

722 commits

Author SHA1 Message Date
chrislu ae93c966d9 ensure memory is aligned 
fix https://github.com/seaweedfs/seaweedfs/issues/3427
 2022-08-10 22:27:13 -07:00
Abirdcfly b0633716b7
delete minor unreachable code (#3423) 
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
 2022-08-09 01:24:43 -07:00
chrislu 0854171d22 3.20 2022-08-07 14:42:28 -07:00
chrislu 67814a5c79 refactor and fix strings.Split 2022-08-07 01:34:32 -07:00
chrislu 1a4bf0dcb5 filer.sync: parallelize the filer.sync 2022-08-07 00:56:15 -07:00
chrislu 0e9478488d filer.sync: fix when excluded paths is empty 2022-08-07 00:55:34 -07:00
Konstantin Lebedev 4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
chrislu 03c6f978f4 3.19 2022-07-31 17:16:51 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 21b6b07dd8 renaming 2022-07-28 23:22:06 -07:00
chrislu 475185fb72 3.18 2022-07-25 22:41:38 -07:00
chrislu dbe4849ffd 3.17 2022-07-24 18:46:50 -07:00
chrislu 2ae3f812f8 minor 2022-07-19 11:43:31 -07:00
chrislu 56ec89625a 3.16 2022-07-18 00:53:05 -07:00
chrislu 93ccc6e05f 3.15 2022-07-11 00:50:07 -07:00
chrislu 3c79c77056 3.14 2022-07-04 13:15:49 -07:00
chrislu 40a9634c3d 3.13 2022-06-26 20:13:52 -07:00
chrislu 4dc27e1ed5 3.12 2022-06-20 19:07:00 -07:00
chrislu d4ef06cdcf 3.11 2022-06-14 12:14:52 -07:00
chrislu e437ec5a68 3.10 2022-06-12 19:30:27 -07:00
chrislu 4a046e4de7 3.09 2022-06-05 19:00:24 -07:00
chrislu 8a49240d64 3.08 2022-05-31 11:57:41 -07:00
chrislu fbd99d53c1 3.07 2022-05-31 00:36:24 -07:00
chrislu 968ca95b49 filer.meta.tail: support untilTimeAgo for a range 2022-05-30 16:16:23 -07:00
chrislu f214dfb1f5 stop when in memory log is done 2022-05-30 15:25:21 -07:00
chrislu 7e25a2d416 reduce busy waiting when reading metadata logs 2022-05-24 00:23:53 -07:00
chrislu 2f846777bb 3.06 2022-05-23 01:21:48 -07:00
chrislu acc318e12b 3.05 2022-05-22 21:43:59 -07:00
chrislu 9ff0d99002 3.04 2022-05-15 21:32:21 -07:00
chrislu cd2d10118c 3.03 2022-05-15 18:17:07 -07:00
chrislu a4ca3ed1f0 3.02 2022-05-08 22:28:32 -07:00
chrislu ef6c6c450e avoid fatal error if port is already in use 2022-05-04 14:55:14 -07:00
chrislu 73961e24d8 3.01 2022-05-01 23:33:45 -07:00
chrislu b2a6111090 skip ipv6 all interfaces and localhost 
fix https://github.com/chrislusf/seaweedfs/issues/2983
 2022-04-27 17:18:09 -07:00
chrislu 2aef5b8b32 3.00 2022-04-24 21:46:03 -07:00
chrislu 9ae66f55fc 2.99 2022-04-17 23:01:42 -07:00
justin 3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu c6ec5269f4 2.98 2022-04-10 18:56:40 -07:00
chrislu d310711de0 avoid possible deadlock: move metadata log flush channel out of lock scope 2022-04-03 21:53:30 -07:00
chrislu 77a7d7253f 2.97 2022-04-03 19:08:01 -07:00
chrislu 0490ee87ef 2.96 2022-03-27 16:11:17 -07:00
chrislu 8f0410af2c 2.95 2022-03-21 01:47:03 -07:00
chrislu 3da2b83b38 Added a "-conf_dir" option to customize *.toml configuration file directory. 
fix https://github.com/chrislusf/seaweedfs/issues/2753
 2022-03-19 00:22:47 -07:00
chrislu f247cab5cd skip localhost if bound to all interfaces already 0.0.0.0 or 127.0.0.1 2022-03-17 16:54:29 -07:00
chrislu 3639cad69c master, filer, s3: also listen to "localhost" in addition to specific ip address 
related to https://github.com/chrislusf/seaweedfs/issues/1937
 2022-03-15 22:28:18 -07:00
chrislu 2eda3a686f 2.94 2022-03-14 00:55:01 -07:00
chrislu bd5c5586b5 generate inode via path and time 2022-03-14 00:03:29 -07:00
chrislu 0ba4e4cd23 2.93 2022-03-06 18:54:12 -08:00
chrislu 784583afc6 avoid pool memory allocation if too large 2022-03-02 13:50:28 -08:00
chrislu ba14307319 2.92 2022-02-28 15:22:19 -08:00
chrislu 09cd00f356 2.91 2022-02-27 04:03:39 -08:00
chrislu d602d68fd1 remove dead code 2022-02-27 03:41:32 -08:00
chrislu 708e14fcfa avoid possible too big memory allocation 2022-02-26 03:22:41 -08:00
chrislu 2ab0ad24a3 use memory pool 2022-02-26 02:59:19 -08:00
chrislu 28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu 3ad5fa6f6f chunk cache adds function ReadChunkAt 2022-02-25 21:55:04 -08:00
chrislu 497ebbbd45 2.90 2022-02-20 22:00:13 -08:00
chrislu 6a40fd1c65 2.89 2022-02-14 01:52:16 -08:00
Eng Zer Jun b92df1654c
test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-02-14 10:38:13 +08:00
root 7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu 76e297d64f sync call to write file, avoid vif loading error 
fix https://github.com/chrislusf/seaweedfs/issues/2633
 2022-02-04 11:14:04 -08:00
chrislu 7270067289 2.88 2022-01-30 20:25:26 -08:00
chrislu e185d90d24 2.87 2022-01-23 16:18:55 -08:00
chrislu b9ae16fbc5 fix memory allocation 2022-01-22 08:05:04 -08:00
chrislu e71dcfb3a6 add logging for memory allocation 2022-01-22 01:35:12 -08:00
chrislu 9b77f0054e 2.86 2022-01-17 23:38:03 -08:00
chrislu de27058d0b POSIX: differentiate device and char device 2022-01-12 21:45:38 -08:00
chrislu fec8428fd8 POSIX: different inode for same named different file types 2022-01-12 11:51:13 -08:00
chrislu 2dcb8cb93b POSIX: ensure file and directory inodes are different 
this is just an in memory representation.

POSIX wants different inode numbers for the same named file or directory.
 2022-01-11 23:44:48 -08:00
chrislu 19555385f7 2.85 2022-01-09 19:30:23 -08:00
chrislu 5799a20f71 2.84 2022-01-02 17:05:19 -08:00
Chris Lu 9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158 
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
 2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst 10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
 2021-12-30 14:45:27 +01:00
chrislu fb434318e3 dynamically adjust connection timeout 
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:44:39 -08:00
chrislu 5788bf2270 s3: increase timeout limit 
https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:21:02 -08:00
chrislu c935b9669e 2.83 2021-12-25 01:01:34 -08:00
chrislu c3b73ec23b 2.82 2021-12-12 23:25:24 -08:00
chrislu 5ea9715721 2.81 
also sync java client version to SeaweedFS version
 2021-12-05 18:05:24 -08:00
Tanmoy Majumdar ea09fb477a return ' shouldRetry=true' so that filer can retry the failed chunk 2021-12-03 11:54:20 +06:00
Chris Lu 7227cfddf5 2.80 2021-11-29 00:57:08 -08:00
Chris Lu 3a19eea97c allocate memory by slabs 2021-11-27 12:13:00 -08:00
Chris Lu f3c789d662 2.79 2021-11-21 18:40:24 -08:00
Chris Lu 100c654ec3 2.78 2021-11-14 23:29:59 -08:00
Chris Lu 5cf332357b 2.77 2021-11-07 13:52:45 -08:00
Chris Lu fc9e246592 2.76 2021-10-31 18:08:28 -07:00
Chris Lu c9d3fb4a30 2.75 2021-10-24 18:15:59 -07:00
Chris Lu 182f43ae5f 2.74 2021-10-18 14:23:54 -07:00
Chris Lu cd4fa7561b 2.73 2021-10-18 10:47:48 -07:00
Chris Lu 97c963bac9 2.72 2021-10-17 17:40:27 -07:00
Chris Lu 3833dac3f7 continue to read from memory if there is no flush 2021-10-17 13:53:04 -07:00
Chris Lu 8965a53c4d add warning error 2021-10-16 15:57:30 -07:00
Chris Lu 5fd4b05c5e
Merge pull request #2381 from Juneezee/deprecate-ioutil 
refactor: move from io/ioutil to io and os package
 2021-10-13 22:38:58 -07:00
Chris Lu 46a09c6074 adjust test 2021-10-13 22:38:47 -07:00
Eng Zer Jun a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu 4cbd390fbe test: add fail message 2021-10-13 20:42:20 -07:00
Chris Lu 3d586be552 2.71 2021-10-10 22:40:44 -07:00
Chris Lu e4830bd93d go fmt 2021-10-07 21:13:31 -07:00
Chris Lu f3d8232e14 reduce one redis lookup on hot path 2021-10-06 22:01:19 -07:00
Chris Lu 371fead8a5 redis3 using redis native sorted set 2021-10-06 18:18:24 -07:00
Chris Lu 893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu 4ed2994555 use tsMemory to determine whether read from disk or memory 
remove lastFlushTime
 2021-10-04 16:02:56 -07:00
Chris Lu 513fed323a SkipListElementReference can be an empty object 2021-10-04 02:30:44 -07:00
Chris Lu 280ab7f95c add test 2021-10-04 02:30:24 -07:00
Chris Lu 366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu ba7fbac07f rename 2021-10-03 19:23:34 -07:00
Chris Lu e6196cdc50 add name list 2021-10-03 17:54:25 -07:00
Chris Lu a481c4a45e return previous element if visited 2021-10-03 13:50:52 -07:00
Chris Lu 22d8684e88 refactor out listStore 2021-10-03 02:19:21 -07:00
Chris Lu d343b0db57 update value 2021-10-03 01:15:14 -07:00
Chris Lu 4f50f8c2ca insert key and value 2021-10-03 01:07:35 -07:00
Chris Lu 69b84bb771 TestFindGreaterOrEqual 2021-10-02 14:15:49 -07:00
Chris Lu 57e2fd3f9b remove bptree 2021-10-02 14:03:54 -07:00
Chris Lu 4c1741fdbb working skiplist 2021-10-02 14:02:56 -07:00
Chris Lu b6694279d7 Merge branch 'master' into bptree 2021-10-01 16:55:44 -07:00
Chris Lu 1e3fdf366f go fmt 2021-10-01 12:10:24 -07:00
Chris Lu cee4d20bc1 2.70 2021-09-26 17:37:46 -07:00
Chris Lu 603ea2db73 avoid looping forever if there are no more metadata updates 2021-09-26 11:55:27 -07:00
Chris Lu 9887610b54 log tsNs should be processing time 2021-09-26 11:54:13 -07:00
Chris Lu 2baed2e1e9 avoid possible metadata subscription data loss 
Previous implementation append filer logs into one file. So one file is not always sorted, which can lead to miss reading some entries, especially when different filers have different write throughput.
 2021-09-25 01:18:44 -07:00
Chris Lu b3d88180ca Merge branch 'master' into bptree 2021-09-19 23:56:59 -07:00
Chris Lu fa7c65bd4b 2.69 2021-09-19 21:44:06 -07:00
Chris Lu e066e2642c add NodeStore 2021-09-18 15:32:17 -07:00
Chris Lu 8f2e4be074 wip 2021-09-18 14:04:30 -07:00
Chris Lu b751debd31 split node based on the last inserted key 2021-09-18 01:29:47 -07:00
Chris Lu 2226c3c8b6 Merge branch 'master' into bptree 2021-09-17 10:35:21 -07:00
Chris Lu 2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu 20ac710ceb 2.68 2021-09-13 02:16:09 -07:00
Chris Lu 574485ec69 better IP v6 support 2021-09-07 19:29:42 -07:00
Chris Lu 0128239c0f handle ipv6 addresses 2021-09-07 16:43:54 -07:00
Chris Lu 9fdf02bcda remove detecting ipv6 
Got this error on my local:

transport: Error while dialing dial tcp [fe80::1]:19333: connect: no route to host

related to https://github.com/chrislusf/seaweedfs/pull/2310
 2021-09-07 02:31:34 -07:00
Chris Lu 8c6d706328 2.67 2021-09-07 00:08:03 -07:00
Chris Lu 6022db6d6a 2.66 2021-09-05 16:21:14 -07:00
Chris Lu 2348e8d8da
Merge pull request #2310 from nivekuil/ipv6 
Detect ipv6 addresses
 2021-09-05 10:56:44 -07:00
nivekuil 0fe9d2997b Detect ipv6 addresses 2021-09-05 06:21:40 -07:00
Chris Lu c5ee03d6af format 2021-09-04 13:57:55 -07:00
Chris Lu 8ec357b3d3 go mod 2021-09-03 23:25:33 -07:00
Chris Lu 03a31587ce go fmt 2021-09-03 20:42:28 -07:00
Chris Lu bca4a9de78 simplify 2021-09-02 23:09:24 -07:00
Chris Lu 958125bd02 conforming to http user agent common practice 2021-09-02 22:55:35 -07:00
Chris Lu 11a496404b reset wait time 2021-09-02 19:55:01 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu bec3f63298 2.65 2021-08-28 05:27:33 -07:00
Chris Lu ff7dc3b44c 2.64 2021-08-23 00:39:15 -07:00
Chris Lu df1d6133a8 bptree does not work well for auto-increasing keys 2021-08-22 18:19:26 -07:00
Chris Lu 51c8f2518f change key type to ItemKey 2021-08-21 15:54:42 -07:00
Chris Lu b3e49d2758 change value type to ItemValue 2021-08-21 15:52:17 -07:00
Chris Lu 38c8470d1d add back non_dedup 2021-08-21 15:13:13 -07:00
Chris Lu 849f185a20 add memory kv store 2021-08-21 15:00:44 -07:00
Chris Lu 5f6cc9a814 make proto node 2021-08-21 13:36:52 -07:00