Commit graph

6961 commits

Author SHA1 Message Date
Chris Lu 0c1f42f4eb
Merge pull request #2654 from chrislusf/dependabot/maven/other/java/hdfs3/org.apache.hadoop-hadoop-common-3.1.4
Bump hadoop-common from 3.1.1 to 3.1.4 in /other/java/hdfs3
2022-02-09 16:15:33 -08:00
Chris Lu 439f0f834d
Merge pull request #2655 from chrislusf/dependabot/maven/other/java/hdfs-over-ftp/org.apache.hadoop-hadoop-common-3.2.2
Bump hadoop-common from 3.2.1 to 3.2.2 in /other/java/hdfs-over-ftp
2022-02-09 16:15:21 -08:00
Chris Lu 47c04db9b3
Merge pull request #2656 from chrislusf/dependabot/maven/other/java/hdfs2/org.apache.hadoop-hadoop-common-2.10.1
Bump hadoop-common from 2.9.2 to 2.10.1 in /other/java/hdfs2
2022-02-09 16:15:06 -08:00
Chris Lu 91876bf628
Merge pull request #2657 from chrislusf/dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-2.10.1
Bump hadoop-common from 2.9.2 to 2.10.1 in /other/java/examples
2022-02-09 16:14:51 -08:00
dependabot[bot] 7a5a717063
Bump hadoop-common from 2.9.2 to 2.10.1 in /other/java/examples
Bumps hadoop-common from 2.9.2 to 2.10.1.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-common
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 22:35:39 +00:00
dependabot[bot] f955096e92
Bump hadoop-common from 2.9.2 to 2.10.1 in /other/java/hdfs2
Bumps hadoop-common from 2.9.2 to 2.10.1.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-common
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 22:32:07 +00:00
dependabot[bot] 2065715712
Bump hadoop-common from 3.2.1 to 3.2.2 in /other/java/hdfs-over-ftp
Bumps hadoop-common from 3.2.1 to 3.2.2.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-common
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 22:29:49 +00:00
dependabot[bot] 1236efb039
Bump hadoop-common from 3.1.1 to 3.1.4 in /other/java/hdfs3
Bumps hadoop-common from 3.1.1 to 3.1.4.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-common
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 22:29:27 +00:00
chrislu eb39f78396 rocksdb: PORTABLE=1 2022-02-09 01:29:22 -08:00
chrislu 13cb609f60 add filer.toml for rocksdb to docker image for rocksdb 2022-02-09 00:12:53 -08:00
chrislu 21aaa4c1f1 ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount)
fix https://github.com/chrislusf/seaweedfs/issues/2642
2022-02-08 01:51:13 -08:00
chrislu 13f6ec1c4e test checking ec distribution 2022-02-08 01:50:05 -08:00
chrislu f34c2ff7c5 use embed txt 2022-02-08 01:14:27 -08:00
chrislu c8c7c10c3f volume.tier.move: avoid double counting
related to https://github.com/chrislusf/seaweedfs/issues/2637
2022-02-08 00:57:35 -08:00
chrislu 0901438bd2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-08 00:53:58 -08:00
chrislu f18803424a volume.balance: add delay during tight loop
fix https://github.com/chrislusf/seaweedfs/issues/2637
2022-02-08 00:53:55 -08:00
Chris Lu 51a5d731ce
Merge pull request #2646 from kmlebedev/graceTimeout
graceful stop cluster http server default timeout 60 sec
2022-02-08 00:22:47 -08:00
Konstantin Lebedev 22a3ef56f2 graceful stop cluster http server default timeout 60 sec
https://github.com/chrislusf/seaweedfs/issues/2630
2022-02-08 13:07:28 +05:00
chrislu 9860405974 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-07 23:08:57 -08:00
chrislu 85c1615b43 filer read empty file may cause OOM in some cases
fix https://github.com/chrislusf/seaweedfs/issues/2641
2022-02-07 23:08:54 -08:00
Chris Lu a2ac540ecc
Merge pull request #2645 from guol-fnst/fix_Precedence
fix preconditions
2022-02-07 18:32:16 -08:00
root 7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu b1cff07ab0 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-07 11:38:01 -08:00
chrislu 81f86c381d s3: avoid nil response
fix https://github.com/chrislusf/seaweedfs/issues/2636
2022-02-07 11:37:59 -08:00
Chris Lu f46763d74f
Merge pull request #2644 from kmlebedev/vacum_fsync
updated needle with fsync
2022-02-07 11:33:58 -08:00
Konstantin Lebedev ef541972f8 updated needle with fsync 2022-02-08 00:10:53 +05:00
Chris Lu 1b7fb3ce84
Merge pull request #2643 from kmlebedev/graceful_volume_stop 2022-02-07 09:58:26 -08:00
Konstantin Lebedev fb97e234c9 skips compact if store is stopping 2022-02-07 20:16:15 +05:00
chrislu 9405eaefdb filer.sync: fix replicating partially updated file
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
2022-02-07 03:46:28 -08:00
chrislu 433fde4b18 move error to a separate file
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Chris Lu a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics
error metrics for filer and store
2022-02-04 22:35:13 -08:00
Chris Lu 247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors
add s3api error for copy in file, not directory
2022-02-04 22:34:34 -08:00
chrislu ced3b89395 add util package 2022-02-04 21:34:58 -08:00
chrislu affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
chrislu 76e297d64f sync call to write file, avoid vif loading error
fix https://github.com/chrislusf/seaweedfs/issues/2633
2022-02-04 11:14:04 -08:00
Chris Lu d8e6bd0a94
Merge pull request #2635 from zerospiel/rearrange_handlers 2022-02-04 09:40:11 -08:00
zerospiel f3364fec99 weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).

Added generic comment rule of traversing methods
2022-02-04 15:14:48 +03:00
Konstantin Lebedev 9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev 3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Lapshinn Vitaly 6bdc274d4d add s3api error for copy in file, not directory 2022-02-04 03:28:37 +03:00
chrislu 6cf2e7d493 s3: listObjectParts return ErrNoSuchUpload if does not exist
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
    "Initiator": null,
    "Owner": null,
    "StorageClass": "STANDARD"
}

If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
2022-02-03 12:34:16 -08:00
Chris Lu a7982bd525
Merge pull request #2631 from zerospiel/bucket_handlers 2022-02-03 10:17:18 -08:00
zerospiel b54a65ba5a weed/s3api: added new bucket handlers for more compatibility with AWS S3
Protocol

Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.

Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment

Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
2022-02-03 17:17:05 +03:00
chrislu 6bee1e9714 [volume.check.disk] fix wrong logic to compare
fix https://github.com/chrislusf/seaweedfs/issues/2626

minuend - subtrahend
2022-02-01 07:48:28 -08:00
chrislu 7270067289 2.88 2022-01-30 20:25:26 -08:00
chrislu 84c9bc4389 edge case: old entry was not replicated to remote storage 2022-01-30 20:23:24 -08:00
chrislu b8490fe427 adjust volume count even when not applying the changes 2022-01-28 19:11:46 -08:00
Chris Lu b3f0f170b4
Merge pull request #2618 from divanikus/master 2022-01-28 03:34:52 -08:00
chrislu b9b684194f remove max connection age
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
2022-01-27 02:28:22 -08:00
divanikus 67e3fe996a async volumeTierMove 2022-01-26 18:22:31 +03:00