gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
7113 commits 28 branches 278 tags 135 MiB
Commit graph

4765 commits

Author SHA1 Message Date
chrislu bb9919b07a add open release, refactor 2022-02-13 19:14:34 -08:00
Eng Zer Jun b92df1654c
test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-02-14 10:38:13 +08:00
chrislu 072f923a9b add doc 2022-02-13 16:56:35 -08:00
chrislu 3d0e9e5197 supports renaming 2022-02-13 16:34:57 -08:00
chrislu 417b59b893 doc 2022-02-13 14:37:47 -08:00
chrislu 24290fed9d add comments 2022-02-13 06:02:21 -08:00
chrislu 6200b6abb1 avoid fatal message in some edge cases 2022-02-13 05:59:10 -08:00
chrislu 6a921e15f3 forget() factor in nlookup 2022-02-13 05:49:29 -08:00
chrislu f8af0f93d9 support link 2022-02-13 05:29:43 -08:00
chrislu 0381338d40 fix import 2022-02-13 04:23:06 -08:00
chrislu a1ef0e48a9 doc 2022-02-13 04:22:02 -08:00
chrislu be40ff6048 add symlink 2022-02-13 03:50:16 -08:00
chrislu 3d93570979 supports forget 2022-02-13 03:31:47 -08:00
chrislu 6a42cb6b0b supports mknod, unlink 2022-02-13 03:09:24 -08:00
chrislu 813b868b9a add rmdir 2022-02-13 01:43:11 -08:00
chrislu e85ca10a1a add mkdir 2022-02-13 01:34:19 -08:00
chrislu 21046c6a28 split files 2022-02-13 01:05:30 -08:00
chrislu a4c9223b9d support xattr 2022-02-13 00:58:46 -08:00
chrislu c81833a192 add directory setAttr 2022-02-12 23:08:56 -08:00
chrislu 5c48c23235 remove println 2022-02-12 22:45:07 -08:00
chrislu 7cfbf1e85f fix ok status 2022-02-12 22:41:45 -08:00
chrislu 4c75fd5f9c sync format 2022-02-12 22:41:29 -08:00
chrislu b0a5193e32 working 2022-02-12 22:21:30 -08:00
chrislu 661a34e23d Merge branch 'master' into mount2 2022-02-12 17:58:32 -08:00
chrislu 5e814afe88 blocks count 2022-02-12 17:58:12 -08:00
Radtoo 724ce04b1c We picked the second flag's name. 2022-02-13 00:53:35 +01:00
chrislu 5a0a709016 it runs, but directory listing output is not showing up 2022-02-12 05:27:16 -08:00
chrislu 866981d8ac rename 2022-02-12 02:49:15 -08:00
chrislu 72faae91e1 implement read directory and read directory plus 2022-02-12 02:48:44 -08:00
chrislu a10c28ba82 simplify 2022-02-12 01:59:36 -08:00
chrislu f4d88862c4 can attr root directory 2022-02-12 01:54:16 -08:00
chrislu 180445f5a8 change to use fuse file system 2022-02-11 21:35:09 -08:00
chrislu 45a0fda9bd need to follow https://github.com/hanwen/go-fuse/blob/master/fuse/api.go 2022-02-11 03:12:52 -08:00
chrislu f87da798a4 to be re-written following fuse virtual file system 2022-02-11 03:09:30 -08:00
chrislu 9a913457dd supports stats 2022-02-10 23:23:47 -08:00
chrislu b6143de52a mount with name 2022-02-10 22:43:55 -08:00
chrislu 7a0c35674c clean up previously mounted folder 2022-02-10 20:46:53 -08:00
chrislu c3f9d9fa2e initial setup 2022-02-10 20:32:13 -08:00
Radtoo fbe7ed7927 Revert the the flag to "reallyDeleteFromVolume". 2022-02-10 20:01:48 +01:00
chrislu 21aaa4c1f1 ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount) 
fix https://github.com/chrislusf/seaweedfs/issues/2642
 2022-02-08 01:51:13 -08:00
chrislu 13f6ec1c4e test checking ec distribution 2022-02-08 01:50:05 -08:00
chrislu f34c2ff7c5 use embed txt 2022-02-08 01:14:27 -08:00
chrislu c8c7c10c3f volume.tier.move: avoid double counting 
related to https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:57:35 -08:00
chrislu f18803424a volume.balance: add delay during tight loop 
fix https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:53:55 -08:00
chrislu 9860405974 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-07 23:08:57 -08:00
chrislu 85c1615b43 filer read empty file may cause OOM in some cases 
fix https://github.com/chrislusf/seaweedfs/issues/2641
 2022-02-07 23:08:54 -08:00
Chris Lu a2ac540ecc
Merge pull request #2645 from guol-fnst/fix_Precedence 
fix preconditions
 2022-02-07 18:32:16 -08:00
root 7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu b1cff07ab0 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-07 11:38:01 -08:00
chrislu 81f86c381d s3: avoid nil response 
fix https://github.com/chrislusf/seaweedfs/issues/2636
 2022-02-07 11:37:59 -08:00
Chris Lu f46763d74f
Merge pull request #2644 from kmlebedev/vacum_fsync 
updated needle with fsync
 2022-02-07 11:33:58 -08:00
Konstantin Lebedev ef541972f8 updated needle with fsync 2022-02-08 00:10:53 +05:00
Chris Lu 1b7fb3ce84
Merge pull request #2643 from kmlebedev/graceful_volume_stop 2022-02-07 09:58:26 -08:00
Konstantin Lebedev fb97e234c9 skips compact if store is stopping 2022-02-07 20:16:15 +05:00
chrislu 9405eaefdb filer.sync: fix replicating partially updated file 
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
 2022-02-07 03:46:28 -08:00
Radtoo fbb14e0ea8 Adding separate toggle to purge absent vols 
While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected.

Also renamed forcePurge to just purge.
 2022-02-06 23:46:52 +01:00
Radtoo 6b17f45da2 Also delete paths for which a volume is entirely absent. 2022-02-06 23:22:04 +01:00
Radtoo fa0cfdfb7f Added basic deletion capabilities to fsck. 2022-02-06 17:05:58 +01:00
chrislu 433fde4b18 move error to a separate file 
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
 2022-02-04 22:57:51 -08:00
Chris Lu a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics 
error metrics for filer and store
 2022-02-04 22:35:13 -08:00
Chris Lu 247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors 
add s3api error for copy in file, not directory
 2022-02-04 22:34:34 -08:00
chrislu ced3b89395 add util package 2022-02-04 21:34:58 -08:00
chrislu affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
chrislu 76e297d64f sync call to write file, avoid vif loading error 
fix https://github.com/chrislusf/seaweedfs/issues/2633
 2022-02-04 11:14:04 -08:00
zerospiel f3364fec99 weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting 
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).

Added generic comment rule of traversing methods
 2022-02-04 15:14:48 +03:00
Konstantin Lebedev 9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev 3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Lapshinn Vitaly 6bdc274d4d add s3api error for copy in file, not directory 2022-02-04 03:28:37 +03:00
chrislu 6cf2e7d493 s3: listObjectParts return ErrNoSuchUpload if does not exist 
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
    "Initiator": null,
    "Owner": null,
    "StorageClass": "STANDARD"
}

If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
 2022-02-03 12:34:16 -08:00
zerospiel b54a65ba5a weed/s3api: added new bucket handlers for more compatibility with AWS S3 
Protocol

Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.

Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment

Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
 2022-02-03 17:17:05 +03:00
chrislu 6bee1e9714 [volume.check.disk] fix wrong logic to compare 
fix https://github.com/chrislusf/seaweedfs/issues/2626

minuend - subtrahend
 2022-02-01 07:48:28 -08:00
chrislu 7270067289 2.88 2022-01-30 20:25:26 -08:00
chrislu 84c9bc4389 edge case: old entry was not replicated to remote storage 2022-01-30 20:23:24 -08:00
chrislu b8490fe427 adjust volume count even when not applying the changes 2022-01-28 19:11:46 -08:00
Chris Lu b3f0f170b4
Merge pull request #2618 from divanikus/master 2022-01-28 03:34:52 -08:00
chrislu b9b684194f remove max connection age 
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
 2022-01-27 02:28:22 -08:00
divanikus 67e3fe996a async volumeTierMove 2022-01-26 18:22:31 +03:00
chrislu 62d815d1ca use limited in memory buffer instead of swap file 2022-01-26 00:36:19 -08:00
Chris Lu 9596fce562
Merge pull request #2613 from kmlebedev/masterPromMetrics 
Master prom metrics
 2022-01-25 08:13:14 -08:00
Konstantin Lebedev 5c9259fa3c fix metrics master name 2022-01-25 14:42:47 +05:00
chrislu 7328962009 revert "POSIX: should not delete if a directory is not empty" 
revert 0c75f15062

Reported:

Hi, about commit: 0c75f15062
POSIX: should not delete if a directory is not empty
It should still delete with a command like rm -rf "${path}"/  because it is a forced delete, but now it gets fail to delete non-empty folder: [...]
Can you enable the delete if it is forced?
 2022-01-24 14:21:50 -08:00
Konstantin Lebedev c9952759c4 metrics master is leader 2022-01-24 20:13:07 +05:00
Konstantin Lebedev 28efe31524 new master metrics 2022-01-24 19:09:43 +05:00
chrislu 3bba2124ef use a sliding window of in-memory writable chunks 2022-01-23 23:02:05 -08:00
chrislu 520591e6ea reset swap file chunk after uploading 2022-01-23 18:30:53 -08:00
chrislu e185d90d24 2.87 2022-01-23 16:18:55 -08:00
chrislu 4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu bb6854b972 adjust log 2022-01-22 12:35:09 -08:00
chrislu 2bdd737971 rename file 2022-01-22 08:28:35 -08:00
chrislu 643bbbeb49 rename 2022-01-22 08:27:40 -08:00
chrislu 18ed06b420 use memory when under 16 chunks 2022-01-22 08:11:01 -08:00
chrislu 8e80f3cd65 move upload pipeline locking to a different file 2022-01-22 08:09:55 -08:00
chrislu dde34fa99d rename 2022-01-22 08:08:01 -08:00
chrislu a2aa542370 rename 2022-01-22 08:07:11 -08:00
chrislu c376ccc5a5 swap file based random write large file upload 2022-01-22 08:06:53 -08:00
chrislu b9ae16fbc5 fix memory allocation 2022-01-22 08:05:04 -08:00
chrislu 1ee828b768 refactor 
do not expose internal offset
 2022-01-22 06:34:29 -08:00
chrislu 02d0c12cdd rename 2022-01-22 06:00:10 -08:00
chrislu 5dea5c0449 refactor 2022-01-22 05:59:07 -08:00
chrislu 8aa6bf0bb9 refactoring 2022-01-22 05:40:10 -08:00
chrislu d97bd54e63 just refactoring 2022-01-22 04:18:54 -08:00
chrislu 482014f9da rename file 2022-01-22 03:50:18 -08:00
chrislu 3b4a9addaf rename 2022-01-22 01:46:10 -08:00
chrislu 4acfc098e9 re-order 2022-01-22 01:43:14 -08:00
chrislu e71dcfb3a6 add logging for memory allocation 2022-01-22 01:35:12 -08:00
chrislu 9d0f58c329 skip printing fs configuration 2022-01-21 13:29:47 -08:00
chrislu ce2049cdb6 refactoring, move genFn before saveFn 2022-01-21 12:08:58 -08:00
chrislu e47f63d159 enforce bucket quota 2022-01-21 02:34:42 -08:00
chrislu 6e57d8d0de s3: check bucket usage and adjust read only according to quota 2022-01-21 02:15:27 -08:00
chrislu 606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
chrislu b1063162b6 display bucket quota 2022-01-21 00:55:04 -08:00
chrislu f103491912 s3: list bucket size from weed shell 2022-01-21 00:26:49 -08:00
chrislu 6c7135d77e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-21 00:17:06 -08:00
chrislu ea57654e34 refactoring 2022-01-21 00:16:50 -08:00
Chris Lu 7c66f3b5fb
Merge pull request #2602 from kmlebedev/master_metrics 
master metricsHttpPort
 2022-01-20 09:26:25 -08:00
chrislu b3e526ba95 url should be always using forward slash 2022-01-19 22:16:26 -08:00
Konstantin Lebedev 77c98b657e master metricsHttpPort 2022-01-19 21:43:22 +05:00
chrislu 77362700e1 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix #2593
 2022-01-18 12:04:40 -08:00
chrislu 05c3c3f56b Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-17 23:38:13 -08:00
chrislu 9b77f0054e 2.86 2022-01-17 23:38:03 -08:00
Chris Lu ec254d8a89
Merge pull request #2597 from guol-fnst/gocql_to 
add gocql timeout setting
 2022-01-17 23:35:13 -08:00
chrislu 9274557552 keep dirty pages based on temp file 2022-01-17 23:23:49 -08:00
guol-fnst da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
chrislu c87b8f4c30 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix https://github.com/chrislusf/seaweedfs/issues/2593
 2022-01-17 23:09:37 -08:00
chrislu b2acfd75e9 ensure entry view cache is invalidated 2022-01-17 23:02:30 -08:00
chrislu f4ad63528a wait for reading threads to complete before dropping sealed chunks 2022-01-17 22:24:44 -08:00
chrislu 0a3f95ca01 more logs 2022-01-17 20:41:00 -08:00
chrislu b068bc291d testing with always resetting entry view cache 2022-01-17 20:07:01 -08:00
chrislu 047446d5ca remove extra async execution 2022-01-17 15:50:11 -08:00
chrislu 7bf7af971b more logs 2022-01-17 14:15:10 -08:00
chrislu fc22071a2f more logs 2022-01-17 14:02:37 -08:00
chrislu 381f4e73a0 delete actual reference first 2022-01-17 13:56:47 -08:00
chrislu 0ba88596e8 invalidate filehandle entry view cache 2022-01-17 13:53:30 -08:00
chrislu 1734017ba1 add test 2022-01-17 13:40:41 -08:00
chrislu da7f13e73e Revert "testing skip memory management" 
This reverts commit 6c908352cb.
 2022-01-17 03:21:31 -08:00
chrislu 6c908352cb testing skip memory management 2022-01-17 03:19:24 -08:00
chrislu 77d9993f38 remove unused variables 2022-01-17 03:19:11 -08:00
chrislu f710d5ffca a little speed up 2022-01-17 03:19:00 -08:00
chrislu fc0628c038 working 2022-01-17 01:53:56 -08:00
chrislu 1bd6d289d4 better locking on file handle 2022-01-15 05:45:29 -08:00
chrislu 2bfeb5d1c8 add filer to iam option 2022-01-15 03:37:52 -08:00
chrislu b17c426e99 weed server: optionally start IAM service 
related to https://github.com/chrislusf/seaweedfs/issues/2560
 2022-01-13 22:49:49 -08:00
chrislu 3c8b74318e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-13 13:03:07 -08:00
chrislu 8907e6a40a add more help messages 2022-01-13 13:03:04 -08:00
banjiaojuhao 45e9c83421 padding zero for sparse file 2022-01-13 22:21:22 +08:00
chrislu fe5b9e39cc POSIX: check permission when removing items 2022-01-13 02:07:39 -08:00
chrislu 1453263b63 remove dead code 2022-01-13 02:02:04 -08:00
chrislu e69c374956 minor 2022-01-13 02:01:53 -08:00
chrislu 9b954dc0d4 adjust make file 2022-01-13 01:33:13 -08:00
chrislu f2847f1266 POSIX: check deletion permission 2022-01-12 23:58:11 -08:00