gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
7095 commits 28 branches 278 tags 135 MiB
Commit graph

4747 commits

Author SHA1 Message Date
chrislu c81833a192 add directory setAttr 2022-02-12 23:08:56 -08:00
chrislu 5c48c23235 remove println 2022-02-12 22:45:07 -08:00
chrislu 7cfbf1e85f fix ok status 2022-02-12 22:41:45 -08:00
chrislu 4c75fd5f9c sync format 2022-02-12 22:41:29 -08:00
chrislu b0a5193e32 working 2022-02-12 22:21:30 -08:00
chrislu 661a34e23d Merge branch 'master' into mount2 2022-02-12 17:58:32 -08:00
chrislu 5e814afe88 blocks count 2022-02-12 17:58:12 -08:00
Radtoo 724ce04b1c We picked the second flag's name. 2022-02-13 00:53:35 +01:00
chrislu 5a0a709016 it runs, but directory listing output is not showing up 2022-02-12 05:27:16 -08:00
chrislu 866981d8ac rename 2022-02-12 02:49:15 -08:00
chrislu 72faae91e1 implement read directory and read directory plus 2022-02-12 02:48:44 -08:00
chrislu a10c28ba82 simplify 2022-02-12 01:59:36 -08:00
chrislu f4d88862c4 can attr root directory 2022-02-12 01:54:16 -08:00
chrislu 180445f5a8 change to use fuse file system 2022-02-11 21:35:09 -08:00
chrislu 45a0fda9bd need to follow https://github.com/hanwen/go-fuse/blob/master/fuse/api.go 2022-02-11 03:12:52 -08:00
chrislu f87da798a4 to be re-written following fuse virtual file system 2022-02-11 03:09:30 -08:00
chrislu 9a913457dd supports stats 2022-02-10 23:23:47 -08:00
chrislu b6143de52a mount with name 2022-02-10 22:43:55 -08:00
chrislu 7a0c35674c clean up previously mounted folder 2022-02-10 20:46:53 -08:00
chrislu c3f9d9fa2e initial setup 2022-02-10 20:32:13 -08:00
Radtoo fbe7ed7927 Revert the the flag to "reallyDeleteFromVolume". 2022-02-10 20:01:48 +01:00
chrislu 21aaa4c1f1 ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount) 
fix https://github.com/chrislusf/seaweedfs/issues/2642
 2022-02-08 01:51:13 -08:00
chrislu 13f6ec1c4e test checking ec distribution 2022-02-08 01:50:05 -08:00
chrislu f34c2ff7c5 use embed txt 2022-02-08 01:14:27 -08:00
chrislu c8c7c10c3f volume.tier.move: avoid double counting 
related to https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:57:35 -08:00
chrislu f18803424a volume.balance: add delay during tight loop 
fix https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:53:55 -08:00
chrislu 9860405974 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-07 23:08:57 -08:00
chrislu 85c1615b43 filer read empty file may cause OOM in some cases 
fix https://github.com/chrislusf/seaweedfs/issues/2641
 2022-02-07 23:08:54 -08:00
Chris Lu a2ac540ecc
Merge pull request #2645 from guol-fnst/fix_Precedence 
fix preconditions
 2022-02-07 18:32:16 -08:00
root 7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu b1cff07ab0 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-07 11:38:01 -08:00
chrislu 81f86c381d s3: avoid nil response 
fix https://github.com/chrislusf/seaweedfs/issues/2636
 2022-02-07 11:37:59 -08:00
Chris Lu f46763d74f
Merge pull request #2644 from kmlebedev/vacum_fsync 
updated needle with fsync
 2022-02-07 11:33:58 -08:00
Konstantin Lebedev ef541972f8 updated needle with fsync 2022-02-08 00:10:53 +05:00
Chris Lu 1b7fb3ce84
Merge pull request #2643 from kmlebedev/graceful_volume_stop 2022-02-07 09:58:26 -08:00
Konstantin Lebedev fb97e234c9 skips compact if store is stopping 2022-02-07 20:16:15 +05:00
chrislu 9405eaefdb filer.sync: fix replicating partially updated file 
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
 2022-02-07 03:46:28 -08:00
Radtoo fbb14e0ea8 Adding separate toggle to purge absent vols 
While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected.

Also renamed forcePurge to just purge.
 2022-02-06 23:46:52 +01:00
Radtoo 6b17f45da2 Also delete paths for which a volume is entirely absent. 2022-02-06 23:22:04 +01:00
Radtoo fa0cfdfb7f Added basic deletion capabilities to fsck. 2022-02-06 17:05:58 +01:00
chrislu 433fde4b18 move error to a separate file 
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
 2022-02-04 22:57:51 -08:00
Chris Lu a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics 
error metrics for filer and store
 2022-02-04 22:35:13 -08:00
Chris Lu 247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors 
add s3api error for copy in file, not directory
 2022-02-04 22:34:34 -08:00
chrislu ced3b89395 add util package 2022-02-04 21:34:58 -08:00
chrislu affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
chrislu 76e297d64f sync call to write file, avoid vif loading error 
fix https://github.com/chrislusf/seaweedfs/issues/2633
 2022-02-04 11:14:04 -08:00
zerospiel f3364fec99 weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting 
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).

Added generic comment rule of traversing methods
 2022-02-04 15:14:48 +03:00
Konstantin Lebedev 9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev 3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Lapshinn Vitaly 6bdc274d4d add s3api error for copy in file, not directory 2022-02-04 03:28:37 +03:00
chrislu 6cf2e7d493 s3: listObjectParts return ErrNoSuchUpload if does not exist 
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
    "Initiator": null,
    "Owner": null,
    "StorageClass": "STANDARD"
}

If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
 2022-02-03 12:34:16 -08:00
zerospiel b54a65ba5a weed/s3api: added new bucket handlers for more compatibility with AWS S3 
Protocol

Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.

Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment

Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
 2022-02-03 17:17:05 +03:00
chrislu 6bee1e9714 [volume.check.disk] fix wrong logic to compare 
fix https://github.com/chrislusf/seaweedfs/issues/2626

minuend - subtrahend
 2022-02-01 07:48:28 -08:00
chrislu 7270067289 2.88 2022-01-30 20:25:26 -08:00
chrislu 84c9bc4389 edge case: old entry was not replicated to remote storage 2022-01-30 20:23:24 -08:00
chrislu b8490fe427 adjust volume count even when not applying the changes 2022-01-28 19:11:46 -08:00
Chris Lu b3f0f170b4
Merge pull request #2618 from divanikus/master 2022-01-28 03:34:52 -08:00
chrislu b9b684194f remove max connection age 
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
 2022-01-27 02:28:22 -08:00
divanikus 67e3fe996a async volumeTierMove 2022-01-26 18:22:31 +03:00
chrislu 62d815d1ca use limited in memory buffer instead of swap file 2022-01-26 00:36:19 -08:00
Chris Lu 9596fce562
Merge pull request #2613 from kmlebedev/masterPromMetrics 
Master prom metrics
 2022-01-25 08:13:14 -08:00
Konstantin Lebedev 5c9259fa3c fix metrics master name 2022-01-25 14:42:47 +05:00
chrislu 7328962009 revert "POSIX: should not delete if a directory is not empty" 
revert 0c75f15062

Reported:

Hi, about commit: 0c75f15062
POSIX: should not delete if a directory is not empty
It should still delete with a command like rm -rf "${path}"/  because it is a forced delete, but now it gets fail to delete non-empty folder: [...]
Can you enable the delete if it is forced?
 2022-01-24 14:21:50 -08:00
Konstantin Lebedev c9952759c4 metrics master is leader 2022-01-24 20:13:07 +05:00
Konstantin Lebedev 28efe31524 new master metrics 2022-01-24 19:09:43 +05:00
chrislu 3bba2124ef use a sliding window of in-memory writable chunks 2022-01-23 23:02:05 -08:00
chrislu 520591e6ea reset swap file chunk after uploading 2022-01-23 18:30:53 -08:00
chrislu e185d90d24 2.87 2022-01-23 16:18:55 -08:00
chrislu 4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu bb6854b972 adjust log 2022-01-22 12:35:09 -08:00
chrislu 2bdd737971 rename file 2022-01-22 08:28:35 -08:00
chrislu 643bbbeb49 rename 2022-01-22 08:27:40 -08:00
chrislu 18ed06b420 use memory when under 16 chunks 2022-01-22 08:11:01 -08:00
chrislu 8e80f3cd65 move upload pipeline locking to a different file 2022-01-22 08:09:55 -08:00
chrislu dde34fa99d rename 2022-01-22 08:08:01 -08:00
chrislu a2aa542370 rename 2022-01-22 08:07:11 -08:00
chrislu c376ccc5a5 swap file based random write large file upload 2022-01-22 08:06:53 -08:00
chrislu b9ae16fbc5 fix memory allocation 2022-01-22 08:05:04 -08:00
chrislu 1ee828b768 refactor 
do not expose internal offset
 2022-01-22 06:34:29 -08:00
chrislu 02d0c12cdd rename 2022-01-22 06:00:10 -08:00
chrislu 5dea5c0449 refactor 2022-01-22 05:59:07 -08:00
chrislu 8aa6bf0bb9 refactoring 2022-01-22 05:40:10 -08:00
chrislu d97bd54e63 just refactoring 2022-01-22 04:18:54 -08:00
chrislu 482014f9da rename file 2022-01-22 03:50:18 -08:00
chrislu 3b4a9addaf rename 2022-01-22 01:46:10 -08:00
chrislu 4acfc098e9 re-order 2022-01-22 01:43:14 -08:00
chrislu e71dcfb3a6 add logging for memory allocation 2022-01-22 01:35:12 -08:00
chrislu 9d0f58c329 skip printing fs configuration 2022-01-21 13:29:47 -08:00
chrislu ce2049cdb6 refactoring, move genFn before saveFn 2022-01-21 12:08:58 -08:00
chrislu e47f63d159 enforce bucket quota 2022-01-21 02:34:42 -08:00
chrislu 6e57d8d0de s3: check bucket usage and adjust read only according to quota 2022-01-21 02:15:27 -08:00
chrislu 606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
chrislu b1063162b6 display bucket quota 2022-01-21 00:55:04 -08:00
chrislu f103491912 s3: list bucket size from weed shell 2022-01-21 00:26:49 -08:00
chrislu 6c7135d77e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-21 00:17:06 -08:00
chrislu ea57654e34 refactoring 2022-01-21 00:16:50 -08:00
Chris Lu 7c66f3b5fb
Merge pull request #2602 from kmlebedev/master_metrics 
master metricsHttpPort
 2022-01-20 09:26:25 -08:00
chrislu b3e526ba95 url should be always using forward slash 2022-01-19 22:16:26 -08:00
Konstantin Lebedev 77c98b657e master metricsHttpPort 2022-01-19 21:43:22 +05:00
chrislu 77362700e1 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix #2593
 2022-01-18 12:04:40 -08:00
chrislu 05c3c3f56b Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-17 23:38:13 -08:00
chrislu 9b77f0054e 2.86 2022-01-17 23:38:03 -08:00
Chris Lu ec254d8a89
Merge pull request #2597 from guol-fnst/gocql_to 
add gocql timeout setting
 2022-01-17 23:35:13 -08:00
chrislu 9274557552 keep dirty pages based on temp file 2022-01-17 23:23:49 -08:00
guol-fnst da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
chrislu c87b8f4c30 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix https://github.com/chrislusf/seaweedfs/issues/2593
 2022-01-17 23:09:37 -08:00
chrislu b2acfd75e9 ensure entry view cache is invalidated 2022-01-17 23:02:30 -08:00
chrislu f4ad63528a wait for reading threads to complete before dropping sealed chunks 2022-01-17 22:24:44 -08:00
chrislu 0a3f95ca01 more logs 2022-01-17 20:41:00 -08:00
chrislu b068bc291d testing with always resetting entry view cache 2022-01-17 20:07:01 -08:00
chrislu 047446d5ca remove extra async execution 2022-01-17 15:50:11 -08:00
chrislu 7bf7af971b more logs 2022-01-17 14:15:10 -08:00
chrislu fc22071a2f more logs 2022-01-17 14:02:37 -08:00
chrislu 381f4e73a0 delete actual reference first 2022-01-17 13:56:47 -08:00
chrislu 0ba88596e8 invalidate filehandle entry view cache 2022-01-17 13:53:30 -08:00
chrislu 1734017ba1 add test 2022-01-17 13:40:41 -08:00
chrislu da7f13e73e Revert "testing skip memory management" 
This reverts commit 6c908352cb.
 2022-01-17 03:21:31 -08:00
chrislu 6c908352cb testing skip memory management 2022-01-17 03:19:24 -08:00
chrislu 77d9993f38 remove unused variables 2022-01-17 03:19:11 -08:00
chrislu f710d5ffca a little speed up 2022-01-17 03:19:00 -08:00
chrislu fc0628c038 working 2022-01-17 01:53:56 -08:00
chrislu 1bd6d289d4 better locking on file handle 2022-01-15 05:45:29 -08:00
chrislu 2bfeb5d1c8 add filer to iam option 2022-01-15 03:37:52 -08:00
chrislu b17c426e99 weed server: optionally start IAM service 
related to https://github.com/chrislusf/seaweedfs/issues/2560
 2022-01-13 22:49:49 -08:00
chrislu 3c8b74318e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-13 13:03:07 -08:00
chrislu 8907e6a40a add more help messages 2022-01-13 13:03:04 -08:00
banjiaojuhao 45e9c83421 padding zero for sparse file 2022-01-13 22:21:22 +08:00
chrislu fe5b9e39cc POSIX: check permission when removing items 2022-01-13 02:07:39 -08:00
chrislu 1453263b63 remove dead code 2022-01-13 02:02:04 -08:00
chrislu e69c374956 minor 2022-01-13 02:01:53 -08:00
chrislu 9b954dc0d4 adjust make file 2022-01-13 01:33:13 -08:00
chrislu f2847f1266 POSIX: check deletion permission 2022-01-12 23:58:11 -08:00
chrislu 0c75f15062 POSIX: should not delete if a directory is not empty 2022-01-12 23:57:54 -08:00
chrislu de27058d0b POSIX: differentiate device and char device 2022-01-12 21:45:38 -08:00
chrislu d400a11832 POSIX: adjust source file ctime 
SeaweedFS uses mtime as ctime
 2022-01-12 21:45:18 -08:00
chrislu b44f05a2d0 POSIX: change timestamp on each attribute change 2022-01-12 19:31:25 -08:00
chrislu 15c01d8b7f add some notes 2022-01-12 15:04:48 -08:00
chrislu 107a4884a8 shell: tighter memory allocation 2022-01-12 14:59:29 -08:00
chrislu fec8428fd8 POSIX: different inode for same named different file types 2022-01-12 11:51:13 -08:00
chrislu e82ad60122 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-12 11:07:49 -08:00
chrislu caf0a3486b POSIX: adjust ctime for file truncate 2022-01-12 11:07:39 -08:00
Konstantin Lebedev edb753ab4d https://github.com/chrislusf/seaweedfs/issues/2583 2022-01-12 16:04:59 +05:00
chrislu adfd54e7c4 fix compilation 2022-01-12 01:24:24 -08:00
chrislu 6cc92817dc add logs for request mode 2022-01-12 01:13:19 -08:00
chrislu 826a7b307e master: remove hard coded filer settings in master.toml 
fix https://github.com/chrislusf/seaweedfs/issues/2529
 2022-01-12 01:11:25 -08:00
chrislu cd1ad88f30 POSIX: check name is too long ENAMETOOLONG 2022-01-12 00:16:00 -08:00
chrislu 2dcb8cb93b POSIX: ensure file and directory inodes are different 
this is just an in memory representation.

POSIX wants different inode numbers for the same named file or directory.
 2022-01-11 23:44:48 -08:00
chrislu 5bb37d5905 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 23:29:12 -08:00
chrislu 10ecf80ca1 add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
Kyle Sanderson 9e012001be
filer.copy: don't crash when volume creation fails 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
 2022-01-11 22:22:39 -08:00