Commit graph

2752 commits

Author SHA1 Message Date
Chris Lu 59c6422777 a better byte buffer pool 2020-10-20 22:53:58 -07:00
Chris Lu 6c92fa349d filer: support default rack
fix https://github.com/chrislusf/seaweedfs/issues/1546
2020-10-20 17:41:39 -07:00
Chris Lu b3aa2fab9a s3: report error to s3 when updating an object but it is already a directory
fix https://github.com/chrislusf/seaweedfs/issues/1545
2020-10-20 10:25:16 -07:00
Chris Lu 410b818aa7 master: avoid timer leakage 2020-10-19 14:24:57 -07:00
Chris Lu 69f0da5d73 mount: pass along replication and collection parameters 2020-10-19 00:12:42 -07:00
Chris Lu c0ab458671 report nil instead of EOF for empty files
related to https://github.com/chrislusf/seaweedfs/issues/1541
2020-10-17 11:03:46 -07:00
Chris Lu 898db14729 2.05 2020-10-17 02:42:33 -07:00
Chris Lu 09bab17aff mount: avoid "send on closed channel" 2020-10-16 12:54:23 -07:00
Chris Lu 9d80a3428c add debug_webdav 2020-10-16 12:53:56 -07:00
Chris Lu 37f165d743 webdav: return io.EOF when at end of a file
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-10-16 12:53:37 -07:00
Chris Lu 68d39c86f1 mysql, postgres, cassandra: change kv key to base64 encoding
The exisitng key-value operation for stores using mysql, postgres, and maybe cassandra are already broken.

The kv is used to store hardlink, filer store signature and replication progress.

So users using hardlink and also uses mysql, postgres, or cassandra will have broken hard links.

Users using filer.sync will need to re-sync the files.
2020-10-16 11:10:12 -07:00
Chris Lu ee1fc6558a refactor 2020-10-16 11:02:44 -07:00
Chris Lu 06e6341097 set initial write time to avoid race condition with delayed chunks 2020-10-15 11:08:45 -07:00
Chris Lu d598c47136 log errors 2020-10-15 11:01:43 -07:00
Chris Lu 5039aff310 release the goroutine to read errors 2020-10-15 10:59:18 -07:00
Chris Lu ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Konstantin Lebedev dc2e13092d add number of read only volumes metric 2020-10-15 15:48:40 +05:00
Chris Lu e714c28a02 mount: async write file chunk 2020-10-14 23:28:03 -07:00
Chris Lu c95d3d9dff adjust tests 2020-10-14 12:27:52 -07:00
Chris Lu 93bcf56514 file read report EOF
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-10-14 12:18:24 -07:00
Chris Lu 1069b325dd shell: volumeServer.evacuate adds printout for ec volumes 2020-10-13 20:26:03 -07:00
Chris Lu 0542911e29 go fmt 2020-10-13 19:50:46 -07:00
Chris Lu 58fa506491 minor 2020-10-13 19:50:22 -07:00
Chris Lu 28d4e1a51b mount: retry for directory listing with filer
related to https://github.com/chrislusf/seaweedfs/issues/1530
2020-10-13 19:49:52 -07:00
Chris Lu c127da1219 filer: linearize timeout for large chunk of data 2020-10-13 14:04:46 -07:00
Chris Lu aac4cb1f0c adds errror on read and write 2020-10-13 13:53:34 -07:00
Chris Lu 9b4f7fed14 mount: report filer IO error
related to https://github.com/chrislusf/seaweedfs/issues/1530
2020-10-13 11:21:13 -07:00
Chris Lu 3f7d1d1bf1 Only wait on retryable requests 2020-10-13 00:29:46 -07:00
Chris Lu b18f21cce1 mount: fix bound tree with filer.path
fix https://github.com/chrislusf/seaweedfs/issues/1528
2020-10-12 21:58:37 -07:00
Chris Lu f022aff289 add back http.StatusPartialContent
revert e7c04af1d0
2020-10-12 12:26:25 -07:00
Chris Lu bbd0afd37e 2.04 2020-10-11 21:25:30 -07:00
Chris Lu 7704469d29 mount: outsideContainerClusterMode changed to use volume server publicUrl 2020-10-11 20:42:15 -07:00
Chris Lu 723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu d155f907c2 mount: configurable read wait time 2020-10-10 20:09:43 -07:00
Chris Lu 9b0e8ef026 filer: added QR code to transfer files with mobile devices 2020-10-10 18:00:21 -07:00
Chris Lu 8a52379ecb add retry if volume can not be found 2020-10-10 16:02:39 -07:00
Chris Lu 70af0ec24c filer: fix hanging on read 2020-10-10 16:02:10 -07:00
Chris Lu cff8bb6554 return proper error 2020-10-10 15:43:22 -07:00
Chris Lu 6da8eef54f fix logic error 2020-10-09 00:01:47 -07:00
Chris Lu 5d01dd28a6 add logging 2020-10-08 23:38:00 -07:00
Chris Lu 8d34eb0050 mount:exponentially backoff if read error for about 10 minutes 2020-10-08 23:31:26 -07:00
Chris Lu 6e1f936efd refactoring 2020-10-08 23:19:42 -07:00
Chris Lu b2ee5873fb fix error not being returned 2020-10-08 23:19:20 -07:00
Chris Lu b067853162 mount: ignore recursion error
fix https://github.com/chrislusf/seaweedfs/issues/1514

the host OS should manage recursive deletion.
2020-10-08 21:55:55 -07:00
Chris Lu e1a8e3f900 fix logging 2020-10-08 18:40:38 -07:00
Chris Lu 8676db0f6a adjust logging 2020-10-08 18:33:37 -07:00
Chris Lu bd8b9b0c2e some changes to prepare fix after failed reads 2020-10-08 18:33:06 -07:00
Chris Lu 78859e35fc s3: adjust config examples 2020-10-08 10:12:09 -07:00
Chris Lu 9e7a2772b1 s3: only admin can list all buckets 2020-10-08 10:12:09 -07:00
Chris Lu eed492b73b randomize file locations 2020-10-07 23:58:32 -07:00
Chris Lu ec08a1670b volume: avoid hanging when stopping a volume server 2020-10-07 23:48:24 -07:00
Chris Lu 62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu e91b9c85a2 s3: Added support for "List" action in weed s3 -config=... in the config file.
fix https://github.com/chrislusf/seaweedfs/issues/1511
2020-10-07 23:22:35 -07:00
Chris Lu a8624c2e4f read from alternative replica
related to https://github.com/chrislusf/seaweedfs/issues/1512
2020-10-07 22:49:04 -07:00
Chris Lu da4edf3651 master: check peers for existing leader before starting a leader election
fix https://github.com/chrislusf/seaweedfs/issues/1509
2020-10-07 01:25:39 -07:00
Chris Lu c543762e23 volume: auto expands -max setting
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
2020-10-06 09:05:30 -07:00
Chris Lu 36492c47ec adjust 2020-10-05 14:06:18 -07:00
Chris Lu 8f8e9ddbde adjust API 2020-10-05 09:47:07 -07:00
Konstantin Lebedev e4f2d9eb4a We return etag using the same algorithm as aws s3
https://teppen.io/2018/06/23/aws_s3_etags/
2020-10-05 14:43:32 +05:00
Chris Lu 4fc673341f 2.03 2020-10-04 21:58:22 -07:00
Chris Lu 6b591b02af adjust log level 2020-10-04 16:21:43 -07:00
Устюжанин Антон Александрович 702b1cb876 fix: remove deleted peers if resumeState = true 2020-10-04 21:56:17 +05:00
Chris Lu d522df0ecd adjust log level 2020-10-04 01:35:32 -07:00
Chris Lu 666859f7cd mount: read prefetching 2020-10-04 01:31:04 -07:00
Chris Lu bc47835997 mount: avoid duplicated reads 2020-10-03 20:16:42 -07:00
Chris Lu 8d65ad1444 Revert "mount: adds read prefetching"
This reverts commit 692f0614d1.
2020-10-03 19:40:56 -07:00
Chris Lu 692f0614d1 mount: adds read prefetching 2020-10-03 16:50:09 -07:00
Chris Lu 707936f482 re-enable caching larger than 16MB
revert 62ce85610e
2020-10-03 14:12:38 -07:00
Chris Lu 2c20ef72ae fix starting problem 2020-10-03 13:56:38 -07:00
Chris Lu d77e3c96e9 FUSE: use the mount directory permissions 2020-10-03 13:37:35 -07:00
Chris Lu 95ffed87c0
Merge pull request #1505 from ustuzhanin/remove_raft_state
Resume raft state
2020-10-03 10:38:47 -07:00
Устюжанин Антон Александрович 6fbfe73683 fix: resumeState 2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович dc31b19469 fix: restore raft state 2020-10-03 14:03:41 +05:00
Chris Lu f781cce500 s3: support object tagging
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
2020-10-02 22:21:51 -07:00
Устюжанин Антон Александрович 8c82fb7e5f fix: restore raft state 2020-10-02 23:01:20 +05:00
Chris Lu 9ab98fa912 s3 metrics adjust the label 2020-10-01 07:17:47 -07:00
Chris Lu f1e879fe7a 2.02 2020-10-01 07:15:49 -07:00
Chris Lu a1c01d716b volume: avoid deadlock when deleting volumes
fix https://github.com/chrislusf/seaweedfs/issues/1501
2020-10-01 07:10:03 -07:00
Konstantin Lebedev 86329bbf2b label name is statusCode 2020-10-01 01:22:38 +05:00
Konstantin Lebedev 68463e92c1 add status code in S3RequestCounter 2020-10-01 00:59:39 +05:00
Chris Lu 79ab10e300 adjust help message 2020-09-30 09:32:00 -07:00
Chris Lu 55cb68c09c filer: replication follows master config if not specified 2020-09-30 09:15:55 -07:00
Chris Lu b9887504e8 fix test 2020-09-27 23:19:50 -07:00
Chris Lu f46eae284e adjust for test 2020-09-27 23:08:11 -07:00
Chris Lu 75aca5e13d 2.01 2020-09-27 23:02:41 -07:00
Chris Lu e6552b5e1e filer: able to start s3 together 2020-09-27 23:00:43 -07:00
Chris Lu 474e2b6ac3 add a hard link marker to 16byte + maker, for future extensions 2020-09-27 22:38:30 -07:00
Chris Lu c49e2bb9a3 adjust 2020-09-27 12:07:45 -07:00
Chris Lu 62ce85610e skip caching too large chunks 2020-09-27 11:58:48 -07:00
Chris Lu 9ad2dcca2b more tests 2020-09-27 11:42:51 -07:00
Chris Lu e43d86c796 fix pre allocated volume size 2020-09-27 10:58:19 -07:00
Chris Lu 31fc7bb2e1 refactor
adjust for faster test
2020-09-27 10:41:29 -07:00
Chris Lu a37535cd9f avoid non utf-8 in filename
fix https://github.com/chrislusf/seaweedfs/issues/1493
2020-09-26 00:13:39 -07:00
Chris Lu 41be7a4c8f filer: upload also set file size
ensure works same as fuse mount
2020-09-25 12:02:06 -07:00
Chris Lu 35f6518c36 weed upload: usePublicUrl was not being used on big files
fix https://github.com/chrislusf/seaweedfs/issues/1492
2020-09-25 02:41:38 -07:00
Chris Lu 22049dc2e6
Merge pull request #1491 from hilimd/master
s3:  fix get object metadata problem
2020-09-25 02:35:00 -07:00
limd 402aef8f30 s3:
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory

reference git revsion number: b41b7ea4d0
2020-09-25 14:37:02 +08:00
Chris Lu 8516517c48 filer store: Cassandra supports username/password 2020-09-24 21:31:06 -07:00
limd ed7816681a Add cassandra authenticator mode 2020-09-25 11:22:47 +08:00
limd 370a98cf6b code style 2020-09-25 11:19:42 +08:00