gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9371 commits 28 branches 278 tags 135 MiB
Commit graph

6108 commits

Author SHA1 Message Date
Patrick Schmidt 71b33faef0
Skip parent directory creation in mount (#4310) 
A POSIX filesystem does not implicitly create parent directories when
they do not exist. Directories must be explicitly created and
permissions be set.

This also fixes a bug where asynchronous operations would create a
file in the filer before the parent directory was created. If the
file was a symlink or another special type of file the directory
would inherit that type and become unusable in the mounted FS.
 2023-03-15 12:37:25 -07:00
chrislu f5854d13df fix test 2023-03-14 22:05:16 -07:00
chrislu 0454bb2d88 fix test 
when size is zero, the "LastModified" is not written
 2023-03-14 22:02:49 -07:00
chrislu 2268d38263 fix naming 
fix https://github.com/seaweedfs/seaweedfs/issues/4305
 2023-03-14 08:38:02 -07:00
chrislu 98bc93cf31 fix tests 2023-03-13 10:00:27 -07:00
Konstantin Lebedev e17429223e
shell script unclean variables (#4298) 2023-03-13 07:16:31 -07:00
Thomas Anderson bd70683871
fix(weed/topology/node.go): typo in error message (#4292) 2023-03-08 23:58:03 -08:00
Rohit Chormale 59706c89fb
#4270 set http status code to 409 if dir already exists (#4287) 2023-03-08 07:45:08 -08:00
chrislu 7111c08289 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-03-08 01:48:30 -08:00
chrislu c589e9837c fix tests 2023-03-08 01:48:28 -08:00
Kevin Liu 244385bf0d
Fix binding metrics to ipv6 (#4286) 
* Fix binding metrics to ipv6

* Update weed/stats/metrics.go

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-03-08 01:04:02 -08:00
zemul bd6f2716d3
fix_read_tomb_needle_meta (#4263) 
Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-03-02 08:26:55 -08:00
zemul 0bf56298d5
fix chunk.ModifiedTsNs (#4264) 
* fix

* fix mtime s > ns

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-03-02 08:24:36 -08:00
chrislu 8241c9a829 adjust ui pagination size 
fix https://github.com/seaweedfs/seaweedfs/issues/4258
 2023-02-28 08:31:26 -08:00
lfhy 1976ca9160
add -disk to filer command (#4247) 
* add -disk to filer command

* add diskType to filer.grpc

* use filer.disk when filerWebDavOptions.disk is empty

* add filer.disk to weed server command.

---------

Co-authored-by: 三千院羽 <3000y@MacBook-Pro.lan>
 2023-02-25 09:48:59 -08:00
chrislu 214b7cd286 volume.fix.replication: adjust the retry checking times 2023-02-22 10:47:52 -08:00
LHHDZ db5515eada
Fix s3api_object_list_handlers returning contents less than the specified limit when more data actually exists (#4240) 
Fix when the stored data is actually enough but s3api_object_list_handlers returns less than the specified limit

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2023-02-21 23:08:52 -08:00
Zachary Walters ef2f741823
Updated the deprecated ioutil dependency (#4239) 2023-02-21 19:47:33 -08:00
chrislu 3227e4175e 3.43 2023-02-20 16:56:05 -08:00
chrislu 9bb0a9e306 clean comments 2023-02-17 01:31:44 -08:00
chrislu 545d5d0cc3 fix for io.ReaderAt used in filer.remote.sync 
fix https://github.com/seaweedfs/seaweedfs/issues/4194
 2023-02-17 01:30:55 -08:00
wusong 79af6b41d4
[mount] Fix read file-data in entry.content (#4207) 2023-02-15 07:18:56 -08:00
chrislu 5869945f16 avoid infinite loop 
fix https://github.com/seaweedfs/seaweedfs/issues/4195#issuecomment-1426100904
 2023-02-10 13:05:04 -08:00
chrislu e037c71ec3 adjust text 2023-02-10 13:04:29 -08:00
chrislu 67b8c2853a add line return 2023-02-10 12:53:43 -08:00
chrislu 302155a6fa fix compilation 2023-02-10 12:32:59 -08:00
chrislu 5520b81551 add some error messages for args 
related to https://github.com/seaweedfs/seaweedfs/issues/4115
 2023-02-10 12:30:53 -08:00
chrislu dc4ed2cd9b do not move cloud tier volumes 
fix https://github.com/seaweedfs/seaweedfs/issues/4195
 2023-02-09 22:17:54 -08:00
chrislu 31bb91583f fix bug when vid not found 
fix https://github.com/seaweedfs/seaweedfs/issues/4193
 2023-02-09 17:30:44 -08:00
chrislu 79d68a81fd text change 2023-02-07 15:00:41 -08:00
chrislu 999a126a3f typo 2023-02-07 14:59:47 -08:00
chrislu 1c2364936a detect invalid bucket name 
fix https://github.com/seaweedfs/seaweedfs/issues/4143
 2023-02-07 14:57:29 -08:00
chrislu 5de93fe442 refactoring 2023-02-07 14:45:20 -08:00
Eric Yang 5083429704
ADHOC: s3 bucket name verification (#4189) 
* ADHOC: s3 bucket name verification

* add test
 2023-02-07 14:37:29 -08:00
Neo d5f77706a8
volume:fix return if Replicate write error (#4188) 2023-02-06 22:34:56 -08:00
chrislu 8821d6b161 3.42 2023-02-05 21:28:02 -08:00
Muhammad Hallaj bin Subery 9bd422d2c9
adding support for B2 region (#4177) 
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
 2023-02-05 21:24:21 -08:00
chrislu 84e9934bf9 fix filer.remote.sync on a S3 cloud mount 
fix https://github.com/seaweedfs/seaweedfs/issues/4175
 2023-02-01 20:44:00 -08:00
chrislu 0932437a1c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-02-01 02:38:09 -08:00
chrislu c47f75eb5b fix null content type caused exception in hdfs client 
fix https://github.com/seaweedfs/seaweedfs/issues/4170
 2023-02-01 02:38:05 -08:00
wusong fe573f26a8
memchunk save content twice (#4174) 2023-01-31 19:36:59 -08:00
Neo b9b613a78e
filter system log dir does not make subscribe event (#4172) 2023-01-31 18:56:11 -08:00
wusong 665917fdf1
[mount]: delete redundant code (#4173) 2023-01-31 18:54:46 -08:00
a 0d085ec175
make directory lister trigger if mime type is httpd/unix-directory (#4169) 2023-01-31 06:54:40 -08:00
chrislu 4d71af87f3 3.41 2023-01-29 22:56:01 -08:00
chrislu ef70415c70 skipping manifest chunk volume id change 2023-01-29 22:55:06 -08:00
chrislu 6e8dbf8868 add fs.meta.changeVolumeId 2023-01-29 22:51:58 -08:00
wusong de081c0d64
[mount] fix metacache update (#4161) 2023-01-29 07:55:27 -08:00
Guo Lei b63d1e9c6b
fix index (#4158) 2023-01-28 06:55:27 -08:00
chrislu 81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
chrislu bfe5d910c6 use one readerCache for the whole file 2023-01-16 22:43:02 -08:00
chrislu 4614e85efa adjust help message 2023-01-15 21:28:36 -08:00
chrislu c24678153c use memory chunks only for sequential writes 2023-01-15 21:10:08 -08:00
chrislu 2885ba0e50 3.40 2023-01-15 20:12:05 -08:00
chrislu 12f8251d32 avoid index out of slice range 2023-01-14 13:31:22 -08:00
chrislu 313fd17ae4 allow random writes for mem chunk 2023-01-13 20:32:18 -08:00
Chris Lu 1cd2e64aac
merge chunks during upload (#4130) 
* merge chunks during upload

* fix test
 2023-01-12 23:07:36 -08:00
chrislu 2452f3b2f7 sort chunks by offset 2023-01-12 20:29:07 -08:00
chrislu 361d97941b mysql: Missed escaping column name 
fix https://github.com/seaweedfs/seaweedfs/issues/4129
 2023-01-11 08:46:56 -08:00
chrislu efcffe4f77 no need to set chunks during flush 2023-01-10 10:01:52 -08:00
chrislu 85f1b70207 reduce group size from 256MB to 64MB 2023-01-10 01:35:44 -08:00
chrislu 86657ed3a8 avoid repeatedly adding manifest chunks 2023-01-10 01:35:10 -08:00
chrislu 28fe578944 avoid possible nil entry 2023-01-10 01:07:34 -08:00
chrislu 47fa5ef979 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-01-10 00:52:50 -08:00
chrislu 9e0d526163 fix compilation 2023-01-10 00:52:05 -08:00
chrislu 7f49c59c14 cluster.ps add filer meta sync progress 2023-01-10 00:51:25 -08:00
chrislu 340e7c3a2e chunk group remove manifestChunks and reset sections in SetChunks() 2023-01-10 00:49:31 -08:00
chrislu e650c8397d ensure section.chunks is always garbage free 2023-01-10 00:48:01 -08:00
chrislu 75bdd4a0d1 refactor 2023-01-10 00:46:46 -08:00
Guo Lei 83e29faa55
use time.NewTicker instead of time.Tick. (#4119) 2023-01-09 06:47:08 -08:00
chrislu 51d4a4b28d 3.39 2023-01-08 18:20:22 -08:00
chrislu 0b52f08226 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-01-08 18:03:25 -08:00
chrislu 469c959e6a add missing options for webdav 2023-01-08 18:03:22 -08:00
monchickey 3e2c9ea73d
Add image cropping. (#4117) 2023-01-06 09:28:07 -08:00
chrislu 296fdc296c mount: faster add chunks 2023-01-06 01:03:29 -08:00
chrislu 2abf817580 fix for stream reader 
fix https://github.com/seaweedfs/seaweedfs/issues/4112
 2023-01-05 11:19:21 -08:00
chrislu e367444586 add notes for full version 2023-01-04 09:52:25 -08:00
chrislu 5423790b2c add back previous chunk upload selection algo 2023-01-03 22:50:39 -08:00
chrislu 7bdae5172e batch delete EC needles 
fix https://github.com/seaweedfs/seaweedfs/issues/4107
 2023-01-03 22:05:26 -08:00
zemul 6b4c033431
add mount log (#4101) 
* filer.backup use replication.source.filer

* add mount log

* Revert "filer.backup use replication.source.filer"

This reverts commit 07bf6f956c.

* fix

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-01-03 00:00:45 -08:00
chrislu 48f2edc065 3.38 2023-01-02 23:23:02 -08:00
Chris Lu d4566d4aaa
more solid weed mount (#4089) 
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
 2023-01-02 23:20:45 -08:00
CommanderRoot c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094) 2023-01-01 05:07:53 -08:00
CommanderRoot c2280e94cf
filer.store.mysql: Replace deprecated upsert syntax (#4096) 2023-01-01 05:06:57 -08:00
CommanderRoot 265a56630b
filer.store.mysql: Escape table columns in SQL query (#4095) 2023-01-01 05:06:41 -08:00
Guo Lei ed5f3f073b
add more help message, in case of misunderstanding (#4092) 2022-12-29 07:05:05 -08:00
Guo Lei d8cfa1552b
support enable/disable vacuum (#4087) 
* stop vacuum

* suspend/resume vacuum

* remove unused code

* rename

* rename param
 2022-12-28 01:36:44 -08:00
Guo Lei 5f993a5f20
Fix ide warning (#4082) 
fix warning message from IDE
"redundant type from array, slice, or map composite literal"
 2022-12-26 00:03:47 -08:00
Konstantin Lebedev 569bc7b54d
fs.verify needles status of file (#4060) 
* fs.verify needles status of file

* fix parse args

* refactor logging

* set modifyTimeAgoAtSec
 2022-12-21 15:54:17 -08:00
chrislu 77b7c9f7cc lock and unlock chunks for read 2022-12-21 14:15:35 -08:00
chrislu d1797deccb remove duplicated logic of removing from writableChunks 2022-12-21 14:15:11 -08:00
chrislu 6ede19e825 add a simple file replication progress bar 2022-12-20 19:47:21 -08:00
chrislu f7beba8515 unlock before submitting the uploading jobs 2022-12-20 12:52:30 -08:00
chrislu 28d479e5c0 mount: adjust locking for upload pipeline 2022-12-19 15:07:22 -08:00
chrislu f9fb3cd1fc Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-19 11:32:00 -08:00
chrislu 6c7fe40305 filer sink retries reading file chunks, skipping missing chunks 
if the file chunk is not available during replication time, the file is skipped
 2022-12-19 11:31:58 -08:00
chrislu 3fc3e7083c metadata skip reading un-available logs 
this means the volumes for metadata histories have been lost.
 2022-12-19 11:30:50 -08:00
lfhy 038bab3726
add -filer.path to webdav command (#4061) 2022-12-17 20:07:02 -08:00
chrislu 8a40fa8993 more detailed logs 2022-12-17 13:18:35 -08:00
LHHDZ f84c13dabd
fix response not allowed (#4057) 2022-12-13 05:48:50 -08:00