gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
9198 commits 28 branches 278 tags 135 MiB
Commit graph

57 commits

Author SHA1 Message Date
chrislu 84e9934bf9 fix filer.remote.sync on a S3 cloud mount 
fix https://github.com/seaweedfs/seaweedfs/issues/4175
 2023-02-01 20:44:00 -08:00
chrislu 2abf817580 fix for stream reader 
fix https://github.com/seaweedfs/seaweedfs/issues/4112
 2023-01-05 11:19:21 -08:00
Chris Lu d4566d4aaa
more solid weed mount (#4089) 
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
 2023-01-02 23:20:45 -08:00
chrislu 70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks() 
for later locking on reading chunks
 2022-11-15 06:33:36 -08:00
chrislu 96caf21d09 less verbose log 2022-08-15 10:03:52 -07:00
LHHDZ 84ec68e11a
Add download speed limit support (#3408) 2022-08-05 01:16:42 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev da9d3e8f6c refactor 2022-07-26 11:56:45 +05:00
Konstantin Lebedev 046c3d5ad4 fix logic else brake 2022-07-26 11:47:11 +05:00
chrislu ec0edb1ac4 filer: fix wrong logic during read 2022-07-25 22:40:00 -07:00
Konstantin Lebedev 7b1497ee63 Use BackoffSchedule for getLookupFileId 2022-07-15 16:05:35 +05:00
Konstantin Lebedev 01996bccf8 Use fallback if urls are not found 2022-07-15 15:29:15 +05:00
justin 3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu fbc9f0eb64 minor 2022-03-14 03:19:16 -07:00
chrislu 28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
banjiaojuhao 45e9c83421 padding zero for sparse file 2022-01-13 22:21:22 +08:00
Chris Lu 96c66ca2aa read deleted chunks when replicating data 2021-11-28 23:33:03 -08:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu 9bcf94b2b1 ensure multi-threaded correctness 2021-08-25 17:28:50 -07:00
Chris Lu 40dc283b2d fix locating data chunks 2021-08-15 23:07:58 -07:00
Chris Lu 72eb6d5b9d ensure no writes to remote storage if content is not changed 2021-08-15 20:23:41 -07:00
Chris Lu 5d5a21ba2d adjust log format 2021-08-15 19:46:45 -07:00
Chris Lu 8f7d2d317f readerAt need to use the right offset 
fix https://github.com/chrislusf/seaweedfs/issues/2259
 2021-08-15 11:55:58 -07:00
Chris Lu b961fcd338 filer: stream read from volume server, reduce memory usage 2021-08-13 11:00:11 -07:00
Chris Lu 13e45e1605 filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
Chris Lu de730b079d ChunkStreamReader implenents io.ReaderAt 2021-08-07 15:41:07 -07:00
Chris Lu 59732a0529 refactoring 2021-08-07 15:35:27 -07:00
Chris Lu ecb234f75a refactor 2021-08-07 14:46:23 -07:00
Konstantin Lebedev 84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Konstantin Lebedev 0e404cd84b Chunk download stats 2021-05-24 12:14:50 +05:00
Konstantin Lebedev bb42633522 rm chunk Download stats 2021-05-24 12:08:12 +05:00
Konstantin Lebedev 3325b850cc rm func CheckAllChunkViews 2021-05-21 16:05:59 +05:00
Konstantin Lebedev 26a4f34a57 del checks 
url err logging
stats chunk fetch
 2021-05-21 15:59:12 +05:00
Konstantin Lebedev 42f631f549 fix https://github.com/chrislusf/seaweedfs/issues/2046 2021-05-20 13:19:35 +05:00
Konstantin Lebedev 03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" 
This reverts commit ac71117e
 2021-05-20 11:45:21 +05:00
Chris Lu 55a8f57381 go fmt 2021-05-06 03:37:51 -07:00
Chris Lu ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu 2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Konstantin Lebedev eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Konstantin Lebedev 06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu 4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Konstantin Lebedev 46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Konstantin Lebedev 7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Konstantin Lebedev 58cdcc6d6e error if urls not found 2021-03-11 23:34:36 +05:00
Konstantin Lebedev 4db21012c1 error if read chunk zero data size 2021-03-11 22:38:59 +05:00
Chris Lu 9abb041763 filer source: support filerProxy mode 2021-02-28 16:19:47 -08:00
Chris Lu 7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Chris Lu 2b76854641 add "weed filer.cat" to read files directly from volume servers 2021-01-06 04:22:00 -08:00
Chris Lu d75bc62196 to read files that are quickly changing 2020-10-22 12:59:15 -07:00
Chris Lu aac4cb1f0c adds errror on read and write 2020-10-13 13:53:34 -07:00