Commit graph

92 commits

Author SHA1 Message Date
Chris Lu d4566d4aaa
more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
chrislu ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
liubaojiang 25471d579a
add md5 header when UploadData to replication in ReplicatedWrite (#3881) 2022-10-19 01:34:14 -07:00
chrislu 4af8c427eb refactor 2022-09-10 16:56:03 -07:00
chrislu 4a4ef3cc3c upload_content: upload close response.Body 2022-08-31 00:25:28 -07:00
chrislu 649555b223 adjust log 2022-08-20 23:34:07 -07:00
chrislu a3553da7f7 add fileId to UploadWithRetry return result 2022-08-20 18:54:59 -07:00
chrislu d49d0a9fc2 filer.copy: retryable upload 2022-08-20 18:50:57 -07:00
Konstantin Lebedev 22181dd018
refactor FilerRequest metrics (#3402)
* refactor FilerRequest metrics

* avoid double count proxy

* defer to
2022-08-04 01:44:54 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev c2e7d663bc reduced the timeout for creating a connection from 30 to 10 seconds 2022-03-15 13:32:05 +05:00
chrislu 4de060daa6 mount: skip special character in the filenames
fix https://github.com/chrislusf/seaweedfs/issues/2559
2022-01-05 03:57:24 -08:00
Eng Zer Jun a23bcbb7ec
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu 3adc3da291 refactor 2021-09-06 16:35:55 -07:00
Chris Lu 6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Konstantin Lebedev c2269123d3 fix aws style Etag for chunks 2021-04-28 22:28:05 +05:00
Chris Lu 2acf6be24e resend the http request if connection is stale 2021-04-18 13:07:28 -07:00
Chris Lu f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu 3be061994f skip connection reset error
fix https://github.com/chrislusf/seaweedfs/issues/1971

this is because the connections are pooled but the volume server has reset the connection
2021-04-07 00:54:13 -07:00
Chris Lu 1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu 3b2a9c98ef fix nil 2021-03-07 11:45:35 -08:00
Chris Lu df72dc206d stats collect chunk upload retry count 2021-03-07 11:26:15 -08:00
Chris Lu 44bdfb2d15 filer: avoid encryption and compression at the same time
fix https://github.com/chrislusf/seaweedfs/issues/1828
2021-02-22 12:22:49 -08:00
bingoohuang 352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu 3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu dd9f3a0104 add sleep between upload retries 2021-02-17 20:55:11 -08:00
Chris Lu 487e435679 adjust http max idle connections per host
related to https://github.com/chrislusf/seaweedfs/issues/1802
2021-02-12 03:47:15 -08:00
Chris Lu a29abd8600 add a {X-,}Idempotency-Key header for http POST
fix https://github.com/chrislusf/seaweedfs/issues/1592

according to https://github.com/golang/go/issues/19943
2020-11-11 12:56:53 -08:00
Chris Lu 8708e7a64d print out data size 2020-11-11 12:51:44 -08:00
Chris Lu 0ecff9521d avoid []byte reuse 2020-11-02 23:27:54 -08:00
Chris Lu 031aa70da6 Revert "revert bytebufferpool, seems problematic"
This reverts commit e0fcab47c0.
2020-11-02 19:11:25 -08:00
Chris Lu e0fcab47c0 revert bytebufferpool, seems problematic 2020-11-02 13:49:38 -08:00
Chris Lu c057dd1f64 add back upload retries 2020-10-30 02:16:34 -07:00
Chris Lu 19772d70d7 print for debugging 2020-10-24 00:12:02 -07:00
Chris Lu ef9c32ea0d adjust compression threshold 2020-10-23 23:05:55 -07:00
Chris Lu 29c9fa2ef2 byte buffer for uploading 2020-10-23 11:29:51 -07:00
Chris Lu 2579edbc60 fix bug found by tests 2020-10-23 10:05:40 -07:00
Chris Lu e0c8507d93 memory pool to reduce memory allocation 2020-10-23 00:41:40 -07:00
Chris Lu c31b254248 mount: shortcut when there is only one chunk 2020-10-20 23:48:29 -07:00
Chris Lu f64252023e Revert "a better byte buffer pool"
This reverts commit 59c6422777.
2020-10-20 23:21:54 -07:00
Chris Lu 59c6422777 a better byte buffer pool 2020-10-20 22:53:58 -07:00
Chris Lu 387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu 50f7e2e89f add log messages 2020-08-26 09:51:20 -07:00
Chris Lu 5f55a87101 close http response 2020-08-24 00:32:44 -07:00
Chris Lu 0b301b0b47 consume all response body in order to release requests 2020-08-23 21:32:29 -07:00
Chris Lu 003d48da21 adjust logs 2020-08-15 19:55:28 -07:00
Chris Lu 83cad3da79 add retry file upload 2020-08-11 20:30:11 -07:00
Chris Lu 20e2ac1add filer: store md5 metadata for files uploaded by filer
fix https://github.com/chrislusf/seaweedfs/issues/1412
2020-08-06 10:04:17 -07:00
Chris Lu bbc8ae110f report errors 2020-07-14 11:27:00 -07:00