gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6671 commits 28 branches 278 tags 135 MiB
Commit graph

1113 commits

Author SHA1 Message Date
Chris Lu c090d6bb25 add ReadRemote(), add read remote setup when filer starts 2021-07-28 22:43:12 -07:00
Chris Lu 10464f47d0
Merge pull request #2217 from combineads/add_webp_format 2021-07-24 09:42:41 -07:00
byunghwa.yun 7374b5b473 Add webp extension for resizing 2021-07-24 14:32:31 +09:00
Chris Lu ac28611817 snowflake sequencer need an unique id 
fix https://github.com/chrislusf/seaweedfs/issues/2213
 2021-07-23 20:54:03 -07:00
Chris Lu 7ab389e7ec optimization: improve random range query for large files 2021-07-19 23:07:22 -07:00
Chris Lu b938df97a2 remove unused parameter 2021-07-19 02:59:12 -07:00
Chris Lu 450222dd64 add remote to filer.Entry and filer_pb entry, add RemoteConf 2021-07-19 02:47:27 -07:00
Chris Lu 18c40686d9 s3: multipart upload miss data if file is chunked in 4MB 
fix https://github.com/chrislusf/seaweedfs/issues/2195
 2021-07-15 11:56:28 -07:00
Chris Lu 49c66e88a0 volume: change all writes to fsync during graceful stopping 
fix https://github.com/chrislusf/seaweedfs/issues/2193
 2021-07-13 01:29:57 -07:00
Chris Lu b194f91f47 add version to filer configuration response 2021-07-12 01:23:20 -07:00
Chris Lu da7bd62822 /etc files are stored inside metadata store 2021-07-09 02:33:14 -07:00
Chris Lu a6d73e0a66
Merge pull request #2185 from bingoohuang/master 
show RemoteVolumes/EcVolumes only if it is not empty
 2021-07-06 00:28:07 -07:00
bingoohuang ed57a55eae show RemoteVolumes/EcVolumes only if it is not empty 2021-07-06 15:20:18 +08:00
Chris Lu 141388367e
Merge pull request #2183 from bingoohuang/master 
extract embed html of master/volume/filer ui to separate files
 2021-07-05 03:22:26 -07:00
bingoohuang 44a2538f67 extract embed html of master/volume/filer ui to separate files 2021-07-05 18:09:44 +08:00
qieqieplus 233103f6b2 sync empty notification with timestamp 2021-07-05 16:01:16 +08:00
Chris Lu 8fe75692ee volume: address "unaligned 64-bit atomic operation" 
fix https://github.com/chrislusf/seaweedfs/issues/2177
 2021-07-02 13:57:43 -07:00
Chris Lu c6d4c16079 S3: add metadata with multipart upload 
fix https://github.com/chrislusf/seaweedfs/issues/2173
 2021-07-01 19:12:11 -07:00
Chris Lu 2420c60fc4 log reading adds delay between retries 2021-07-01 14:01:25 -07:00
Chris Lu 067eb15e70 remove debug messages 2021-07-01 01:24:07 -07:00
Chris Lu b624090398 go fmt 2021-07-01 01:21:14 -07:00
Chris Lu 215b169562 mount: recursively rename locally 2021-07-01 01:19:31 -07:00
zhangsong 20d33ae025 add proxy mode to read non-local volumes 2021-06-30 18:33:18 +08:00
zhangsong 7566782c2e add proxy mode to read non-local volumes 2021-06-30 17:28:37 +08:00
Chris Lu 3668d10664 range query for mp4 video play 
fix https://github.com/chrislusf/seaweedfs/issues/2156
 2021-06-29 02:13:29 -07:00
Chris Lu 17477b37d5 sleep before re-reading the messages 
If there are no more metadata changes and the client disconnects, it would go into a busy loop without this fix.
 2021-06-27 06:31:04 -07:00
Chris Lu cc7714fdbe logging changes to debug 2021-06-27 05:54:16 -07:00
Chris Lu c2e0a75c1f adjust logs 2021-06-24 12:46:00 -07:00
Chris Lu 78b1fb921c adjust log level 2021-06-23 20:59:54 -07:00
Chris Lu f24bb9e688 mount: fix for deletion stopped working since 2.53 
fix https://github.com/chrislusf/seaweedfs/issues/2138

due to 4d0cbd2700
 2021-06-19 03:46:39 -07:00
Chris Lu 88d52adfdd remove unused fields 2021-06-18 15:35:22 -07:00
Chris Lu ed6aa13520 minor 2021-06-12 18:36:25 -07:00
Chris Lu ee6c67682c minor 2021-06-12 02:52:41 -07:00
Chris Lu 5e6dfbc25f locks for data racing 2021-06-07 12:04:50 -07:00
Chris Lu 452c6ef183 limits concurrent uploads for one file 2021-06-06 23:05:17 -07:00
Chris Lu 8295e2feb6 skip md5 checking for now because of race condition 2021-06-06 22:16:32 -07:00
Chris Lu 19caeb7b02 fix writing the small file 2021-06-06 20:57:03 -07:00
Chris Lu bb45dea15a filer: parallel data upload 2021-06-06 20:23:36 -07:00
Chris Lu 44f1ba6894 refactor 2021-06-06 18:43:04 -07:00
Chris Lu 6c82326575 use bytes.Buffer to reduce memory allocation and gc 2021-06-06 13:42:36 -07:00
Chris Lu 9cba5cca0b optionally disable concurrent upload limit 2021-06-06 13:13:33 -07:00
Chris Lu ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu fb8036385a s3: save metadata during put-object 
fix https://github.com/chrislusf/seaweedfs/issues/2092
 2021-05-29 14:14:30 -07:00
Chris Lu ac9bf71544 use jquery 3.6.0 2021-05-26 00:21:11 -07:00
Konstantin Lebedev b612d5aebd s3 test get w/ If-Match: bogus ETag 2021-05-24 16:59:44 +05:00
Konstantin Lebedev 69c768870b - object write cache control 
- object write expires
 2021-05-24 15:43:55 +05:00
Konstantin Lebedev 84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Chris Lu 064269bb57 filer: Support sqlite as filer meta store 2021-05-23 23:58:28 -07:00
Chris Lu e5a2bf1287 s3: deprecating filer.options.buckets_fsync 2021-05-22 00:24:23 -07:00
Konstantin Lebedev 26a4f34a57 del checks 
url err logging
stats chunk fetch
 2021-05-21 15:59:12 +05:00
Konstantin Lebedev 03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" 
This reverts commit ac71117e
 2021-05-20 11:45:21 +05:00
Chris Lu da0a4e775b Revert "Revert "Merge pull request #2027 from bingoohuang/master"" 
This reverts commit d74cdf0115.
 2021-05-10 00:13:55 -07:00
Chris Lu 7ca75347ec minor 2021-05-07 21:56:45 -07:00
Chris Lu 8f8738867f add retry to assign volume 
fix https://github.com/chrislusf/seaweedfs/issues/2056
 2021-05-07 07:29:26 -07:00
Chris Lu aaad4b578b
Merge pull request #2049 from qieqieplus/async-assign 
ahead of time volume assignment
 2021-05-06 10:00:25 -07:00
qieqieplus c4d32f6937 ahead of time volume assignment 2021-05-06 18:55:44 +08:00
Chris Lu c899bdf063 a little optimization 2021-05-06 03:03:00 -07:00
Chris Lu 3eb336e0b0 report error only for the first multipart upload 
the glog.Errorf would always print for s3 multipart uploads
 2021-05-06 01:57:54 -07:00
Chris Lu ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu 24efa31e49
Merge pull request #2045 from qieqieplus/fix-vacuum-commit 2021-05-05 07:41:38 -07:00
qieqieplus ac26080bd2 fix concurrent vacuum & delete panic 2021-05-05 17:54:50 +08:00
Chris Lu e24ba2aadc filer: delete specific tags 
fix https://github.com/chrislusf/seaweedfs/issues/2041
 2021-05-02 21:53:43 -07:00
Chris Lu d74cdf0115 Revert "Merge pull request #2027 from bingoohuang/master" 
Need to revert because docker image build failed. The docker apk package only has go 1.15.
 2021-04-30 03:36:15 -07:00
Konstantin Lebedev c2269123d3 fix aws style Etag for chunks 2021-04-28 22:28:05 +05:00
bingoohuang 7a9d27fce8 promote to go:embed instead of github.com/rakyll/statik 2021-04-27 17:22:24 +08:00
bingoohuang cf552417a7 minFreeSpace refactored 2021-04-27 10:37:24 +08:00
bingoohuang 31f1cdeac2 minFreeSpace argument allows size like 10GiB 2021-04-26 18:48:34 +08:00
Chris Lu ddc8643ee0 filer: directory listing adds namePatternExclude 
fix https://github.com/chrislusf/seaweedfs/issues/2023
 2021-04-24 11:49:03 -07:00
Chris Lu f0ad172e80 shell: show which server holds the lock 
fix https://github.com/chrislusf/seaweedfs/issues/1983
 2021-04-22 23:56:35 -07:00
Chris Lu 11c120c040 master UI adds volume size limit 2021-04-22 14:22:48 -07:00
Chris Lu 6e5df901e4 adjust package names 2021-04-22 14:22:48 -07:00
Konstantin Lebedev 198688c717 revert volume etag 2021-04-16 23:22:31 +05:00
Chris Lu 1e033d45b8 simpler logic 
related to https://github.com/chrislusf/seaweedfs/pull/1981
 2021-04-12 12:04:53 -07:00
Chris Lu 52200a903b
Merge pull request #1981 from ueni-ltd/if-modified-since 
Fix If-Modified-Since behavior
 2021-04-12 12:02:54 -07:00
Chris Lu 519b0e1e49 filer: upload to a directory without "/" suffix 
fix https://github.com/chrislusf/seaweedfs/issues/1988
 2021-04-12 11:56:56 -07:00
Chris Lu af313dff58 add gateway for easier POST and DELETE blobs 2021-04-10 23:47:47 -07:00
Chris Lu 0b82edc0d2 filer: avoid stuck uploader 
fix https://github.com/chrislusf/seaweedfs/issues/1980

reverting the file upload batch executor
 2021-04-09 13:05:15 -07:00
Merlin Gaillard f952f979d1 filer: return 304 when If-Modified-Since == Last-Modified 2021-04-09 15:04:17 +02:00
Chris Lu 2327c0756b fix to avoid loop 2021-04-05 23:24:26 -07:00
Chris Lu c131764c34 ensure error is sent back in channel 
fix https://github.com/chrislusf/seaweedfs/issues/1966

avoid shared readErr variable
 2021-04-05 19:40:12 -07:00
Chris Lu 8251d1140e refactor 2021-04-04 18:38:33 -07:00
Chris Lu 8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu 67e019d54b add missing changes 2021-04-02 01:11:44 -07:00
Chris Lu 7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu cefe66f159 dedicated upload processor 
avoid thundering effect of overloading volume servers
 2021-04-01 02:21:40 -07:00
Chris Lu b5880334fc refactor 2021-03-30 21:07:34 -07:00
Chris Lu e79e2ddeed remove unused MoveEvents 2021-03-30 20:42:44 -07:00
Chris Lu ac875976c0 filer, volume: add concurrent upload size limit to avoid OOM 
add some back pressure when writes are slow
 2021-03-30 02:10:53 -07:00
李海 06be5dc6c3 log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid 2021-03-25 18:59:40 +08:00
李海 69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Chris Lu a801332b0d filer: return 409 if file conflicts with a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1938
 2021-03-24 01:03:11 -07:00
Chris Lu 5d931eff27 avoid possible nil 
fix https://github.com/chrislusf/seaweedfs/issues/1928

The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.
 2021-03-22 13:24:07 -07:00
Chris Lu f315eb2bb8 fix build error 2021-03-22 00:12:53 -07:00
Chris Lu b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Konstantin Lebedev 06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Konstantin Lebedev 90510e3137 If WriteHeader is not called explicitly, the first call to Write 
will trigger an implicit WriteHeader(http.StatusOK).
WriteHeader: Only one header may be written. Go does not currently!
 2021-03-15 21:22:59 +05:00
Konstantin Lebedev 46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Chris Lu 5d446673a6 fix error printing 2021-03-10 17:13:04 -08:00
Chris Lu df72dc206d stats collect chunk upload retry count 2021-03-07 11:26:15 -08:00
Chris Lu 8363be8548 trim out trailing return character 2021-03-06 14:26:27 -08:00
Chris Lu 400de380f4 volume server: support tcp direct put/get/delete 2021-03-05 02:29:38 -08:00
Chris Lu e982b06bcd support IPv6 2021-03-02 20:59:39 -08:00
Chris Lu 543c8daa6d
Merge pull request #1844 from kmlebedev/processRangeRequest 2021-02-26 09:18:22 -08:00
Patrick Schmidt 5f7b024891 Show the real disk usage in stats calls 
Currently the file size of only one volume location is taken into
account in the stats. This commit multiplies the disk usages by the
amount of nodes holding a replica of the volume.
This will yield the expected amount of disk usage and matches the
total size calculations from before.
 2021-02-26 13:58:40 +01:00
Konstantin Lebedev ee21c0042e log error write entry 2021-02-26 14:18:01 +05:00
Chris Lu 7ba75e3d5a filer: do not return no content for empty files 
fix https://github.com/chrislusf/seaweedfs/issues/1831
fix https://github.com/chrislusf/seaweedfs/issues/1830
 2021-02-23 12:25:27 -08:00
Chris Lu 30b30b8fe0 volume.tier.move: passing non-empty disk type 2021-02-22 01:59:03 -08:00
Chris Lu 73958e357d add descriptive error if no free volumes 2021-02-18 19:10:20 -08:00
Chris Lu c2ad6f1047 webdav add replication setting 
fix https://github.com/chrislusf/seaweedfs/issues/1817
 2021-02-18 12:15:09 -08:00
bingoohuang 352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu 3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu 6daa932f5c refactoring to get master function, instead of passing master values directly 
this will enable retrying later
 2021-02-17 20:55:55 -08:00
Chris Lu aa17311063 s3: set filename when uploading file chunks 2021-02-17 20:54:53 -08:00
Chris Lu cd866664a8 skip JWT if fileId is empty 
related to https://github.com/chrislusf/seaweedfs/issues/1808
 2021-02-16 15:39:12 -08:00
Chris Lu 3fe628f04e use hdd instead of empty string 2021-02-16 03:03:00 -08:00
Chris Lu f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu 7ce647f27e support customizable disk type 2021-02-13 15:42:42 -08:00
Chris Lu 4bd8a692d8 disk type can be generic tags 2021-02-13 13:50:14 -08:00
Chris Lu 487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu 770393a48c volume: add capability to change disk type when moving a volume 2021-02-09 23:58:08 -08:00
Chris Lu 821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
bingoohuang 7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
bingoohuang 94ea3bd3a5 renaming NeedleMapType to NeedleMapKind 2021-02-07 09:00:03 +08:00
Chris Lu a331bbb3ae filer: should return 204 on DELETE to nonexistent file 
related to

https://github.com/chrislusf/seaweedfs/issues/1776
https://github.com/chrislusf/seaweedfs/issues/1160
 2021-02-03 00:40:31 -08:00
Chris Lu d67ccb66c0 webdav: can start together with "weed server" or "weed filer" 2021-01-31 22:16:52 -08:00
Chris Lu 76430790b9 webdav: cache to version specific folder 2021-01-31 22:02:03 -08:00
Chris Lu d475c89fcc go fmt 2021-01-28 15:23:46 -08:00
Chris Lu 990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu 00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu 6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu 096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu 80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu f8dbb03bdd filer: append operation returns final file size 2021-01-23 04:19:49 -08:00
Chris Lu 759482e2ac filer: proxy all http methods to volume servers 2021-01-23 03:52:38 -08:00
Chris Lu 937cfacc01 filer: add "proxyToFileId" to reverse proxy to a volume server 2021-01-23 03:43:48 -08:00
Chris Lu 711c3f3939 filer: fix http status 206 setting 
fix https://github.com/chrislusf/seaweedfs/issues/1753
 2021-01-22 00:26:30 -08:00
Chris Lu f98817cfe6 filer: support appending to a file 2021-01-20 13:40:32 -08:00
Chris Lu 1b8e3da295 refactoring 2021-01-20 12:59:54 -08:00
Chris Lu d5add83e85 filer store: add postgres2 2021-01-19 18:07:29 -08:00
Chris Lu 52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu a4063a5437 add stream list directory entries 2021-01-15 23:56:24 -08:00
Chris Lu f002e668de change limit to int64 in case of overflow 2021-01-14 23:10:37 -08:00
Chris Lu 698f58f7c4 filer, s3: add http status 206 as late as possible 
fix https://github.com/chrislusf/seaweedfs/issues/1746
 2021-01-14 02:59:40 -08:00
Chris Lu 0ef43a23a7 go fmt 2021-01-12 02:30:12 -08:00
Chris Lu 629c9962e7 filer: add leveldb3 
support dedicated leveldb instance for each bucket
 2021-01-12 02:29:44 -08:00
Chris Lu 394513f598 filer: ensure seamless meta data updates 2021-01-11 02:08:55 -08:00
Chris Lu 1efb51ba84 filer: change to saveToFilerLimit from cacheToFilerLimit 
short circuit saving small files to volume server
 2021-01-10 23:14:46 -08:00
henry b42c5b71c1 Seaweed custom header are not visible to Vue or javascript 2021-01-05 13:01:29 +08:00