gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
4061 commits 28 branches 278 tags 135 MiB
Commit graph

729 commits

Author SHA1 Message Date
Chris Lu 69eda558ef reduce the connections 
fix https://github.com/chrislusf/seaweedfs/issues/1277

this avoids sending error to wdclients, and clients retry the connections
 2020-04-22 20:43:34 -07:00
Chris Lu e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu 27128c7875 adjust logging 2020-04-21 16:13:13 -07:00
Chris Lu cb3985be70 go fmt 2020-04-20 17:48:06 -07:00
Chris Lu ebfab42a50 refactoring 2020-04-20 00:08:47 -07:00
Chris Lu 2955b96ef1 refactor 2020-04-19 23:54:32 -07:00
Chris Lu f373232227
Merge pull request #1280 from bukton123/filer_mongodb 
Add new filer support mongodb
 2020-04-19 20:35:19 -07:00
Chris Lu 076c8bd3bc filer master start up with default ip address instead of just localhost 2020-04-18 15:17:27 -07:00
bukton 290c6b7f01 Merge remote-tracking branch 'origin/master' into filer_mongodb 
# Conflicts:
#	go.mod
#	go.sum
#	weed/server/filer_server.go
 2020-04-19 00:21:45 +07:00
bukton cd9cccec55 add import mongodb to filer server 2020-04-18 22:30:13 +07:00
Chris Lu 826f96b6f3 fix unreachable code 2020-04-17 10:04:28 -07:00
Chris Lu bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu f5a748d33c refactoring 2020-04-16 02:55:09 -07:00
Chris Lu ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Reed aada5b9aa7 making volume stats size info readable. 2020-04-16 10:08:08 +08:00
Chris Lu 2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu 7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu 8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu 5361f999ed shell: only one shell is allowed to connect to the cluster 
fix https://github.com/chrislusf/seaweedfs/issues/1266
 2020-04-12 17:48:39 -07:00
Chris Lu d30483d642 re-enable system logs 2020-04-12 14:03:07 -07:00
Chris Lu 6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu b5a713be68 filer store: add redis2 redis_cluster2 as default redis store 2020-04-12 02:50:41 -07:00
Chris Lu 1c65656fb4 s3: add option to fsync buckets 2020-04-11 23:37:10 -07:00
Chris Lu e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu 1101a42e5c mv : create new folder, move children, and delete old folder 2020-04-10 01:37:03 -07:00
Chris Lu f6a7e79dc3 weed shell: simplify CLI option for filer 2020-04-08 23:57:15 -07:00
Chris Lu dc08e4098f add etag only for PUT or large chunked uploads 2020-04-08 09:13:26 -07:00
Chris Lu ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu 745f5d2a25 fix setting default value 2020-04-07 01:58:48 -07:00
Chris Lu 54debdc6f7 filer: fix configuration settings 2020-04-07 01:30:53 -07:00
Chris Lu f14b6a09a8 remove duplicated update events 2020-04-05 13:22:15 -07:00
Chris Lu 91da7057b1 refactoring 2020-04-05 13:11:43 -07:00
Chris Lu 2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu 7bc3c93512 add util.PathJoin 2020-04-05 12:40:46 -07:00
Chris Lu bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu 6e776aaafc loop through all masters 2020-04-03 00:47:48 -07:00
Chris Lu 4a5e381feb fail fast if filer configured wrong masters 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-03 00:47:33 -07:00
Chris Lu aaf5e658cc adjust parameter name 2020-04-03 00:41:05 -07:00
Chris Lu 6c96e51414 re-ordering 2020-04-03 00:40:54 -07:00
Chris Lu fee3319cbf handle gzip for image resizing 2020-04-02 23:44:24 -07:00
Chris Lu 73bc286377 volume: passing ext info via URL 2020-04-02 21:23:15 -07:00
Chris Lu cea52a4faf volume copying adds cleaning up on error 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 15:36:55 -07:00
Chris Lu c446438ca5
Merge pull request #1255 from levenlabs/ignore 
Added treat_replication_as_minimums master toml option
 2020-04-01 16:48:58 -07:00
James Hartig eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu 50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu 9dc0b1df8f refactoring to compile for windows OS 
windows os does not like to work with fuse
 2020-03-29 21:07:55 -07:00
Chris Lu a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu 826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu d1439c5bd3 fix FUSE read for large files 
FUSE expects ReadAt do not return partial filled buffer with a nil error.
 2020-03-27 04:50:51 -07:00
Chris Lu 34a0d424f8 FUSE mount: report max volume count available disk space 2020-03-26 23:06:21 -07:00
Chris Lu a5ce359999 refactoring 2020-03-26 22:46:49 -07:00
Chris Lu 6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu cddcb5cafe same booleans as before 2020-03-23 01:25:23 -07:00
Chris Lu f84c288852 purge 2020-03-23 01:14:55 -07:00
Chris Lu a5aa8be19b Update webdav_server.go 2020-03-23 01:14:42 -07:00
Chris Lu c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu c16dc57a58 fix max volume count reporting 2020-03-22 18:33:19 -07:00
Chris Lu 3137777d83 volume: automatically detect max volume count 2020-03-22 16:21:42 -07:00
Chris Lu 9f2c9b666b adjust log level 2020-03-22 01:39:08 -07:00
Chris Lu ae2ee379c0 consistent 64bit size 2020-03-22 01:37:46 -07:00
Chris Lu e93e986727 webdav: stream read 2020-03-22 01:30:27 -07:00
Chris Lu 65d2ea9fb0 FUSE mount: stream read data with buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1244
 2020-03-22 01:00:36 -07:00
Chris Lu 308688c8d0 filer: add back image resizing capability 2020-03-20 20:31:11 -07:00
Chris Lu f251d03673 refactoring 2020-03-20 15:46:16 -07:00
Chris Lu 11372dd548 refactoring 2020-03-20 15:38:04 -07:00
Chris Lu c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu bec6ec7db6 go fmt 2020-03-17 10:01:55 -07:00
Chris Lu c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu 560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu 7b37178716 filer: close stores if interrupted 2020-03-14 20:30:26 -07:00
Chris Lu e85da50ad4 remove tikv, move its support to "tikv" branch 2020-03-11 20:59:00 -07:00
Chris Lu 4237a813cc skip printouts 2020-03-11 01:13:54 -07:00
Chris Lu d214cefc2e filer: list diretory pagination make up for the expired entries 2020-03-09 23:28:01 -07:00
Chris Lu 0871d2cff0 volume: fix memory leak during compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1222
 2020-03-09 22:29:02 -07:00
Chris Lu 89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu 8a899992f2 filer: fix ttl parsing 
fix https://github.com/chrislusf/seaweedfs/issues/1225
 2020-03-09 00:16:10 -07:00
Chris Lu 2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu e04c1a1361 filer: remove chunk manifest file support 
This is not needed for filer
 2020-03-08 17:03:27 -07:00
Chris Lu 9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu 11fceaf2f7 refactoring 2020-03-07 18:09:30 -08:00
Chris Lu 1a75fbfce0 refactoring 2020-03-07 18:06:48 -08:00
Chris Lu afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu 9228ff192c encryption works 2020-03-07 11:08:57 -08:00
Chris Lu 0375ce2c2e filer: set mime type from volume server 2020-03-07 07:25:15 -08:00
Chris Lu dba35404e4 filer: HEAD response add content type 2020-03-07 06:08:08 -08:00
Chris Lu ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu 13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu 9b7dae8037 filer: fix uploaded chunked file size reporting bug. 2020-03-02 20:27:14 -08:00
Chris Lu dc40413847 fix error 2020-03-01 22:21:13 -08:00
Chris Lu 6a8484b4ae master able to list all master clients by type 2020-03-01 22:13:47 -08:00
Chris Lu 555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu 8c1da71402 remove dead code 2020-02-26 17:27:49 -08:00
Chris Lu 0156e2975a mount: add mode to run external to SeaweedFS container cluster 2020-02-26 16:46:01 -08:00
Chris Lu 97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu 892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu 0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu 6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu 5bcb44eda9 filer: move buckets folder to filer.toml since it should not be changed often 2020-02-24 14:42:57 -08:00
Chris Lu d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu b06b7ca6e6 adjust UI 2020-02-23 22:01:03 -08:00
Yoni Nakache 0d60bb4427
display bug, header is Free but data was Used 2020-02-24 07:47:27 +02:00
Chris Lu 346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server 
Update stats and status volume server
 2020-02-23 18:05:25 -08:00
LazyDBA247-Anyvision 4ff513d64d staus route: add DiskStatuses for disk in the volume server status 
whem monitoring server, better to know the status of the disks & volumes in a single route.
 2020-02-23 23:27:09 +02:00
Yoni Nakache cd4c9a365b
DiskStats: adding Total & Percent Usage 
making relevant data visible and readable.
 2020-02-23 22:33:47 +02:00
Chris Lu f55f49970f move volume server stats to grpc 2020-02-21 21:45:03 -08:00
Chris Lu 9b6296e77a volume: add grpc file read operation 
This is added more for performance benchmarking.
 2020-02-14 00:37:32 -08:00
Chris Lu 2a6db0fd43 volume: return 204 for unchanged file uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1196
 2020-02-11 09:45:02 -08:00
Chris Lu 382ff218d3 filer: set file size, streaming chunk file uploading 
fix https://github.com/chrislusf/seaweedfs/issues/1193
 2020-02-03 17:04:06 -08:00
Chris Lu 40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu d4cde5df49 return empty response if not found 2020-01-24 18:07:34 -08:00
Chris Lu 3cdb27fafc adjust logging 2020-01-24 17:55:39 -08:00
Chris Lu e6f9f5da3a adjust logging 2020-01-24 17:26:03 -08:00
Chris Lu d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu 9b01a99d9a adjust logging 2020-01-21 22:45:50 -08:00
Chris Lu 1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu 4e731f1c8b volume: copy volumes also include .vif file 2020-01-04 11:28:29 -08:00
Chris Lu df636e4c14 remove .vif files 2020-01-03 12:46:42 -08:00
Chris Lu 3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu 86d4b18eb5 filer: optional recursive deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1176
 2019-12-31 11:52:54 -08:00
Chris Lu 509f314350 keep .vif file for ec to normal volume conversion 2019-12-28 21:52:55 -08:00
Chris Lu 0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu 8fbc0a9163 fix edge cases 2019-12-28 12:59:31 -08:00
Chris Lu 37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu 2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu b3c83f9404 fix spelling 2019-12-28 11:16:10 -08:00
Chris Lu 48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
Chris Lu d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
Chris Lu 7ba6be2312 volume ui add readonly status 2019-12-24 14:55:26 -08:00
stlpmo f87ccbddb4 remove the http 304 response body 2019-12-24 17:56:50 +08:00
Chris Lu 8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu 09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu 36ddca9d1f filer: return not found 404 status if not found 
fix https://github.com/chrislusf/seaweedfs/issues/1160
 2019-12-18 21:04:40 -08:00
Chris Lu 58f88e530c volume: use sorted index map for readonly volumes 2019-12-18 01:21:21 -08:00
Chris Lu 014487b1a6 passing entry.Extended 2019-12-17 21:39:48 -08:00
Chris Lu e707211dad passing entry.exended 2019-12-17 21:22:16 -08:00
Chris Lu ad9d5ff5d0 adjust UI 2019-12-15 20:57:17 -08:00
Chris Lu 392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu bbb6ebc3c0 filer: DeleteFolderChildren for deleting large folders 2019-12-13 00:23:05 -08:00
Chris Lu 0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu d0b423bbc0 filer: increase directory listing pagination size 2019-12-12 09:11:10 -08:00