gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3501 commits 28 branches 278 tags 135 MiB
Commit graph

2048 commits

Author SHA1 Message Date
Chris Lu ffa69b24ae fix travis 2020-04-17 10:01:11 -07:00
Chris Lu 3f3dba5a68 broker: append message logs 2020-04-17 02:29:38 -07:00
Chris Lu 2a45897237 broker: read cipher value from filer 2020-04-17 02:29:00 -07:00
Chris Lu bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu 722b14337e add cipher option to meta data updates 2020-04-17 01:26:27 -07:00
Chris Lu 7d3672c60a simplify 2020-04-17 00:00:48 -07:00
Chris Lu 508f3490a0 update messaging proto 2020-04-16 03:29:57 -07:00
Chris Lu f5a748d33c refactoring 2020-04-16 02:55:09 -07:00
Chris Lu ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Reed aada5b9aa7 making volume stats size info readable. 2020-04-16 10:08:08 +08:00
Chris Lu 9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu 3f1d79512f only detect when not sure about the types 2020-04-14 11:02:05 -07:00
Chris Lu d2d1597850 volume: detect gzip effectiveness only when content type is empty 
revert part of 2286eda575
 2020-04-14 10:52:53 -07:00
Chris Lu 23715af402
Merge pull request #1273 from wyxloading/config_race 
Fix data race about config
 2020-04-13 22:24:51 -07:00
Chris Lu 2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu f282ed444b refactoring 2020-04-13 21:58:10 -07:00
wuyuxiang b556d3d035 Fix data race about config 2020-04-14 10:22:40 +08:00
Chris Lu d8f5985e5e add logs 2020-04-13 12:58:45 -07:00
Chris Lu 5c9e91393f 1.74 2020-04-13 10:25:48 -07:00
Chris Lu 7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu 8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu b062393f3f refactoring 2020-04-12 17:51:31 -07:00
Chris Lu 5361f999ed shell: only one shell is allowed to connect to the cluster 
fix https://github.com/chrislusf/seaweedfs/issues/1266
 2020-04-12 17:48:39 -07:00
Chris Lu eb16bb0287 go mod cleanup 2020-04-12 16:44:58 -07:00
Chris Lu d30483d642 re-enable system logs 2020-04-12 14:03:07 -07:00
Chris Lu 6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu 94e35cdb35 mount: fix 
fix bug found by git bisect, but I do not understand why it can cause error!
 2020-04-12 03:34:36 -07:00
Chris Lu 1c2e920fa2 fix compilation error 2020-04-12 03:11:19 -07:00
Chris Lu b5a713be68 filer store: add redis2 redis_cluster2 as default redis store 2020-04-12 02:50:41 -07:00
Chris Lu e613695af7 a little optimization 2020-04-12 01:13:57 -07:00
Chris Lu 2a1f396df5 avoid duplicated setting chunks into cache 2020-04-12 01:06:50 -07:00
Chris Lu b9b7da905e handle nil chunk cache 2020-04-12 01:00:12 -07:00
Chris Lu 211d87cf4c mount: option to disable caching 2020-04-12 00:52:54 -07:00
Chris Lu 1c65656fb4 s3: add option to fsync buckets 2020-04-11 23:37:10 -07:00
Chris Lu e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu 5987810e5e set default chunk size to 16 2020-04-11 21:22:52 -07:00
Chris Lu df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu c8ca234773 refactoring 2020-04-11 14:27:25 -07:00
Chris Lu df9d538044 rename function 2020-04-11 14:19:44 -07:00
Chris Lu d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu ed54682598 refactoring 2020-04-11 12:43:17 -07:00
Chris Lu e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu b7f0ba3800 refactoring 2020-04-11 12:37:41 -07:00
Chris Lu 417125457e add additional buffers for logs 2020-04-11 02:19:48 -07:00
Chris Lu 4b27afe176 fix sqs message reading 2020-04-10 15:27:47 -07:00
Chris Lu dce2702e1b fix the instruction 2020-04-10 14:50:10 -07:00
Chris Lu 13ab1fdaf0 mount: use file default permission 0666 2020-04-10 02:23:23 -07:00
Chris Lu 1101a42e5c mv : create new folder, move children, and delete old folder 2020-04-10 01:37:03 -07:00
Chris Lu bcf37346ef add timestamp inside lock 2020-04-10 01:35:59 -07:00
Chris Lu 9fa065f600 typo in logs 2020-04-09 23:43:09 -07:00
Chris Lu 8db2120bee add logs 2020-04-09 23:42:59 -07:00
Chris Lu 43c62a8a95 1.73 2020-04-09 21:44:30 -07:00
Chris Lu eb39df2704 avoid dead loop 2020-04-09 21:43:26 -07:00
Chris Lu 56ec8c4513 fix to avoid nil file an dirty pages 2020-04-09 21:43:05 -07:00
Chris Lu 59f40e2027 volume: best effort to detect ip address 
fix https://github.com/chrislusf/seaweedfs/issues/1264
 2020-04-09 00:26:24 -07:00
Chris Lu f6a7e79dc3 weed shell: simplify CLI option for filer 2020-04-08 23:57:15 -07:00
Chris Lu 2329d9e0c1 add volume.fix.replication to default scaffold 2020-04-08 23:36:22 -07:00
Chris Lu bb78ab9915 wait for master to be aware of the failed volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1268
 2020-04-08 23:12:37 -07:00
Chris Lu 8764bdb9df 1.72 2020-04-08 22:35:14 -07:00
Chris Lu 8a73276602 refactor a bit 2020-04-08 22:33:36 -07:00
Chris Lu 006f78b70b make a copy of request data 
to avoid concurrency issues
 2020-04-08 22:31:19 -07:00
Chris Lu 00b993a234 add util for md5 2020-04-08 22:30:38 -07:00
Chris Lu ed98223b08 still log, but not persisting the changes 2020-04-08 20:32:57 -07:00
Chris Lu 6630541399 ensure correct file size when opening existing files 2020-04-08 12:50:59 -07:00
Chris Lu b524a40375 add locking to fs cache 2020-04-08 12:50:34 -07:00
Chris Lu bd56172b82 simplify file handle management 2020-04-08 12:50:20 -07:00
Chris Lu dc08e4098f add etag only for PUT or large chunked uploads 2020-04-08 09:13:26 -07:00
Chris Lu ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu 4c498e73de filer replication: add s3 endpoint 2020-04-07 17:49:00 -07:00
Chris Lu 745f5d2a25 fix setting default value 2020-04-07 01:58:48 -07:00
Chris Lu 54debdc6f7 filer: fix configuration settings 2020-04-07 01:30:53 -07:00
Chris Lu 90d576affe disable meta data change event logging for now. 2020-04-07 01:05:24 -07:00
Chris Lu b51d8f03bb 1.71 2020-04-05 22:04:16 -07:00
Chris Lu 5a5908407d filer: support larger file size 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-05 21:48:45 -07:00
Chris Lu 19edd9c091 comment out debug messages 2020-04-05 20:15:17 -07:00
Chris Lu 78ea937606 fix 2020-04-05 17:34:26 -07:00
Chris Lu 53626734d4 iterate through the log buffer 2020-04-05 16:51:30 -07:00
Chris Lu f71b855db5 adjust caching and notification for recursive directory deletion 2020-04-05 15:03:25 -07:00
Chris Lu f14b6a09a8 remove duplicated update events 2020-04-05 13:22:15 -07:00
Chris Lu 91da7057b1 refactoring 2020-04-05 13:11:43 -07:00
Chris Lu 2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu 7bc3c93512 add util.PathJoin 2020-04-05 12:40:46 -07:00
Chris Lu af1f64d244 change from 'weed tail' to 'weed watch' 2020-04-05 12:39:20 -07:00
Chris Lu b10679fcf0 adjust log file names 2020-04-05 12:38:46 -07:00
Chris Lu 2cb1700ecb set to 4MB meta log buffer size 2020-04-05 12:38:29 -07:00
Chris Lu e78ee087e9 fix tests 2020-04-05 01:17:35 -07:00
Chris Lu dcc6b6a3ed go fmt 2020-04-05 00:52:31 -07:00
Chris Lu bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu f7b5a1d697 add disk stat to logs 2020-04-04 13:17:23 -07:00
Chris Lu 6e776aaafc loop through all masters 2020-04-03 00:47:48 -07:00
Chris Lu 4a5e381feb fail fast if filer configured wrong masters 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-03 00:47:33 -07:00
Chris Lu aaf5e658cc adjust parameter name 2020-04-03 00:41:05 -07:00
Chris Lu 6c96e51414 re-ordering 2020-04-03 00:40:54 -07:00
Chris Lu d2f8e50459 make error a little bit more clearer 
fix https://github.com/chrislusf/seaweedfs/issues/1259
 2020-04-03 00:24:09 -07:00
Chris Lu 5f881d2fd5 mount: fail early rather than after mounted the directory 
fix https://github.com/chrislusf/seaweedfs/issues/1258
 2020-04-03 00:21:03 -07:00
Chris Lu fee3319cbf handle gzip for image resizing 2020-04-02 23:44:24 -07:00
Chris Lu 73bc286377 volume: passing ext info via URL 2020-04-02 21:23:15 -07:00
Chris Lu cea52a4faf volume copying adds cleaning up on error 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 15:36:55 -07:00
Chris Lu c981975627 volume.fix.replication: add test cases for complicated moving 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 02:16:16 -07:00
Chris Lu ae3d08883f volume.fsck: fix for erasure encoded volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1254
 2020-04-01 22:10:13 -07:00
Chris Lu c446438ca5
Merge pull request #1255 from levenlabs/ignore 
Added treat_replication_as_minimums master toml option
 2020-04-01 16:48:58 -07:00
James Hartig eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu 2fc2eb74dd refactoring 2020-03-30 22:57:45 -07:00
Chris Lu c22094d340 adjust logging 2020-03-30 13:03:43 -07:00
Chris Lu 50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu 9dc0b1df8f refactoring to compile for windows OS 
windows os does not like to work with fuse
 2020-03-29 21:07:55 -07:00
Chris Lu be6c2f8511 1.70 2020-03-29 13:40:55 -07:00
Chris Lu 54768d0761 fix: delete a file and then create a directory with the same name 2020-03-29 01:39:48 -07:00
Chris Lu 057722bbf4 return part of the chunk if chunkview is not the full chunk 2020-03-29 00:54:39 -07:00
Chris Lu af6d9976ad adjust print out 2020-03-28 14:07:25 -07:00
Chris Lu a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu 826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu 4aa82c95e6 refactor 2020-03-28 13:42:35 -07:00
Chris Lu eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu 005b4ab3fe 1.69 2020-03-27 04:51:31 -07:00
Chris Lu d1439c5bd3 fix FUSE read for large files 
FUSE expects ReadAt do not return partial filled buffer with a nil error.
 2020-03-27 04:50:51 -07:00
Chris Lu f06ca04451 avoid overflow 2020-03-27 04:35:31 -07:00
Chris Lu e1911760a7 refactoring 2020-03-27 00:30:55 -07:00
Chris Lu 2f5ed29c37 clean up 2020-03-27 00:30:39 -07:00
Chris Lu 2286eda575 sampling whether the data can be gzipped 2020-03-26 23:50:48 -07:00
Chris Lu 5463560912 small test 2020-03-26 23:50:07 -07:00
Chris Lu 34a0d424f8 FUSE mount: report max volume count available disk space 2020-03-26 23:06:21 -07:00
Chris Lu a5ce359999 refactoring 2020-03-26 22:46:49 -07:00
Chris Lu 0a05052b5c 1.68 2020-03-26 22:22:36 -07:00
Chris Lu e7e8691720 add some tests 2020-03-26 22:21:52 -07:00
Chris Lu 86398d1efe S3 API: set deletion to recursive 
fix https://github.com/chrislusf/seaweedfs/issues/1251
 2020-03-26 11:05:54 -07:00
Chris Lu 1fef598347 set target Dir object when moving 2020-03-26 10:56:40 -07:00
Chris Lu 9e2e07b238 tweak 2020-03-26 10:56:18 -07:00
Chris Lu cbce793753 rename: set new name for renamed files 2020-03-26 03:30:23 -07:00
Chris Lu c763a492f5 refactor 2020-03-26 03:30:02 -07:00
Chris Lu 712720864d error checking for remove 2020-03-26 02:21:30 -07:00
Chris Lu 8d9a76005d add back running output 2020-03-26 02:14:48 -07:00
Chris Lu dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu 7f0d87b206 tree structured fs cache 
FsCache for FsNode, wrapping fs.Node
 2020-03-26 00:08:14 -07:00
Chris Lu 2e4fadd10a fix fscache move 2020-03-25 22:19:19 -07:00
Chris Lu 966df2ae27 purging skip EC volumes 2020-03-25 02:41:22 -07:00
Chris Lu 6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu 5ebc95b69b refactoring 2020-03-25 02:20:19 -07:00
Chris Lu d79f0a76b1 format output 2020-03-25 00:56:47 -07:00
Chris Lu d9b56ed88b shell: volume.fsck add options 2020-03-24 22:38:36 -07:00
Chris Lu e63a79ade8 better handle lock in case of exception 2020-03-24 18:41:25 -07:00
Chris Lu 4d5554b16f fix compilation on travis 2020-03-24 18:40:52 -07:00
Chris Lu d83e648c0b auto add prefix "fs." 2020-03-24 02:40:51 -07:00
Chris Lu a875f67a08 fix pwd printout 2020-03-24 02:40:29 -07:00
Chris Lu 00d3f01876 better output format 2020-03-24 02:34:28 -07:00
Chris Lu 0820935290 clean up 2020-03-24 02:20:37 -07:00
Chris Lu 49440cbd2f shell: add volume.fsck 
volume.fsck can find out how much space is not used by the filer
 2020-03-24 02:18:13 -07:00
Chris Lu 0f10833e0b fix compilation 2020-03-24 00:16:12 -07:00
Chris Lu 3794f2721f fix logic 2020-03-24 00:09:12 -07:00
Chris Lu 443b229ef7 disable chunk file output 2020-03-24 00:08:02 -07:00