gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3538 commits 28 branches 278 tags 135 MiB
Commit graph

2077 commits

Author SHA1 Message Date
Chris Lu 2955b96ef1 refactor 2020-04-19 23:54:32 -07:00
Chris Lu df25ed3c5b logs 2020-04-19 23:37:50 -07:00
Chris Lu ce3cb25cfb working for in memory single log buffer 2020-04-19 23:37:50 -07:00
Chris Lu f373232227
Merge pull request #1280 from bukton123/filer_mongodb 
Add new filer support mongodb
 2020-04-19 20:35:19 -07:00
bukton da37455254 filer mongo add func create index unique directory, name 2020-04-20 08:13:57 +07:00
Chris Lu 4642ed565e fix tests 2020-04-19 10:42:58 -07:00
bukton 3defa718a1 add default config filer mongodb to scaffold.go 2020-04-19 17:52:38 +07:00
bukton d6d245dc89 filer mongodb add option poll size and fixed order by list meta 2020-04-19 17:51:32 +07:00
Chris Lu 71ffb98475 broker: add profiling 2020-04-19 03:03:40 -07:00
Chris Lu d278b4c228 basic pub and sub works! 2020-04-19 00:18:32 -07:00
bukton 082f079ead filer mongo change func update to insert entry 2020-04-19 11:08:10 +07:00
Chris Lu 788578d4ea add subscriber api 2020-04-18 16:05:29 -07:00
Chris Lu 767f14dfcd master, volume default to detected host 2020-04-18 15:20:20 -07:00
Chris Lu 076c8bd3bc filer master start up with default ip address instead of just localhost 2020-04-18 15:17:27 -07:00
Chris Lu 5d346d44bd Revert "move go client out" 
This reverts commit ef1274edc9.
 2020-04-18 12:44:55 -07:00
Chris Lu ef1274edc9 move go client out 2020-04-18 12:11:50 -07:00
Chris Lu 5af49c7aa5 adjust API 2020-04-18 11:45:57 -07:00
bukton 8c58b0d017 filer mongodb delete folder children 2020-04-19 00:31:49 +07:00
bukton 290c6b7f01 Merge remote-tracking branch 'origin/master' into filer_mongodb 
# Conflicts:
#	go.mod
#	go.sum
#	weed/server/filer_server.go
 2020-04-19 00:21:45 +07:00
bukton 6234ea441b filer mongodb delete 2020-04-19 00:20:00 +07:00
bukton 24a8192298 filer mongo list 2020-04-19 00:16:50 +07:00
bukton 3c70163798 filer mongodb insert entry and find entry 2020-04-18 23:48:38 +07:00
bukton cd9cccec55 add import mongodb to filer server 2020-04-18 22:30:13 +07:00
bukton fd4576651b add new package mongodb and install lib mongodb 2020-04-18 22:26:57 +07:00
Chris Lu 11f5a6d913 messaging can compile now 2020-04-18 01:12:01 -07:00
Chris Lu 5bea77010f refactor 2020-04-17 22:39:21 -07:00
Chris Lu 2e2537a9ea rename files 2020-04-17 22:35:44 -07:00
Chris Lu e0f5996560 fix "call of Unmarshal passes non-pointer as second argument" 2020-04-17 10:04:41 -07:00
Chris Lu 826f96b6f3 fix unreachable code 2020-04-17 10:04:28 -07:00
Chris Lu ffa69b24ae fix travis 2020-04-17 10:01:11 -07:00
Chris Lu 3f3dba5a68 broker: append message logs 2020-04-17 02:29:38 -07:00
Chris Lu 2a45897237 broker: read cipher value from filer 2020-04-17 02:29:00 -07:00
Chris Lu bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu 722b14337e add cipher option to meta data updates 2020-04-17 01:26:27 -07:00
Chris Lu 7d3672c60a simplify 2020-04-17 00:00:48 -07:00
Chris Lu 508f3490a0 update messaging proto 2020-04-16 03:29:57 -07:00
Chris Lu f5a748d33c refactoring 2020-04-16 02:55:09 -07:00
Chris Lu ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Reed aada5b9aa7 making volume stats size info readable. 2020-04-16 10:08:08 +08:00
Chris Lu 9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu 3f1d79512f only detect when not sure about the types 2020-04-14 11:02:05 -07:00
Chris Lu d2d1597850 volume: detect gzip effectiveness only when content type is empty 
revert part of 2286eda575
 2020-04-14 10:52:53 -07:00
Chris Lu 23715af402
Merge pull request #1273 from wyxloading/config_race 
Fix data race about config
 2020-04-13 22:24:51 -07:00
Chris Lu 2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu f282ed444b refactoring 2020-04-13 21:58:10 -07:00
wuyuxiang b556d3d035 Fix data race about config 2020-04-14 10:22:40 +08:00
Chris Lu d8f5985e5e add logs 2020-04-13 12:58:45 -07:00
Chris Lu 5c9e91393f 1.74 2020-04-13 10:25:48 -07:00
Chris Lu 7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu 8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu b062393f3f refactoring 2020-04-12 17:51:31 -07:00
Chris Lu 5361f999ed shell: only one shell is allowed to connect to the cluster 
fix https://github.com/chrislusf/seaweedfs/issues/1266
 2020-04-12 17:48:39 -07:00
Chris Lu eb16bb0287 go mod cleanup 2020-04-12 16:44:58 -07:00
Chris Lu d30483d642 re-enable system logs 2020-04-12 14:03:07 -07:00
Chris Lu 6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu 94e35cdb35 mount: fix 
fix bug found by git bisect, but I do not understand why it can cause error!
 2020-04-12 03:34:36 -07:00
Chris Lu 1c2e920fa2 fix compilation error 2020-04-12 03:11:19 -07:00
Chris Lu b5a713be68 filer store: add redis2 redis_cluster2 as default redis store 2020-04-12 02:50:41 -07:00
Chris Lu e613695af7 a little optimization 2020-04-12 01:13:57 -07:00
Chris Lu 2a1f396df5 avoid duplicated setting chunks into cache 2020-04-12 01:06:50 -07:00
Chris Lu b9b7da905e handle nil chunk cache 2020-04-12 01:00:12 -07:00
Chris Lu 211d87cf4c mount: option to disable caching 2020-04-12 00:52:54 -07:00
Chris Lu 1c65656fb4 s3: add option to fsync buckets 2020-04-11 23:37:10 -07:00
Chris Lu e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu 5987810e5e set default chunk size to 16 2020-04-11 21:22:52 -07:00
Chris Lu df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu c8ca234773 refactoring 2020-04-11 14:27:25 -07:00
Chris Lu df9d538044 rename function 2020-04-11 14:19:44 -07:00
Chris Lu d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu ed54682598 refactoring 2020-04-11 12:43:17 -07:00
Chris Lu e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu b7f0ba3800 refactoring 2020-04-11 12:37:41 -07:00
Chris Lu 417125457e add additional buffers for logs 2020-04-11 02:19:48 -07:00
Chris Lu 4b27afe176 fix sqs message reading 2020-04-10 15:27:47 -07:00
Chris Lu dce2702e1b fix the instruction 2020-04-10 14:50:10 -07:00
Chris Lu 13ab1fdaf0 mount: use file default permission 0666 2020-04-10 02:23:23 -07:00
Chris Lu 1101a42e5c mv : create new folder, move children, and delete old folder 2020-04-10 01:37:03 -07:00
Chris Lu bcf37346ef add timestamp inside lock 2020-04-10 01:35:59 -07:00
Chris Lu 9fa065f600 typo in logs 2020-04-09 23:43:09 -07:00
Chris Lu 8db2120bee add logs 2020-04-09 23:42:59 -07:00
Chris Lu 43c62a8a95 1.73 2020-04-09 21:44:30 -07:00
Chris Lu eb39df2704 avoid dead loop 2020-04-09 21:43:26 -07:00
Chris Lu 56ec8c4513 fix to avoid nil file an dirty pages 2020-04-09 21:43:05 -07:00
Chris Lu 59f40e2027 volume: best effort to detect ip address 
fix https://github.com/chrislusf/seaweedfs/issues/1264
 2020-04-09 00:26:24 -07:00
Chris Lu f6a7e79dc3 weed shell: simplify CLI option for filer 2020-04-08 23:57:15 -07:00
Chris Lu 2329d9e0c1 add volume.fix.replication to default scaffold 2020-04-08 23:36:22 -07:00
Chris Lu bb78ab9915 wait for master to be aware of the failed volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1268
 2020-04-08 23:12:37 -07:00
Chris Lu 8764bdb9df 1.72 2020-04-08 22:35:14 -07:00
Chris Lu 8a73276602 refactor a bit 2020-04-08 22:33:36 -07:00
Chris Lu 006f78b70b make a copy of request data 
to avoid concurrency issues
 2020-04-08 22:31:19 -07:00
Chris Lu 00b993a234 add util for md5 2020-04-08 22:30:38 -07:00
Chris Lu ed98223b08 still log, but not persisting the changes 2020-04-08 20:32:57 -07:00
Chris Lu 6630541399 ensure correct file size when opening existing files 2020-04-08 12:50:59 -07:00
Chris Lu b524a40375 add locking to fs cache 2020-04-08 12:50:34 -07:00
Chris Lu bd56172b82 simplify file handle management 2020-04-08 12:50:20 -07:00
Chris Lu dc08e4098f add etag only for PUT or large chunked uploads 2020-04-08 09:13:26 -07:00
Chris Lu ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu 4c498e73de filer replication: add s3 endpoint 2020-04-07 17:49:00 -07:00
Chris Lu 745f5d2a25 fix setting default value 2020-04-07 01:58:48 -07:00
Chris Lu 54debdc6f7 filer: fix configuration settings 2020-04-07 01:30:53 -07:00
Chris Lu 90d576affe disable meta data change event logging for now. 2020-04-07 01:05:24 -07:00
Chris Lu b51d8f03bb 1.71 2020-04-05 22:04:16 -07:00
Chris Lu 5a5908407d filer: support larger file size 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-05 21:48:45 -07:00
Chris Lu 19edd9c091 comment out debug messages 2020-04-05 20:15:17 -07:00
Chris Lu 78ea937606 fix 2020-04-05 17:34:26 -07:00
Chris Lu 53626734d4 iterate through the log buffer 2020-04-05 16:51:30 -07:00
Chris Lu f71b855db5 adjust caching and notification for recursive directory deletion 2020-04-05 15:03:25 -07:00
Chris Lu f14b6a09a8 remove duplicated update events 2020-04-05 13:22:15 -07:00
Chris Lu 91da7057b1 refactoring 2020-04-05 13:11:43 -07:00
Chris Lu 2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu 7bc3c93512 add util.PathJoin 2020-04-05 12:40:46 -07:00
Chris Lu af1f64d244 change from 'weed tail' to 'weed watch' 2020-04-05 12:39:20 -07:00
Chris Lu b10679fcf0 adjust log file names 2020-04-05 12:38:46 -07:00
Chris Lu 2cb1700ecb set to 4MB meta log buffer size 2020-04-05 12:38:29 -07:00
Chris Lu e78ee087e9 fix tests 2020-04-05 01:17:35 -07:00
Chris Lu dcc6b6a3ed go fmt 2020-04-05 00:52:31 -07:00
Chris Lu bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu f7b5a1d697 add disk stat to logs 2020-04-04 13:17:23 -07:00
Chris Lu 6e776aaafc loop through all masters 2020-04-03 00:47:48 -07:00
Chris Lu 4a5e381feb fail fast if filer configured wrong masters 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-03 00:47:33 -07:00
Chris Lu aaf5e658cc adjust parameter name 2020-04-03 00:41:05 -07:00
Chris Lu 6c96e51414 re-ordering 2020-04-03 00:40:54 -07:00
Chris Lu d2f8e50459 make error a little bit more clearer 
fix https://github.com/chrislusf/seaweedfs/issues/1259
 2020-04-03 00:24:09 -07:00
Chris Lu 5f881d2fd5 mount: fail early rather than after mounted the directory 
fix https://github.com/chrislusf/seaweedfs/issues/1258
 2020-04-03 00:21:03 -07:00
Chris Lu fee3319cbf handle gzip for image resizing 2020-04-02 23:44:24 -07:00
Chris Lu 73bc286377 volume: passing ext info via URL 2020-04-02 21:23:15 -07:00
Chris Lu cea52a4faf volume copying adds cleaning up on error 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 15:36:55 -07:00
Chris Lu c981975627 volume.fix.replication: add test cases for complicated moving 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 02:16:16 -07:00
Chris Lu ae3d08883f volume.fsck: fix for erasure encoded volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1254
 2020-04-01 22:10:13 -07:00
Chris Lu c446438ca5
Merge pull request #1255 from levenlabs/ignore 
Added treat_replication_as_minimums master toml option
 2020-04-01 16:48:58 -07:00
James Hartig eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu 2fc2eb74dd refactoring 2020-03-30 22:57:45 -07:00
Chris Lu c22094d340 adjust logging 2020-03-30 13:03:43 -07:00
Chris Lu 50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu 9dc0b1df8f refactoring to compile for windows OS 
windows os does not like to work with fuse
 2020-03-29 21:07:55 -07:00
Chris Lu be6c2f8511 1.70 2020-03-29 13:40:55 -07:00
Chris Lu 54768d0761 fix: delete a file and then create a directory with the same name 2020-03-29 01:39:48 -07:00
Chris Lu 057722bbf4 return part of the chunk if chunkview is not the full chunk 2020-03-29 00:54:39 -07:00
Chris Lu af6d9976ad adjust print out 2020-03-28 14:07:25 -07:00
Chris Lu a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu 826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu 4aa82c95e6 refactor 2020-03-28 13:42:35 -07:00
Chris Lu eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu 005b4ab3fe 1.69 2020-03-27 04:51:31 -07:00
Chris Lu d1439c5bd3 fix FUSE read for large files 
FUSE expects ReadAt do not return partial filled buffer with a nil error.
 2020-03-27 04:50:51 -07:00
Chris Lu f06ca04451 avoid overflow 2020-03-27 04:35:31 -07:00
Chris Lu e1911760a7 refactoring 2020-03-27 00:30:55 -07:00
Chris Lu 2f5ed29c37 clean up 2020-03-27 00:30:39 -07:00
Chris Lu 2286eda575 sampling whether the data can be gzipped 2020-03-26 23:50:48 -07:00
Chris Lu 5463560912 small test 2020-03-26 23:50:07 -07:00