gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3627 commits 28 branches 278 tags 135 MiB
Commit graph

2144 commits

Author SHA1 Message Date
Chris Lu 97a504b0bb fix possible directory deletion error 2020-05-02 01:11:53 -07:00
Chris Lu 78422c2f69
Merge pull request #1298 from song-zhang/master 
add fsync option for benchmark
 2020-04-30 20:16:42 -07:00
Chris Lu 871efa4fc1 refactoring 
some previous chunk etag was using md5, which should be wrong.
 2020-04-30 17:20:47 -07:00
zhangsong d819865fe7 add fsync option for benchmark 2020-04-30 23:31:08 +08:00
Chris Lu ea809a45ab add cipher key 2020-04-30 03:05:43 -07:00
Chris Lu f9b6178b8f log messages 2020-04-30 03:05:34 -07:00
Chris Lu 8c73410a51 subscribe from a timestamp 2020-04-30 02:19:51 -07:00
Chris Lu 4e16a90454 for debugging 2020-04-30 02:19:32 -07:00
Chris Lu a25a8d8822 s3: print out time in UTC format 
fix https://github.com/chrislusf/seaweedfs/issues/1297
 2020-04-29 21:35:24 -07:00
Chris Lu e93588ec78 FUSE mount: atomic local cache updates 2020-04-29 18:20:54 -07:00
Chris Lu 9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
Chris Lu 7c10602b49 read in case cross chunks 2020-04-29 02:42:58 -07:00
Chris Lu f9da859720 better error message 2020-04-29 02:42:06 -07:00
Chris Lu d57559d3f7 chunk rememober encryption or not 2020-04-29 02:41:47 -07:00
Chris Lu 5a0986dca0 reset on getting the buffer 2020-04-29 02:41:01 -07:00
Chris Lu 8d42a1263b meta subscription: update the last read time 2020-04-28 17:30:04 -07:00
Chris Lu 31583b5d6c master and volume server: avoid race condition 
The volume server may disconnect and reconnect to the same master.

The master's unregistration may happen after the reconnection. Thus the volume server will disappear.
 2020-04-28 17:29:10 -07:00
Chris Lu 98edec7c3e remove printouts 2020-04-28 16:45:48 -07:00
Chris Lu 8ed490164e refactoring 2020-04-28 02:05:44 -07:00
Chris Lu 66effaed9e fix compilation 2020-04-28 02:05:32 -07:00
Chris Lu eab4c9219c randomize benchmark content 
fix https://github.com/chrislusf/seaweedfs/issues/1294
 2020-04-28 01:50:56 -07:00
Chris Lu fb81f12686 filer: simplify image resize 2020-04-28 00:05:57 -07:00
Chris Lu 5c57297bd1 metadata log: read from any timestamp 2020-04-28 00:05:57 -07:00
wuyuxiang 6850d28d6b refacotr: move signal handling and pprof to grace package 2020-04-28 14:10:23 +08:00
Chris Lu 2a7957b4ca FUSE: subcribe meta events based on timestamp 2020-04-27 16:52:20 -07:00
Chris Lu 6190fd665d printout error 2020-04-27 12:41:31 -07:00
Chris Lu 725e47dcc3 filer: pass along user name and group name for HDFS 2020-04-27 05:48:48 -07:00
Chris Lu 508ec01ab0 tweaking 2020-04-26 16:11:06 -07:00
Chris Lu f4fcde05b6 1.77 2020-04-26 05:54:47 -07:00
Chris Lu b52b8ec685 Hadoop: fix entry not found for HCFS 
also fix cipher related changes.
 2020-04-26 05:21:54 -07:00
Chris Lu 0c2248f83a add lock/unlock to example admin scripts 2020-04-23 18:41:48 -07:00
Chris Lu 913aa0f14e mount: default to enable asyncMetaDataCaching 2020-04-23 14:14:28 -07:00
Chris Lu 662b5d0cf7 support semicolon seperated command lines 2020-04-23 14:01:46 -07:00
Chris Lu 73564e6a01 master: add cluster wide lock/unlock operation in weed shell 
fix https://github.com/chrislusf/seaweedfs/issues/1286
 2020-04-23 13:37:31 -07:00
Chris Lu bdc337a719 make lock/unlock optional 2020-04-23 03:50:05 -07:00
Chris Lu 57ec7f2ba9 fix builds 2020-04-23 03:43:45 -07:00
Chris Lu eab9701823 allow lock with an existing lock 2020-04-23 03:35:52 -07:00
Chris Lu 842e3301ee make second lock instant 2020-04-23 03:32:18 -07:00
Chris Lu 69f336e59f shell: add lock/unlock command 2020-04-23 03:11:07 -07:00
Chris Lu ff0a7c1d18 clean up 2020-04-23 03:10:47 -07:00
Chris Lu 30ee4f3291 add exclusive lock library on shell 2020-04-23 02:31:04 -07:00
Chris Lu 77873b832b add master side code for cluster wise exclusive lock 2020-04-23 01:55:44 -07:00
Chris Lu 369aa8a10a 1.76 2020-04-22 23:31:59 -07:00
Chris Lu 69eda558ef reduce the connections 
fix https://github.com/chrislusf/seaweedfs/issues/1277

this avoids sending error to wdclients, and clients retry the connections
 2020-04-22 20:43:34 -07:00
Chris Lu aebe39a803 avoid repeated grpc connection creation 
fix https://github.com/chrislusf/seaweedfs/issues/1277
 2020-04-22 18:26:24 -07:00
Chris Lu 5d0e1d8d74 also writes to local meta cache 
before waiting for subscribed meta events
 2020-04-22 15:40:47 -07:00
Chris Lu 72c5dba4d4 1.75 2020-04-21 22:03:07 -07:00
Chris Lu a207285af7 cache metadata on startup 2020-04-21 22:00:34 -07:00
Chris Lu 9f9826e95a refactor 2020-04-21 21:28:47 -07:00
Chris Lu e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu 4f02f7121d read from meta cache 
meta cache is not initialized
 2020-04-21 18:50:30 -07:00
Chris Lu b8e4238ad2 add placeholder for cached meta data reading 2020-04-21 18:02:08 -07:00
Chris Lu 27128c7875 adjust logging 2020-04-21 16:13:13 -07:00
Chris Lu bafa95045b volume: deletion checks all disk locations 
fix https://github.com/chrislusf/seaweedfs/issues/1283
 2020-04-21 14:49:58 -07:00
Chris Lu 57df14f76f report error if copying failed 2020-04-21 14:28:46 -07:00
Chris Lu cf54d70311 filer: add -ip.bind option 2020-04-21 14:21:06 -07:00
Chris Lu 5c348087dc messaging: able to pub sub multiple partitions 2020-04-21 00:59:55 -07:00
Chris Lu cb3985be70 go fmt 2020-04-20 17:48:06 -07:00
Chris Lu 5eb83dfbd3 add memory buffer size limit 2020-04-20 17:43:50 -07:00
Chris Lu 258fba8a0f continue for reading from sealed memory buffer 2020-04-20 17:28:18 -07:00
Chris Lu 4bf959edf0 message broker: read also from sealed memory buffer 2020-04-20 17:26:38 -07:00
Chris Lu bd43c62fbd lower the test criteria 2020-04-20 03:08:10 -07:00
Chris Lu c3f9fdf48f adjust test 2020-04-20 02:54:21 -07:00
Chris Lu 5e8bb86995 close the connection with better accouting 2020-04-20 02:53:50 -07:00
Chris Lu 978da73cdb revert this grpc related change 
maybe related to https://github.com/chrislusf/seaweedfs/issues/1277
 2020-04-20 02:53:22 -07:00
Chris Lu ebfab42a50 refactoring 2020-04-20 00:08:47 -07:00
Chris Lu 2955b96ef1 refactor 2020-04-19 23:54:32 -07:00
Chris Lu df25ed3c5b logs 2020-04-19 23:37:50 -07:00
Chris Lu ce3cb25cfb working for in memory single log buffer 2020-04-19 23:37:50 -07:00
Chris Lu f373232227
Merge pull request #1280 from bukton123/filer_mongodb 
Add new filer support mongodb
 2020-04-19 20:35:19 -07:00
bukton da37455254 filer mongo add func create index unique directory, name 2020-04-20 08:13:57 +07:00
Chris Lu 4642ed565e fix tests 2020-04-19 10:42:58 -07:00
bukton 3defa718a1 add default config filer mongodb to scaffold.go 2020-04-19 17:52:38 +07:00
bukton d6d245dc89 filer mongodb add option poll size and fixed order by list meta 2020-04-19 17:51:32 +07:00
Chris Lu 71ffb98475 broker: add profiling 2020-04-19 03:03:40 -07:00
Chris Lu d278b4c228 basic pub and sub works! 2020-04-19 00:18:32 -07:00
bukton 082f079ead filer mongo change func update to insert entry 2020-04-19 11:08:10 +07:00
Chris Lu 788578d4ea add subscriber api 2020-04-18 16:05:29 -07:00
Chris Lu 767f14dfcd master, volume default to detected host 2020-04-18 15:20:20 -07:00
Chris Lu 076c8bd3bc filer master start up with default ip address instead of just localhost 2020-04-18 15:17:27 -07:00
Chris Lu 5d346d44bd Revert "move go client out" 
This reverts commit ef1274edc9.
 2020-04-18 12:44:55 -07:00
Chris Lu ef1274edc9 move go client out 2020-04-18 12:11:50 -07:00
Chris Lu 5af49c7aa5 adjust API 2020-04-18 11:45:57 -07:00
bukton 8c58b0d017 filer mongodb delete folder children 2020-04-19 00:31:49 +07:00
bukton 290c6b7f01 Merge remote-tracking branch 'origin/master' into filer_mongodb 
# Conflicts:
#	go.mod
#	go.sum
#	weed/server/filer_server.go
 2020-04-19 00:21:45 +07:00
bukton 6234ea441b filer mongodb delete 2020-04-19 00:20:00 +07:00
bukton 24a8192298 filer mongo list 2020-04-19 00:16:50 +07:00
bukton 3c70163798 filer mongodb insert entry and find entry 2020-04-18 23:48:38 +07:00
bukton cd9cccec55 add import mongodb to filer server 2020-04-18 22:30:13 +07:00
bukton fd4576651b add new package mongodb and install lib mongodb 2020-04-18 22:26:57 +07:00
Chris Lu 11f5a6d913 messaging can compile now 2020-04-18 01:12:01 -07:00
Chris Lu 5bea77010f refactor 2020-04-17 22:39:21 -07:00
Chris Lu 2e2537a9ea rename files 2020-04-17 22:35:44 -07:00
Chris Lu e0f5996560 fix "call of Unmarshal passes non-pointer as second argument" 2020-04-17 10:04:41 -07:00
Chris Lu 826f96b6f3 fix unreachable code 2020-04-17 10:04:28 -07:00
Chris Lu ffa69b24ae fix travis 2020-04-17 10:01:11 -07:00
Chris Lu 3f3dba5a68 broker: append message logs 2020-04-17 02:29:38 -07:00
Chris Lu 2a45897237 broker: read cipher value from filer 2020-04-17 02:29:00 -07:00
Chris Lu bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu 722b14337e add cipher option to meta data updates 2020-04-17 01:26:27 -07:00
Chris Lu 7d3672c60a simplify 2020-04-17 00:00:48 -07:00
Chris Lu 508f3490a0 update messaging proto 2020-04-16 03:29:57 -07:00
Chris Lu f5a748d33c refactoring 2020-04-16 02:55:09 -07:00
Chris Lu ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Reed aada5b9aa7 making volume stats size info readable. 2020-04-16 10:08:08 +08:00
Chris Lu 9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu 3f1d79512f only detect when not sure about the types 2020-04-14 11:02:05 -07:00
Chris Lu d2d1597850 volume: detect gzip effectiveness only when content type is empty 
revert part of 2286eda575
 2020-04-14 10:52:53 -07:00
Chris Lu 23715af402
Merge pull request #1273 from wyxloading/config_race 
Fix data race about config
 2020-04-13 22:24:51 -07:00
Chris Lu 2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu f282ed444b refactoring 2020-04-13 21:58:10 -07:00
wuyuxiang b556d3d035 Fix data race about config 2020-04-14 10:22:40 +08:00
Chris Lu d8f5985e5e add logs 2020-04-13 12:58:45 -07:00
Chris Lu 5c9e91393f 1.74 2020-04-13 10:25:48 -07:00
Chris Lu 7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu 8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu b062393f3f refactoring 2020-04-12 17:51:31 -07:00
Chris Lu 5361f999ed shell: only one shell is allowed to connect to the cluster 
fix https://github.com/chrislusf/seaweedfs/issues/1266
 2020-04-12 17:48:39 -07:00
Chris Lu eb16bb0287 go mod cleanup 2020-04-12 16:44:58 -07:00
Chris Lu d30483d642 re-enable system logs 2020-04-12 14:03:07 -07:00
Chris Lu 6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu 94e35cdb35 mount: fix 
fix bug found by git bisect, but I do not understand why it can cause error!
 2020-04-12 03:34:36 -07:00
Chris Lu 1c2e920fa2 fix compilation error 2020-04-12 03:11:19 -07:00
Chris Lu b5a713be68 filer store: add redis2 redis_cluster2 as default redis store 2020-04-12 02:50:41 -07:00
Chris Lu e613695af7 a little optimization 2020-04-12 01:13:57 -07:00
Chris Lu 2a1f396df5 avoid duplicated setting chunks into cache 2020-04-12 01:06:50 -07:00
Chris Lu b9b7da905e handle nil chunk cache 2020-04-12 01:00:12 -07:00
Chris Lu 211d87cf4c mount: option to disable caching 2020-04-12 00:52:54 -07:00
Chris Lu 1c65656fb4 s3: add option to fsync buckets 2020-04-11 23:37:10 -07:00
Chris Lu e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu 5987810e5e set default chunk size to 16 2020-04-11 21:22:52 -07:00
Chris Lu df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu c8ca234773 refactoring 2020-04-11 14:27:25 -07:00
Chris Lu df9d538044 rename function 2020-04-11 14:19:44 -07:00
Chris Lu d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu ed54682598 refactoring 2020-04-11 12:43:17 -07:00
Chris Lu e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu b7f0ba3800 refactoring 2020-04-11 12:37:41 -07:00
Chris Lu 417125457e add additional buffers for logs 2020-04-11 02:19:48 -07:00
Chris Lu 4b27afe176 fix sqs message reading 2020-04-10 15:27:47 -07:00
Chris Lu dce2702e1b fix the instruction 2020-04-10 14:50:10 -07:00
Chris Lu 13ab1fdaf0 mount: use file default permission 0666 2020-04-10 02:23:23 -07:00
Chris Lu 1101a42e5c mv : create new folder, move children, and delete old folder 2020-04-10 01:37:03 -07:00
Chris Lu bcf37346ef add timestamp inside lock 2020-04-10 01:35:59 -07:00
Chris Lu 9fa065f600 typo in logs 2020-04-09 23:43:09 -07:00
Chris Lu 8db2120bee add logs 2020-04-09 23:42:59 -07:00
Chris Lu 43c62a8a95 1.73 2020-04-09 21:44:30 -07:00
Chris Lu eb39df2704 avoid dead loop 2020-04-09 21:43:26 -07:00
Chris Lu 56ec8c4513 fix to avoid nil file an dirty pages 2020-04-09 21:43:05 -07:00