gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3635 commits 28 branches 278 tags 135 MiB
Commit graph

2151 commits

Author SHA1 Message Date
Chris Lu 5568395edd Revert "Revert "Merge pull request #1299 from song-zhang/master"" 
This reverts commit afb6a1dbb4.
 2020-05-06 15:37:17 -07:00
zhangsong 3703c7e176 write request keep compatible with before 2020-05-06 21:35:32 +08:00
Chris Lu 1e3e4b3072 add broker connects to filer 2020-05-05 02:05:28 -07:00
Chris Lu afb6a1dbb4 Revert "Merge pull request #1299 from song-zhang/master" 
This reverts commit 9016fa19ba, reversing
changes made to 47234760f4.
 2020-05-04 20:34:26 -07:00
Chris Lu 9016fa19ba
Merge pull request #1299 from song-zhang/master 
use async write to persistent file to disk - part1
 2020-05-04 15:51:03 -07:00
zhangsong f9e8702bb4 use async write to persistent file to disk - part1 2020-05-04 17:39:44 +08:00
Chris Lu 47234760f4 log adjust format 2020-05-02 01:33:49 -07:00
Chris Lu 97a504b0bb fix possible directory deletion error 2020-05-02 01:11:53 -07:00
Chris Lu 78422c2f69
Merge pull request #1298 from song-zhang/master 
add fsync option for benchmark
 2020-04-30 20:16:42 -07:00
Chris Lu 871efa4fc1 refactoring 
some previous chunk etag was using md5, which should be wrong.
 2020-04-30 17:20:47 -07:00
zhangsong d819865fe7 add fsync option for benchmark 2020-04-30 23:31:08 +08:00
Chris Lu ea809a45ab add cipher key 2020-04-30 03:05:43 -07:00
Chris Lu f9b6178b8f log messages 2020-04-30 03:05:34 -07:00
Chris Lu 8c73410a51 subscribe from a timestamp 2020-04-30 02:19:51 -07:00
Chris Lu 4e16a90454 for debugging 2020-04-30 02:19:32 -07:00
Chris Lu a25a8d8822 s3: print out time in UTC format 
fix https://github.com/chrislusf/seaweedfs/issues/1297
 2020-04-29 21:35:24 -07:00
Chris Lu e93588ec78 FUSE mount: atomic local cache updates 2020-04-29 18:20:54 -07:00
Chris Lu 9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
Chris Lu 7c10602b49 read in case cross chunks 2020-04-29 02:42:58 -07:00
Chris Lu f9da859720 better error message 2020-04-29 02:42:06 -07:00
Chris Lu d57559d3f7 chunk rememober encryption or not 2020-04-29 02:41:47 -07:00
Chris Lu 5a0986dca0 reset on getting the buffer 2020-04-29 02:41:01 -07:00
Chris Lu 8d42a1263b meta subscription: update the last read time 2020-04-28 17:30:04 -07:00
Chris Lu 31583b5d6c master and volume server: avoid race condition 
The volume server may disconnect and reconnect to the same master.

The master's unregistration may happen after the reconnection. Thus the volume server will disappear.
 2020-04-28 17:29:10 -07:00
Chris Lu 98edec7c3e remove printouts 2020-04-28 16:45:48 -07:00
Chris Lu 8ed490164e refactoring 2020-04-28 02:05:44 -07:00
Chris Lu 66effaed9e fix compilation 2020-04-28 02:05:32 -07:00
Chris Lu eab4c9219c randomize benchmark content 
fix https://github.com/chrislusf/seaweedfs/issues/1294
 2020-04-28 01:50:56 -07:00
Chris Lu fb81f12686 filer: simplify image resize 2020-04-28 00:05:57 -07:00
Chris Lu 5c57297bd1 metadata log: read from any timestamp 2020-04-28 00:05:57 -07:00
wuyuxiang 6850d28d6b refacotr: move signal handling and pprof to grace package 2020-04-28 14:10:23 +08:00
Chris Lu 2a7957b4ca FUSE: subcribe meta events based on timestamp 2020-04-27 16:52:20 -07:00
Chris Lu 6190fd665d printout error 2020-04-27 12:41:31 -07:00
Chris Lu 725e47dcc3 filer: pass along user name and group name for HDFS 2020-04-27 05:48:48 -07:00
Chris Lu 508ec01ab0 tweaking 2020-04-26 16:11:06 -07:00
Chris Lu f4fcde05b6 1.77 2020-04-26 05:54:47 -07:00
Chris Lu b52b8ec685 Hadoop: fix entry not found for HCFS 
also fix cipher related changes.
 2020-04-26 05:21:54 -07:00
Chris Lu 0c2248f83a add lock/unlock to example admin scripts 2020-04-23 18:41:48 -07:00
Chris Lu 913aa0f14e mount: default to enable asyncMetaDataCaching 2020-04-23 14:14:28 -07:00
Chris Lu 662b5d0cf7 support semicolon seperated command lines 2020-04-23 14:01:46 -07:00
Chris Lu 73564e6a01 master: add cluster wide lock/unlock operation in weed shell 
fix https://github.com/chrislusf/seaweedfs/issues/1286
 2020-04-23 13:37:31 -07:00
Chris Lu bdc337a719 make lock/unlock optional 2020-04-23 03:50:05 -07:00
Chris Lu 57ec7f2ba9 fix builds 2020-04-23 03:43:45 -07:00
Chris Lu eab9701823 allow lock with an existing lock 2020-04-23 03:35:52 -07:00
Chris Lu 842e3301ee make second lock instant 2020-04-23 03:32:18 -07:00
Chris Lu 69f336e59f shell: add lock/unlock command 2020-04-23 03:11:07 -07:00
Chris Lu ff0a7c1d18 clean up 2020-04-23 03:10:47 -07:00
Chris Lu 30ee4f3291 add exclusive lock library on shell 2020-04-23 02:31:04 -07:00
Chris Lu 77873b832b add master side code for cluster wise exclusive lock 2020-04-23 01:55:44 -07:00
Chris Lu 369aa8a10a 1.76 2020-04-22 23:31:59 -07:00
Chris Lu 69eda558ef reduce the connections 
fix https://github.com/chrislusf/seaweedfs/issues/1277

this avoids sending error to wdclients, and clients retry the connections
 2020-04-22 20:43:34 -07:00
Chris Lu aebe39a803 avoid repeated grpc connection creation 
fix https://github.com/chrislusf/seaweedfs/issues/1277
 2020-04-22 18:26:24 -07:00
Chris Lu 5d0e1d8d74 also writes to local meta cache 
before waiting for subscribed meta events
 2020-04-22 15:40:47 -07:00
Chris Lu 72c5dba4d4 1.75 2020-04-21 22:03:07 -07:00
Chris Lu a207285af7 cache metadata on startup 2020-04-21 22:00:34 -07:00
Chris Lu 9f9826e95a refactor 2020-04-21 21:28:47 -07:00
Chris Lu e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu 4f02f7121d read from meta cache 
meta cache is not initialized
 2020-04-21 18:50:30 -07:00
Chris Lu b8e4238ad2 add placeholder for cached meta data reading 2020-04-21 18:02:08 -07:00
Chris Lu 27128c7875 adjust logging 2020-04-21 16:13:13 -07:00
Chris Lu bafa95045b volume: deletion checks all disk locations 
fix https://github.com/chrislusf/seaweedfs/issues/1283
 2020-04-21 14:49:58 -07:00
Chris Lu 57df14f76f report error if copying failed 2020-04-21 14:28:46 -07:00
Chris Lu cf54d70311 filer: add -ip.bind option 2020-04-21 14:21:06 -07:00
Chris Lu 5c348087dc messaging: able to pub sub multiple partitions 2020-04-21 00:59:55 -07:00
Chris Lu cb3985be70 go fmt 2020-04-20 17:48:06 -07:00
Chris Lu 5eb83dfbd3 add memory buffer size limit 2020-04-20 17:43:50 -07:00
Chris Lu 258fba8a0f continue for reading from sealed memory buffer 2020-04-20 17:28:18 -07:00
Chris Lu 4bf959edf0 message broker: read also from sealed memory buffer 2020-04-20 17:26:38 -07:00
Chris Lu bd43c62fbd lower the test criteria 2020-04-20 03:08:10 -07:00
Chris Lu c3f9fdf48f adjust test 2020-04-20 02:54:21 -07:00
Chris Lu 5e8bb86995 close the connection with better accouting 2020-04-20 02:53:50 -07:00
Chris Lu 978da73cdb revert this grpc related change 
maybe related to https://github.com/chrislusf/seaweedfs/issues/1277
 2020-04-20 02:53:22 -07:00
Chris Lu ebfab42a50 refactoring 2020-04-20 00:08:47 -07:00
Chris Lu 2955b96ef1 refactor 2020-04-19 23:54:32 -07:00
Chris Lu df25ed3c5b logs 2020-04-19 23:37:50 -07:00
Chris Lu ce3cb25cfb working for in memory single log buffer 2020-04-19 23:37:50 -07:00
Chris Lu f373232227
Merge pull request #1280 from bukton123/filer_mongodb 
Add new filer support mongodb
 2020-04-19 20:35:19 -07:00
bukton da37455254 filer mongo add func create index unique directory, name 2020-04-20 08:13:57 +07:00
Chris Lu 4642ed565e fix tests 2020-04-19 10:42:58 -07:00
bukton 3defa718a1 add default config filer mongodb to scaffold.go 2020-04-19 17:52:38 +07:00
bukton d6d245dc89 filer mongodb add option poll size and fixed order by list meta 2020-04-19 17:51:32 +07:00
Chris Lu 71ffb98475 broker: add profiling 2020-04-19 03:03:40 -07:00
Chris Lu d278b4c228 basic pub and sub works! 2020-04-19 00:18:32 -07:00
bukton 082f079ead filer mongo change func update to insert entry 2020-04-19 11:08:10 +07:00
Chris Lu 788578d4ea add subscriber api 2020-04-18 16:05:29 -07:00
Chris Lu 767f14dfcd master, volume default to detected host 2020-04-18 15:20:20 -07:00
Chris Lu 076c8bd3bc filer master start up with default ip address instead of just localhost 2020-04-18 15:17:27 -07:00
Chris Lu 5d346d44bd Revert "move go client out" 
This reverts commit ef1274edc9.
 2020-04-18 12:44:55 -07:00
Chris Lu ef1274edc9 move go client out 2020-04-18 12:11:50 -07:00
Chris Lu 5af49c7aa5 adjust API 2020-04-18 11:45:57 -07:00
bukton 8c58b0d017 filer mongodb delete folder children 2020-04-19 00:31:49 +07:00
bukton 290c6b7f01 Merge remote-tracking branch 'origin/master' into filer_mongodb 
# Conflicts:
#	go.mod
#	go.sum
#	weed/server/filer_server.go
 2020-04-19 00:21:45 +07:00
bukton 6234ea441b filer mongodb delete 2020-04-19 00:20:00 +07:00
bukton 24a8192298 filer mongo list 2020-04-19 00:16:50 +07:00
bukton 3c70163798 filer mongodb insert entry and find entry 2020-04-18 23:48:38 +07:00
bukton cd9cccec55 add import mongodb to filer server 2020-04-18 22:30:13 +07:00
bukton fd4576651b add new package mongodb and install lib mongodb 2020-04-18 22:26:57 +07:00
Chris Lu 11f5a6d913 messaging can compile now 2020-04-18 01:12:01 -07:00
Chris Lu 5bea77010f refactor 2020-04-17 22:39:21 -07:00
Chris Lu 2e2537a9ea rename files 2020-04-17 22:35:44 -07:00
Chris Lu e0f5996560 fix "call of Unmarshal passes non-pointer as second argument" 2020-04-17 10:04:41 -07:00
Chris Lu 826f96b6f3 fix unreachable code 2020-04-17 10:04:28 -07:00
Chris Lu ffa69b24ae fix travis 2020-04-17 10:01:11 -07:00
Chris Lu 3f3dba5a68 broker: append message logs 2020-04-17 02:29:38 -07:00
Chris Lu 2a45897237 broker: read cipher value from filer 2020-04-17 02:29:00 -07:00
Chris Lu bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu 722b14337e add cipher option to meta data updates 2020-04-17 01:26:27 -07:00
Chris Lu 7d3672c60a simplify 2020-04-17 00:00:48 -07:00
Chris Lu 508f3490a0 update messaging proto 2020-04-16 03:29:57 -07:00
Chris Lu f5a748d33c refactoring 2020-04-16 02:55:09 -07:00
Chris Lu ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Reed aada5b9aa7 making volume stats size info readable. 2020-04-16 10:08:08 +08:00
Chris Lu 9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu 3f1d79512f only detect when not sure about the types 2020-04-14 11:02:05 -07:00
Chris Lu d2d1597850 volume: detect gzip effectiveness only when content type is empty 
revert part of 2286eda575
 2020-04-14 10:52:53 -07:00
Chris Lu 23715af402
Merge pull request #1273 from wyxloading/config_race 
Fix data race about config
 2020-04-13 22:24:51 -07:00
Chris Lu 2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu f282ed444b refactoring 2020-04-13 21:58:10 -07:00
wuyuxiang b556d3d035 Fix data race about config 2020-04-14 10:22:40 +08:00
Chris Lu d8f5985e5e add logs 2020-04-13 12:58:45 -07:00
Chris Lu 5c9e91393f 1.74 2020-04-13 10:25:48 -07:00
Chris Lu 7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu 8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu b062393f3f refactoring 2020-04-12 17:51:31 -07:00
Chris Lu 5361f999ed shell: only one shell is allowed to connect to the cluster 
fix https://github.com/chrislusf/seaweedfs/issues/1266
 2020-04-12 17:48:39 -07:00
Chris Lu eb16bb0287 go mod cleanup 2020-04-12 16:44:58 -07:00
Chris Lu d30483d642 re-enable system logs 2020-04-12 14:03:07 -07:00
Chris Lu 6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu 94e35cdb35 mount: fix 
fix bug found by git bisect, but I do not understand why it can cause error!
 2020-04-12 03:34:36 -07:00
Chris Lu 1c2e920fa2 fix compilation error 2020-04-12 03:11:19 -07:00
Chris Lu b5a713be68 filer store: add redis2 redis_cluster2 as default redis store 2020-04-12 02:50:41 -07:00
Chris Lu e613695af7 a little optimization 2020-04-12 01:13:57 -07:00
Chris Lu 2a1f396df5 avoid duplicated setting chunks into cache 2020-04-12 01:06:50 -07:00
Chris Lu b9b7da905e handle nil chunk cache 2020-04-12 01:00:12 -07:00
Chris Lu 211d87cf4c mount: option to disable caching 2020-04-12 00:52:54 -07:00
Chris Lu 1c65656fb4 s3: add option to fsync buckets 2020-04-11 23:37:10 -07:00
Chris Lu e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu 5987810e5e set default chunk size to 16 2020-04-11 21:22:52 -07:00
Chris Lu df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu c8ca234773 refactoring 2020-04-11 14:27:25 -07:00
Chris Lu df9d538044 rename function 2020-04-11 14:19:44 -07:00
Chris Lu d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu ed54682598 refactoring 2020-04-11 12:43:17 -07:00
Chris Lu e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu b7f0ba3800 refactoring 2020-04-11 12:37:41 -07:00
Chris Lu 417125457e add additional buffers for logs 2020-04-11 02:19:48 -07:00
Chris Lu 4b27afe176 fix sqs message reading 2020-04-10 15:27:47 -07:00
Chris Lu dce2702e1b fix the instruction 2020-04-10 14:50:10 -07:00
Chris Lu 13ab1fdaf0 mount: use file default permission 0666 2020-04-10 02:23:23 -07:00