seaweedfs/note/memory_usage.txt

64 32G volumes consumes 10G memory
Each volume has 25M index, so each cost 160MB memory



Things happened when I use lots of threads ( almost 120 ) keeping read file from SeaweedFS.
But I'm not so familiar with linux so I can't tell you exactly what happened.
Next I'll show you things I know , if you need more info , contact me

My SeaweedFS version is about 0.12

1. top

top - 12:07:37 up 1 day,  3:17,  2 users,  load average: 0.00, 0.00, 0.00
Tasks: 152 total,   1 running, 151 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.8%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16269880k total, 16192364k used,    77516k free,    58172k buffers
Swap:  2064376k total,    12324k used,  2052052k free,  2827520k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1499 root      20   0 11.6g  10g 1424 S  0.0 65.6   7:32.53 weedvolume
 1498 root      20   0 3204m 2.1g 1428 S  0.0 13.5   4:36.59 weedvolume
 1737 root      20   0 98868 4932 2920 S  0.0  0.0   0:00.56 sshd
 1497 root      20   0  151m 4404 1152 S  0.0  0.0   1:21.40 weedmaster
 1335 root      20   0 97816 3044 2896 S  0.0  0.0   0:00.76 sshd

After system became steady ,  weedvolume used 65.6% memory .

2. free -m

             total       used       free     shared    buffers     cached
Mem:         15888      15809         79          0         56       2758
-/+ buffers/cache:      12994       2894
Swap:         2015         12       2003

3. startup cmd

screen -d -m /opt/weed/weedmaster -mdir /data/weeddata/ > /data/logs/weed/master.log &
screen -d -m /opt/weed/weedvolume -volumes=0-64 -dir /data/weeddata/ -pulseSeconds=20 -publicUrl="x.y.z:9334" -port 9334 > /data/logs/weed/s01.log &
screen -d -m /opt/weed/weedvolume -volumes=65-107 -dir /data/weeddata/ -pulseSeconds=20 -publicUrl="x.y.z:9335" -port 9335 > /data/logs/weed/s02.log &

4. du -sh .

32G     0.dat
26M     0.idx
8.2G    100.dat
6.8M    100.idx
8.2G    101.dat
6.9M    101.idx
8.2G    102.dat
6.8M    102.idx
8.2G    103.dat
6.8M    103.idx
8.2G    104.dat
6.8M    104.idx
8.2G    105.dat
6.9M    105.idx
8.2G    106.dat
6.9M    106.idx
8.2G    107.dat
6.9M    107.idx
32G     10.dat
25M     10.idx
32G     11.dat
25M     11.idx
32G     12.dat
25M     12.idx
32G     13.dat
25M     13.idx
32G     14.dat
25M     14.idx
32G     15.dat
25M     15.idx
32G     16.dat
25M     16.idx
32G     17.dat
25M     17.idx
32G     18.dat
25M     18.idx
32G     19.dat
25M     19.idx
32G     1.dat
26M     1.idx
32G     20.dat
25M     20.idx
32G     21.dat
25M     21.idx
32G     22.dat
25M     22.idx
32G     23.dat
25M     23.idx
32G     24.dat
25M     24.idx
32G     25.dat
25M     25.idx
32G     26.dat
25M     26.idx
32G     27.dat
25M     27.idx
32G     28.dat
25M     28.idx
32G     29.dat
25M     29.idx
32G     2.dat
26M     2.idx
32G     30.dat
25M     30.idx
32G     31.dat
25M     31.idx
32G     32.dat
25M     32.idx
32G     33.dat
25M     33.idx
32G     34.dat
25M     34.idx
32G     35.dat
25M     35.idx
32G     36.dat
25M     36.idx
32G     37.dat
25M     37.idx
32G     38.dat
25M     38.idx
32G     39.dat
25M     39.idx
32G     3.dat
26M     3.idx
32G     40.dat
25M     40.idx
32G     41.dat
25M     41.idx
32G     42.dat
25M     42.idx
32G     43.dat
25M     43.idx
32G     44.dat
25M     44.idx
32G     45.dat
25M     45.idx
32G     46.dat
25M     46.idx
32G     47.dat
25M     47.idx
32G     48.dat
25M     48.idx
32G     49.dat
25M     49.idx
32G     4.dat
26M     4.idx
32G     50.dat
25M     50.idx
32G     51.dat
25M     51.idx
32G     52.dat
25M     52.idx
32G     53.dat
25M     53.idx
32G     54.dat
25M     54.idx
32G     55.dat
25M     55.idx
32G     56.dat
25M     56.idx
32G     57.dat
25M     57.idx
32G     58.dat
25M     58.idx
32G     59.dat
25M     59.idx
32G     5.dat
26M     5.idx
32G     60.dat
25M     60.idx
32G     61.dat
25M     61.idx
32G     62.dat
25M     62.idx
32G     63.dat
25M     63.idx
32G     64.dat
25M     64.idx
8.2G    65.dat
6.9M    65.idx
8.2G    66.dat
6.9M    66.idx
8.2G    67.dat
6.9M    67.idx
8.2G    68.dat
6.8M    68.idx
8.2G    69.dat
6.9M    69.idx
32G     6.dat
25M     6.idx
8.2G    70.dat
6.8M    70.idx
8.2G    71.dat
6.9M    71.idx
8.2G    72.dat
6.9M    72.idx
8.2G    73.dat
6.9M    73.idx
8.2G    74.dat
6.9M    74.idx
8.2G    75.dat
6.9M    75.idx
8.1G    76.dat
6.8M    76.idx
8.2G    77.dat
6.8M    77.idx
8.2G    78.dat
6.8M    78.idx
8.1G    79.dat
6.8M    79.idx
32G     7.dat
25M     7.idx
8.2G    80.dat
6.8M    80.idx
8.2G    81.dat
6.9M    81.idx
8.2G    82.dat
6.9M    82.idx
8.2G    83.dat
6.9M    83.idx
8.2G    84.dat
6.9M    84.idx
8.2G    85.dat
6.8M    85.idx
8.2G    86.dat
6.9M    86.idx
8.2G    87.dat
6.9M    87.idx
8.2G    88.dat
6.9M    88.idx
8.2G    89.dat
6.8M    89.idx
32G     8.dat
25M     8.idx
8.2G    90.dat
6.9M    90.idx
8.1G    91.dat
6.8M    91.idx
8.1G    92.dat
6.8M    92.idx
8.1G    93.dat
6.8M    93.idx
8.2G    94.dat
6.9M    94.idx
8.2G    95.dat
6.9M    95.idx
8.2G    96.dat
6.9M    96.idx
8.2G    97.dat
6.9M    97.idx
8.2G    98.dat
6.9M    98.idx
8.2G    99.dat
6.9M    99.idx
32G     9.dat
25M     9.idx
4.0K    directory.seq

You can see the volume 1-64 is now full.

5. more log

see logs.zip

In messages you can see these lines: (Line 51095)

Sep 26 06:14:31 wedb-01 kernel: auditd: page allocation failure. order:0, mode:0x20
Sep 26 06:14:31 wedb-01 kernel: Pid: 1009, comm: auditd Not tainted 2.6.32-220.el6.x86_64 #1
Sep 26 06:14:31 wedb-01 kernel: Call Trace:

After those lines , the system deny any new network connect request

6. /dir/status

{"Machines":[{"Server":{"Url":"127.0.0.1:9335","PublicUrl":"x.y.z:9335"},"Volumes":[{"Id":106,"Size":8728909632},{"Id":66,"Size":8729852744},{"Id":90,"Size":8747834896},{"Id":103,"Size":8718106024},{"Id":87,"Size":8732133512},{"Id":96,"Size":8737251904},{"Id":80,"Size":8704130712},{"Id":77,"Size":8717989496},{"Id":70,"Size":8731474744},{"Id":94,"Size":8758656144},{"Id":107,"Size":8729599232},{"Id":67,"Size":8736848088},{"Id":91,"Size":8665847760},{"Id":100,"Size":8703272552},{"Id":84,"Size":8745121528},{"Id":97,"Size":8713031744},{"Id":81,"Size":8726088872},{"Id":74,"Size":8738588152},{"Id":71,"Size":8729349920},{"Id":95,"Size":8741526896},{"Id":104,"Size":8699374736},{"Id":88,"Size":8740362880},{"Id":101,"Size":8711832992},{"Id":85,"Size":8723479552},{"Id":78,"Size":8700345400},{"Id":75,"Size":8727796912},{"Id":68,"Size":8698607440},{"Id":92,"Size":8682683056},{"Id":105,"Size":8741226152},{"Id":65,"Size":8725365752},{"Id":89,"Size":8703062600},{"Id":98,"Size":8742331560},{"Id":82,"Size":8762554952},{"Id":79,"Size":8696300376},{"Id":72,"Size":8708217304},{"Id":69,"Size":8740268144},{"Id":93,"Size":8685060320},{"Id":102,"Size":8708695352},{"Id":86,"Size":8783247776},{"Id":99,"Size":8753463608},{"Id":83,"Size":8725963952},{"Id":76,"Size":8694693536},{"Id":73,"Size":8733560832}]},{"Server":{"Url":"127.0.0.1:9334","PublicUrl":"x.y.z:9334"},"Volumes":[{"Id":34,"Size":33415706800},{"Id":58,"Size":33569224784},{"Id":18,"Size":33474649968},{"Id":55,"Size":33542422680},{"Id":15,"Size":33517247576},{"Id":48,"Size":33574860328},{"Id":8,"Size":33511257144},{"Id":45,"Size":33463948408},{"Id":5,"Size":34317702920},{"Id":29,"Size":33465695776},{"Id":38,"Size":33553119624},{"Id":62,"Size":33448316736},{"Id":22,"Size":33566586296},{"Id":35,"Size":33493733728},{"Id":59,"Size":33498554904},{"Id":19,"Size":33493313784},{"Id":52,"Size":33552978448},{"Id":12,"Size":33505183752},{"Id":49,"Size":33603029896},{"Id":9,"Size":33515778064},{"Id":42,"Size":33500402248},{"Id":2,"Size":34223232992},{"Id":26,"Size":33526519600},{"Id":39,"Size":33580414336},{"Id":63,"Size":33476332456},{"Id":23,"Size":33543872592},{"Id":32,"Size":33515290168},{"Id":56,"Size":33499171184},{"Id":16,"Size":33556591168},{"Id":64,"Size":33495148616},{"Id":53,"Size":33467738560},{"Id":13,"Size":33596873960},{"Id":46,"Size":33508120448},{"Id":6,"Size":33417470256},{"Id":30,"Size":33532933992},{"Id":43,"Size":33591802008},{"Id":3,"Size":34270682080},{"Id":27,"Size":33525736944},{"Id":36,"Size":33443597824},{"Id":60,"Size":33427931336},{"Id":20,"Size":33499083096},{"Id":33,"Size":33531396280},{"Id":57,"Size":33578015104},{"Id":17,"Size":33510525480},{"Id":50,"Size":33503123704},{"Id":10,"Size":33502391608},{"Id":47,"Size":33521868568},{"Id":7,"Size":33497101664},{"Id":31,"Size":33426905232},{"Id":40,"Size":33472978696},{"Id":0,"Size":34337344304},{"Id":24,"Size":33550157192},{"Id":37,"Size":33477162720},{"Id":61,"Size":33537175080},{"Id":21,"Size":33517192456},{"Id":54,"Size":33480720288},{"Id":14,"Size":33513192896},{"Id":51,"Size":33531336080},{"Id":11,"Size":33562385088},{"Id":44,"Size":33554479104},{"Id":4,"Size":34333127520},{"Id":28,"Size":33510503000},{"Id":41,"Size":33574922928},{"Id":1,"Size":34307181368},{"Id":25,"Size":33542834568}]}],"Writers":[106,66,90,103,87,96,80,77,70,94,107,67,91,100,84,97,81,74,71,95,104,88,101,85,78,75,68,92,105,65,89,98,82,79,72,69,93,102,86,99,83,76,73,34,58,18,55,15,48,8,45,5,29,38,62,22,35,59,19,52,12,49,9,42,2,26,39,63,23,32,56,16,64,53,13,46,6,30,43,3,27,36,60,20,33,57,17,50,10,47,7,31,40,0,24,37,61,21,54,14,51,11,44,4,28,41,1,25],"FileIdSequence":110250000}