gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Hadoop Benchmark (markdown)

Chris Lu 2020-07-16 13:58:12 -07:00
parent a01417075f
commit d7a077bf8f
1 changed files with 20 additions and 106 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

126
Hadoop-Benchmark.md

@ -37,121 +37,35 @@ The TestDFSIO benchmark is used for measuring I/O (read/write) performance.
Start the TestDFSIO write tests: Start the TestDFSIO write tests:
``` ```
bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -write -nrFiles 16 -fileSize 16GB -resFile /tmp/TestDFSIOwrite.txt bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -write -nrFiles 8 -fileSize 1GB -resFile /tmp/TestDFSIOwrite.txt
... ...
20/07/14 18:27:34 INFO mapreduce.Job: map 100% reduce 100% 20/07/16 13:45:25 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
20/07/14 18:27:34 INFO mapreduce.Job: Job job_local485352420_0001 completed successfully 20/07/16 13:45:25 INFO fs.TestDFSIO: Date & time: Thu Jul 16 13:45:25 PDT 2020
20/07/14 18:27:34 INFO mapreduce.Job: Counters: 35 20/07/16 13:45:25 INFO fs.TestDFSIO: Number of files: 8
File System Counters 20/07/16 13:45:25 INFO fs.TestDFSIO: Total MBytes processed: 8192
FILE: Number of bytes read=27928633 20/07/16 13:45:25 INFO fs.TestDFSIO: Throughput mb/sec: 37.96
FILE: Number of bytes written=35966385 20/07/16 13:45:25 INFO fs.TestDFSIO: Average IO rate mb/sec: 38.03
FILE: Number of read operations=0 20/07/16 13:45:25 INFO fs.TestDFSIO: IO rate std deviation: 1.68
FILE: Number of large read operations=0 20/07/16 13:45:25 INFO fs.TestDFSIO: Test exec time sec: 218.24
FILE: Number of write operations=0 20/07/16 13:45:25 INFO fs.TestDFSIO:```
SEAWEEDFS: Number of bytes read=17111
SEAWEEDFS: Number of bytes written=2611340146618
SEAWEEDFS: Number of read operations=0
SEAWEEDFS: Number of large read operations=0
SEAWEEDFS: Number of write operations=0
Map-Reduce Framework
Map input records=16
Map output records=80
Map output bytes=1276
Map output materialized bytes=1532
Input split bytes=2054
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1532
Reduce input records=80
Reduce output records=5
Spilled Records=160
Shuffled Maps =16
Failed Shuffles=0
Merged Map outputs=16
GC time elapsed (ms)=151632
Total committed heap usage (bytes)=39777730560
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1798
File Output Format Counters
Bytes Written=84
20/07/14 18:27:34 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
20/07/14 18:27:34 INFO fs.TestDFSIO: Date & time: Tue Jul 14 18:27:34 PDT 2020
20/07/14 18:27:34 INFO fs.TestDFSIO: Number of files: 16
20/07/14 18:27:34 INFO fs.TestDFSIO: Total MBytes processed: 262144
20/07/14 18:27:34 INFO fs.TestDFSIO: Throughput mb/sec: 310.47
20/07/14 18:27:34 INFO fs.TestDFSIO: Average IO rate mb/sec: 315.63
20/07/14 18:27:34 INFO fs.TestDFSIO: IO rate std deviation: 43.43
20/07/14 18:27:34 INFO fs.TestDFSIO: Test exec time sec: 847.32
20/07/14 18:27:34 INFO fs.TestDFSIO:
```
## TestDFSIO read tests ## TestDFSIO read tests
Start the TestDFSIO read tests: Start the TestDFSIO read tests:
``` ```
bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -read -nrFiles 64 -fileSize 16GB -resFile /tmp/TestDFSIOwrite.txt bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -read -nrFiles 8 -fileSize 1GB -resFile /tmp/TestDFSIOwrite.txt
... ...
20/07/14 21:48:00 INFO mapreduce.Job: Counters: 35 20/07/16 13:46:57 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
File System Counters 20/07/16 13:46:57 INFO fs.TestDFSIO: Date & time: Thu Jul 16 13:46:57 PDT 2020
FILE: Number of bytes read=27928585 20/07/16 13:46:57 INFO fs.TestDFSIO: Number of files: 8
FILE: Number of bytes written=36004955 20/07/16 13:46:57 INFO fs.TestDFSIO: Total MBytes processed: 8192
FILE: Number of read operations=0 20/07/16 13:46:57 INFO fs.TestDFSIO: Throughput mb/sec: 406.01
FILE: Number of large read operations=0 20/07/16 13:46:57 INFO fs.TestDFSIO: Average IO rate mb/sec: 406.71
FILE: Number of write operations=0 20/07/16 13:46:57 INFO fs.TestDFSIO: IO rate std deviation: 16.61
SEAWEEDFS: Number of bytes read=2611340133079 20/07/16 13:46:57 INFO fs.TestDFSIO: Test exec time sec: 21.68
SEAWEEDFS: Number of bytes written=30649 20/07/16 13:46:57 INFO fs.TestDFSIO:
SEAWEEDFS: Number of read operations=0
SEAWEEDFS: Number of large read operations=0
SEAWEEDFS: Number of write operations=0
Map-Reduce Framework
Map input records=16
Map output records=80
Map output bytes=1252
Map output materialized bytes=1508
Input split bytes=2054
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1508
Reduce input records=80
Reduce output records=5
Spilled Records=160
Shuffled Maps =16
Failed Shuffles=0
Merged Map outputs=16
GC time elapsed (ms)=145687
Total committed heap usage (bytes)=38852886528
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1798
File Output Format Counters
Bytes Written=83
20/07/14 21:48:00 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
20/07/14 21:48:00 INFO fs.TestDFSIO: Date & time: Tue Jul 14 21:48:00 PDT 2020
20/07/14 21:48:00 INFO fs.TestDFSIO: Number of files: 16
20/07/14 21:48:00 INFO fs.TestDFSIO: Total MBytes processed: 262144
20/07/14 21:48:00 INFO fs.TestDFSIO: Throughput mb/sec: 22.14
20/07/14 21:48:00 INFO fs.TestDFSIO: Average IO rate mb/sec: 22.91
20/07/14 21:48:00 INFO fs.TestDFSIO: IO rate std deviation: 3.79
20/07/14 21:48:00 INFO fs.TestDFSIO: Test exec time sec: 11871.4
20/07/14 21:48:00 INFO fs.TestDFSIO:
``` ```