gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Hadoop Benchmark (markdown)

Chris Lu 2020-07-17 16:02:12 -07:00
parent 2dc3b650e9
commit 065ae4d31b
1 changed files with 20 additions and 18 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

38
Hadoop-Benchmark.md

@ -33,6 +33,8 @@ wget https://oss.sonatype.org/service/local/repositories/releases/content/com/gi
The TestDFSIO benchmark is used for measuring I/O (read/write) performance. The TestDFSIO benchmark is used for measuring I/O (read/write) performance.
**However, the generated data to write is all zero. SeaweedFS automatically compress this kind of data. So this is not scientific for now.**
## TestDFSIO write tests ## TestDFSIO write tests
Start the TestDFSIO write tests: Start the TestDFSIO write tests:
@ -40,15 +42,15 @@ Start the TestDFSIO write tests:
bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -write -nrFiles 8 -fileSize 1GB -resFile /tmp/TestDFSIOwrite.txt bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -write -nrFiles 8 -fileSize 1GB -resFile /tmp/TestDFSIOwrite.txt
... ...
20/07/16 13:45:25 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write 20/07/17 15:57:08 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
20/07/16 13:45:25 INFO fs.TestDFSIO: Date & time: Thu Jul 16 13:45:25 PDT 2020 20/07/17 15:57:08 INFO fs.TestDFSIO: Date & time: Fri Jul 17 15:57:08 PDT 2020
20/07/16 13:45:25 INFO fs.TestDFSIO: Number of files: 8 20/07/17 15:57:08 INFO fs.TestDFSIO: Number of files: 8
20/07/16 13:45:25 INFO fs.TestDFSIO: Total MBytes processed: 8192 20/07/17 15:57:08 INFO fs.TestDFSIO: Total MBytes processed: 8192
20/07/16 13:45:25 INFO fs.TestDFSIO: Throughput mb/sec: 37.96 20/07/17 15:57:08 INFO fs.TestDFSIO: Throughput mb/sec: 379.29
20/07/16 13:45:25 INFO fs.TestDFSIO: Average IO rate mb/sec: 38.03 20/07/17 15:57:08 INFO fs.TestDFSIO: Average IO rate mb/sec: 389.95
20/07/16 13:45:25 INFO fs.TestDFSIO: IO rate std deviation: 1.68 20/07/17 15:57:08 INFO fs.TestDFSIO: IO rate std deviation: 74.91
20/07/16 13:45:25 INFO fs.TestDFSIO: Test exec time sec: 218.24 20/07/17 15:57:08 INFO fs.TestDFSIO: Test exec time sec: 24.05
20/07/16 13:45:25 INFO fs.TestDFSIO: 20/07/17 15:57:08 INFO fs.TestDFSIO:
``` ```
## TestDFSIO read tests ## TestDFSIO read tests
@ -60,13 +62,13 @@ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0
... ...
20/07/16 13:46:57 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read 20/07/17 15:59:38 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
20/07/16 13:46:57 INFO fs.TestDFSIO: Date & time: Thu Jul 16 13:46:57 PDT 2020 20/07/17 15:59:38 INFO fs.TestDFSIO: Date & time: Fri Jul 17 15:59:38 PDT 2020
20/07/16 13:46:57 INFO fs.TestDFSIO: Number of files: 8 20/07/17 15:59:38 INFO fs.TestDFSIO: Number of files: 8
20/07/16 13:46:57 INFO fs.TestDFSIO: Total MBytes processed: 8192 20/07/17 15:59:38 INFO fs.TestDFSIO: Total MBytes processed: 8192
20/07/16 13:46:57 INFO fs.TestDFSIO: Throughput mb/sec: 406.01 20/07/17 15:59:38 INFO fs.TestDFSIO: Throughput mb/sec: 393.26
20/07/16 13:46:57 INFO fs.TestDFSIO: Average IO rate mb/sec: 406.71 20/07/17 15:59:38 INFO fs.TestDFSIO: Average IO rate mb/sec: 393.72
20/07/16 13:46:57 INFO fs.TestDFSIO: IO rate std deviation: 16.61 20/07/17 15:59:38 INFO fs.TestDFSIO: IO rate std deviation: 13.33
20/07/16 13:46:57 INFO fs.TestDFSIO: Test exec time sec: 21.68 20/07/17 15:59:38 INFO fs.TestDFSIO: Test exec time sec: 22.76
20/07/16 13:46:57 INFO fs.TestDFSIO: 20/07/17 15:59:38 INFO fs.TestDFSIO:
``` ```