From 1e6f89ef0b11e962e49d640662a66086354d87a2 Mon Sep 17 00:00:00 2001
From: Chris Lu <chrislusf@users.noreply.github.com>
Date: Tue, 14 Jul 2020 11:01:29 -0700
Subject: [PATCH] Created Hadoop Benchmark (markdown)

---
 Hadoop-Benchmark.md | 48 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
 create mode 100644 Hadoop-Benchmark.md

diff --git a/Hadoop-Benchmark.md b/Hadoop-Benchmark.md
new file mode 100644
index 0000000..7a82555
--- /dev/null
+++ b/Hadoop-Benchmark.md
@@ -0,0 +1,48 @@
+## Setup Hadoop Benchmark
+
+Here are my steps. First, checkout hadoop 2.10.0 binary, untar, and cd in to the hadoop directory.
+```
+wget http://apache.mirrors.hoobly.com/hadoop/common/hadoop-2.10.0/hadoop-2.10.0.tar.gz
+tar xvf hadoop-2.10.0.tar.gz
+cd hadoop-2.10.0
+```
+
+Modify the file `./etc/hadoop/core-site.xml`
+
+```
+<configuration>
+    <property>
+        <name>fs.seaweedfs.impl</name>
+        <value>seaweed.hdfs.SeaweedFileSystem</value>
+    </property>
+    <property>
+        <name>fs.defaultFS</name>
+        <value>seaweedfs://localhost:8888</value>
+    </property>
+</configuration>
+```
+
+Then get the seaweedfs hadoop client jar.
+
+```
+cd share/hadoop/common/lib/
+wget https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop2-client/1.3.2/seaweedfs-hadoop2-client-1.3.2.jar
+```
+
+Start the TestDFSIO tests:
+
+```
+bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -write -nrFiles 64 -fileSize 16GB -resFile /tmp/TestDFSIOwrite.txt
+
+...
+
+20/07/14 10:41:02 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
+20/07/14 10:41:02 INFO fs.TestDFSIO:             Date & time: Tue Jul 14 10:41:02 PDT 2020
+20/07/14 10:41:02 INFO fs.TestDFSIO:         Number of files: 64
+20/07/14 10:41:02 INFO fs.TestDFSIO:  Total MBytes processed: 1048576
+20/07/14 10:41:02 INFO fs.TestDFSIO:       Throughput mb/sec: 381.28
+20/07/14 10:41:02 INFO fs.TestDFSIO:  Average IO rate mb/sec: 383.42
+20/07/14 10:41:02 INFO fs.TestDFSIO:   IO rate std deviation: 28.81
+20/07/14 10:41:02 INFO fs.TestDFSIO:      Test exec time sec: 2756.42
+20/07/14 10:41:02 INFO fs.TestDFSIO:
+```
\ No newline at end of file