From 941e31e1274f9f5ce9803a3f3817dc99f96ccc60 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 21 Nov 2020 13:57:01 -0800 Subject: [PATCH] Updated Independent Benchmarks (markdown) --- Independent-Benchmarks.md | 70 +-------------------------------------- 1 file changed, 1 insertion(+), 69 deletions(-) diff --git a/Independent-Benchmarks.md b/Independent-Benchmarks.md index 3b8059b..e36e1d6 100644 --- a/Independent-Benchmarks.md +++ b/Independent-Benchmarks.md @@ -1,70 +1,2 @@ -Here are a few independent benchmarks collected. - -## Sugon (中科曙光) -To test the performance of HDFS and HCFS, we did a comparision by running 4 common spark oprators, such as `count`, `group by`, `join` and `write`, for `group by` and `join` , there is a `count` followed to act. - -The basic configuration information of cluster is as follows: - -- HDFS： - + Node number: 25 - + Total disks: 36disk * 25node = 900disk - + Disk capacity: 3.7T SATA - + Total disk capacity: 3.19PB - + Replication: 5 - -- HCFS: - - + Node number: 6（3+3 rack） - + Disk capacity: 3.7T SATA - + Cluster max volume: 21500 - + Total disk capacity: 799TB - + Replication policy: 010 - -Here are the details and results of our test. At the beginning of the test, we put our data to both HDFS and HCFS. The amount of the data is 100 million records, and stored in 200 parquet files. The size of each parquet file is about 89 MB. We ran spark on yarn with 20 executors. In spark, we got two DataFrames by reading parquet from HDFS and HCFS separately, then executed `count`, `group by` and `join` by 100 times , and `write` by 10 times, on each DataFrame. - -As for `count`, HCFS's advantage is obvious. The average time of the DataFrame from HDFS is 4.05 seconds, while HCFS is only 0.659. Following is the result: - -| Summary | HDFS | HCFS | -| ------- | ----- | ----- | -| Count | 100 | 100 | -| Mean | 4.050 | 0.659 | -| Stddev | 0.264 | 0.941 | -| Min | 3.678 | 0.392 | -| Max | 5.692 | 9.688 | - - - -As for `write`, we wrote the DataFrame from HDFS to HDFS, and wrote the DataFrame from HCFS to HCFS. Following is the result: - -| Summary | HDFS | HCFS | -| ------- | ------- | ------- | -| Count | 10 | 10 | -| Mean | 234.279 | 232.078 | -| Stddev | 26.823 | 12.652 | -| Min | 216.931 | 214.349 | -| Max | 307.330 | 252.375 | - - - -As for `group by`, following is the result: - -| Summary | HDFS | HCFS | -| ------- | ------ | ------ | -| Count | 100 | 100 | -| Mean | 14.121 | 12.515 | -| Stddev | 1.972 | 1.255 | -| Min | 12.879 | 11.322 | -| Max | 32.296 | 22.573 | - - - -As for `join`, every DataFrame join with itself on one column. Following is the result: - -| Summary | HDFS | HCFS | -| ------- | ------ | ------ | -| Count | 100 | 100 | -| Mean | 25.684 | 23.897 | -| Stddev | 0.934 | 1.381 | -| Min | 24.006 | 22.275 | -| Max | 30.991 | 30.279 | +Please submit independent benchmarks here. They will be re-organized to the right wiki place later.