gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated Independent Benchmarks (markdown)

Chris Lu 2020-11-21 13:57:01 -08:00
parent 8035c7ca05
commit 941e31e127
1 changed files with 1 additions and 69 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

70
Independent-Benchmarks.md

@ -1,70 +1,2 @@
Here are a few independent benchmarks collected.
## Sugon (中科曙光)
To test the performance of HDFS and HCFS, we did a comparision by running 4 common spark oprators, such as `count`, `group by`, `join` and `write`, for `group by` and `join` , there is a `count` followed to act.
The basic configuration information of cluster is as follows:
- HDFS：
+ Node number: 25
+ Total disks: 36disk * 25node = 900disk
+ Disk capacity: 3.7T SATA
+ Total disk capacity: 3.19PB
+ Replication: 5
- HCFS:
+ Node number: 6（3+3 rack）
+ Disk capacity: 3.7T SATA
+ Cluster max volume: 21500
+ Total disk capacity: 799TB
+ Replication policy: 010
Here are the details and results of our test. At the beginning of the test, we put our data to both HDFS and HCFS. The amount of the data is 100 million records, and stored in 200 parquet files. The size of each parquet file is about 89 MB. We ran spark on yarn with 20 executors. In spark, we got two DataFrames by reading parquet from HDFS and HCFS separately, then executed `count`, `group by` and `join` by 100 times , and `write` by 10 times, on each DataFrame.
As for `count`, HCFS's advantage is obvious. The average time of the DataFrame from HDFS is 4.05 seconds, while HCFS is only 0.659. Following is the result:
| Summary | HDFS | HCFS |
| ------- | ----- | ----- |
| Count | 100 | 100 |
| Mean | 4.050 | 0.659 |
| Stddev | 0.264 | 0.941 |
| Min | 3.678 | 0.392 |
| Max | 5.692 | 9.688 |
As for `write`, we wrote the DataFrame from HDFS to HDFS, and wrote the DataFrame from HCFS to HCFS. Following is the result:
| Summary | HDFS | HCFS |
| ------- | ------- | ------- |
| Count | 10 | 10 |
| Mean | 234.279 | 232.078 |
| Stddev | 26.823 | 12.652 |
| Min | 216.931 | 214.349 |
| Max | 307.330 | 252.375 |
As for `group by`, following is the result:
| Summary | HDFS | HCFS |
| ------- | ------ | ------ |
| Count | 100 | 100 |
| Mean | 14.121 | 12.515 |
| Stddev | 1.972 | 1.255 |
| Min | 12.879 | 11.322 |
| Max | 32.296 | 22.573 |
As for `join`, every DataFrame join with itself on one column. Following is the result:
| Summary | HDFS | HCFS |
| ------- | ------ | ------ |
| Count | 100 | 100 |
| Mean | 25.684 | 23.897 |
| Stddev | 0.934 | 1.381 |
| Min | 24.006 | 22.275 |
| Max | 30.991 | 30.279 |
Please submit independent benchmarks here. They will be re-organized to the right wiki place later.