gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

Updated run Spark on SeaweedFS (markdown)

Chris Lu 2020-10-18 14:19:18 -07:00
parent b4a6e86656
commit 4579555f4e
1 changed files with 4 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
run-Spark-on-SeaweedFS.md

@ -60,8 +60,9 @@ file:///usr/local/spark/examples/jars/spark-examples_2.12-3.0.0.jar
# A Full Example # A Full Example
Here is my local example switching everything to SeaweedFS. Here is my local example switching everything to SeaweedFS. In the `/usr/local/spark/conf/spark-defaults.conf` file,
1. this is my local spark-defaults.conf
1. this is my local `/usr/local/spark/conf/spark-defaults.conf`
``` ```
spark.eventLog.enabled=true spark.eventLog.enabled=true
spark.sql.hive.convertMetastoreOrc=true spark.sql.hive.convertMetastoreOrc=true
@ -87,7 +88,7 @@ spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888
``` ```
2. create the spark history folder 2. create the spark history folder
``` ```
$ curl -X POST http://192.168.2.3:8888/spark2-history/ $ curl -X POST http://192.168.2.4:8888/spark2-history/
``` ```
3. Run a spark shell 3. Run a spark shell
``` ```