gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity

1.4.9 fix reading ORC file

Chris Lu 2020-10-10 11:47:16 -07:00
parent 550a445e07
commit dfe3f19dbb
3 changed files with 10 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
Hadoop-Benchmark.md

@ -26,7 +26,7 @@ Then get the seaweedfs hadoop client jar.
``` ```
cd share/hadoop/common/lib/ cd share/hadoop/common/lib/
wget https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop2-client/1.4.8/seaweedfs-hadoop2-client-1.4.8.jar wget https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop2-client/1.4.9/seaweedfs-hadoop2-client-1.4.9.jar
``` ```
# TestDFSIO Benchmark # TestDFSIO Benchmark

8
Hadoop-Compatible-File-System.md

@ -23,7 +23,7 @@ Maven
<dependency> <dependency>
<groupId>com.github.chrislusf</groupId> <groupId>com.github.chrislusf</groupId>
<artifactId>seaweedfs-hadoop3-client</artifactId> <artifactId>seaweedfs-hadoop3-client</artifactId>
<version>1.4.8</version> <version>1.4.9</version>
</dependency> </dependency>
or or
@ -31,16 +31,16 @@ or
<dependency> <dependency>
<groupId>com.github.chrislusf</groupId> <groupId>com.github.chrislusf</groupId>
<artifactId>seaweedfs-hadoop2-client</artifactId> <artifactId>seaweedfs-hadoop2-client</artifactId>
<version>1.4.8</version> <version>1.4.9</version>
</dependency> </dependency>
``` ```
Or you can download the latest version from MavenCentral Or you can download the latest version from MavenCentral
* https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-hadoop2-client * https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-hadoop2-client
* [seaweedfs-hadoop2-client-1.4.8.jar](https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop2-client/1.4.8/seaweedfs-hadoop2-client-1.4.8.jar) * [seaweedfs-hadoop2-client-1.4.9.jar](https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop2-client/1.4.9/seaweedfs-hadoop2-client-1.4.9.jar)
* https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-hadoop3-client * https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-hadoop3-client
* [seaweedfs-hadoop3-client-1.4.8.jar](https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop3-client/1.4.8/seaweedfs-hadoop3-client-1.4.8.jar) * [seaweedfs-hadoop3-client-1.4.9.jar](https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop3-client/1.4.9/seaweedfs-hadoop3-client-1.4.9.jar)
# Test SeaweedFS on Hadoop # Test SeaweedFS on Hadoop

10
run-Spark-on-SeaweedFS.md

@ -11,12 +11,12 @@ To make these files visible to Spark, set HADOOP_CONF_DIR in $SPARK_HOME/conf/sp
## installation not inheriting from Hadoop cluster configuration ## installation not inheriting from Hadoop cluster configuration
Copy the seaweedfs-hadoop2-client-1.4.8.jar to all executor machines. Copy the seaweedfs-hadoop2-client-1.4.9.jar to all executor machines.
Add the following to spark/conf/spark-defaults.conf on every node running Spark Add the following to spark/conf/spark-defaults.conf on every node running Spark
``` ```
spark.driver.extraClassPath=/path/to/seaweedfs-hadoop2-client-1.4.8.jar spark.driver.extraClassPath=/path/to/seaweedfs-hadoop2-client-1.4.9.jar
spark.executor.extraClassPath=/path/to/seaweedfs-hadoop2-client-1.4.8.jar spark.executor.extraClassPath=/path/to/seaweedfs-hadoop2-client-1.4.9.jar
``` ```
And modify the configuration at runtime: And modify the configuration at runtime:
@ -37,8 +37,8 @@ And modify the configuration at runtime:
1. change the spark-defaults.conf 1. change the spark-defaults.conf
``` ```
spark.driver.extraClassPath=/Users/chris/go/src/github.com/chrislusf/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-1.4.8.jar spark.driver.extraClassPath=/Users/chris/go/src/github.com/chrislusf/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-1.4.9.jar
spark.executor.extraClassPath=/Users/chris/go/src/github.com/chrislusf/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-1.4.8.jar spark.executor.extraClassPath=/Users/chris/go/src/github.com/chrislusf/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-1.4.9.jar
spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
``` ```