Hive On Spark环境搭建

Spark源码编译与环境搭建

Note that you must have a version of Spark which does not include the Hive jars;

Spark编译:

git clone https://github.com/apache/spark.git spark_src
cd spark_src
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
./make-distribution.sh --name "spark-without-hive" --tgz -Phadoop-2.4 -Dhadoop.version=2.5.-cdh5.3.1 -Pyarn -DskipTests package

Spark搭建：见Spark环境搭建章节

Hive源码编译与环境搭建

Hive编译：

git clone https://github.com/apache/hive.git hive_on_spark
git checkout spark
cd hive_on_spark
mvn clean install -Phadoop-,dist -DskipTests

编译完成后，hive安装包的位置： /packaging/target/apache-hive-1.2.0-SNAPSHOT-bin.tar.gz

注意pom.xml中spark.version要和spark的版本号对应

<spark.version>1.3.0</spark.version>

Hive安装：见Hive环境搭建章节

本案例中Spark和Hive的安装路径如下：

Spark安装目录：/home/spark/app/spark-1.3.0-bin-spark-without-hive

Hive安装目录：/home/spark/app/apache-hive-1.2.0-SNAPSHOT-bin

添加Spark的依赖到Hive的方法

方式一: Set the property 'spark.home' to point to the Spark installation:

hive> set spark.home=/home/spark/app/spark-1.3.-bin-spark-without-hive;

方式二: Define the SPARK_HOME environment variable before starting Hive CLI/HiveServer2:

export SPARK_HOME=/home/spark/app/spark-1.3.-bin-spark-without-hive

方式三: Set the spark-assembly jar on the Hive auxpath:

hive --auxpath /home/spark/app/spark-1.3.-bin-spark-without-hive/lib/spark-assembly-*.jar

方式四: Add the spark-assembly jar for the current user session:

hive> add jar /home/spark/app/spark-1.3.-bin-spark-without-hive/lib/spark-assembly-*.jar;

方式五: Link the spark-assembly jar to $HIVE_HOME/lib.

启动Hive过程中可能出现的错误：

[ERROR] Terminal initialization failed; falling back to unsupported
java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
        at jline.TerminalFactory.create(TerminalFactory.java:)
        at jline.TerminalFactory.get(TerminalFactory.java:)
        at jline.console.ConsoleReader.<init>(ConsoleReader.java:)
        at jline.console.ConsoleReader.<init>(ConsoleReader.java:)
        at jline.console.ConsoleReader.<init>(ConsoleReader.java:)
        at org.apache.hadoop.hive.cli.CliDriver.getConsoleReader(CliDriver.java:)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
        at java.lang.reflect.Method.invoke(Method.java:)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:)

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected

解决方法：export HADOOP_USER_CLASSPATH_FIRST=true

其他场景的错误解决方法参见：https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started

还有一个坑：需要设置spark.eventLog.dir参数，比如：

set spark.eventLog.dir= hdfs://hadoop000:/directory

否则查询会报错，这个坑深啊。。。。。。，否则一直报错:/tmp/spark-event类似的文件夹不存在。。。。

启动hive后设置执行引擎为spark：

hive> set hive.execution.engine=spark;

设置spark的运行模式：

hive> set spark.master=spark://hadoop000:7077

或者yarn：spark.master=yarn

Configure Spark-application configs for Hive

可以配置在spark-defaults.conf或者hive-site.xml

spark.master=<Spark Master URL>
spark.eventLog.enabled=true;
spark.executor.memory=512m;
spark.serializer=org.apache.spark.serializer.KryoSerializer;
spark.executor.memory=...  #Amount of memory to use per executor process.
spark.executor.cores=...  #Number of cores per executor.
spark.yarn.executor.memoryOverhead=...
spark.executor.instances=...  #The number of executors assigned to each application.
spark.driver.memory=...  #The amount of memory assigned to the Remote Spark Context (RSC). We recommend 4GB.
spark.yarn.driver.memoryOverhead=...  #We recommend  (MB).

参数配置详见文档：https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started

执行sql语句后可以在监控页面查看job/stages等信息

hive (default)> select city_id, count(*) c from page_views group by city_id order by c desc limit 5;
Query ID = spark_20150309173838_444cb5b1-b72e-4fc3-87db-4162e364cb1e
Total jobs =
Launching Job  out of
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
state = SENT
state = STARTED
state = STARTED
state = STARTED
state = STARTED
Query Hive on Spark job[] stages:

Status: Running (Hive on Spark job[])
Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
-- ::, Stage-0_0: (+)/      Stage-1_0: /  Stage-2_0: /
state = STARTED
state = STARTED
state = STARTED
-- ::, Stage-0_0: (+)/      Stage-1_0: /  Stage-2_0: /
state = STARTED
state = STARTED
-- ::, Stage-0_0: / Finished Stage-1_0: (+)/      Stage-2_0: /
state = SUCCEEDED
-- ::, Stage-0_0: / Finished Stage-1_0: / Finished Stage-2_0: / Finished
Status: Finished successfully in 10.07 seconds
OK
city_id c
-
-
-
-      

Time taken: 18.417 seconds, Fetched:  row(s)