Dream------hive on spark

一、Hive on Spark是Hive跑在Spark上，用的是Spark执行引擎，而不是MapReduce，和Hive on Tez的道理一样。

并且用的是$HIVE_HOME/bin/hive，liunx命令运行客户端

 

这个时候需要下载spark的源码并且要重新编译，一个不支持hive的版本。

 

步骤：
1、下载spark1.4.1的源码
https://github.com/apache/spark/tree/v1.4.1

并解压

2、使用编译命令：
./make-distribution.sh --name "hadoop-2.6.0" --tgz "-Dyarn.version=2.6.0 -Dhadoop.version=2.6.0 -Pyarn"

 

3、配置spark-env.sh文件

 

export JAVA_HOME=/usr/local/soft/jdk1.7.0

     #export SPARK_MASTER_IP=hadoop-spark01

     export SPARK_MASTER_WEBUI_PORT=8099

     #export SPARK_MASTER_IP=localhost

     export SPARK_MASTER_PORT=7077

     export SPARK_WORKER_CORES=2

     export SPARK_WORKER_INSTANCES=2

     export SPARK_WORKER_MEMORY=1g

     #export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=FILESYSTEM -Dspark.deploy.recoveryDirectory=/nfs/spark/recovery"

     export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop-spark01:2181,hadoop-spark02:2181,hadoop-spark03:2181 -Dspark.deploy.zookeeper.dir=/home/data/spark/zkdir" （这是spark的HA配置）    

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

     export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

     export HIVE_CONF_DIR=$HIVE_HOME/conf

     export SPARK_HOME=/usr/local/soft/spark-1.4.1-bin-hadoop-2.6.0

     export SPARK_CLASSPATH=/usr/local/soft/sparkclasspath/mysql-connector-java-5.1.38-bin.jar:/usr/local/soft/sparkclasspath/hiv

e-hbase-handler-1.2.1.jar:/usr/local/soft/sparkclasspath/hbase-common-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-client-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-protocol-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-server-1.1.2.jar:/usr/local/soft/sparkclasspath/protobuf-java-2.5.0.jar:/usr/local/soft/sparkclasspath/htrace-core-3.1.0-incubating.jar:/usr/local/soft/sparkclasspath/guava-12.0.1.jar:/usr/local/soft/sparkclasspath/hive-exec-1.2.1.jar     

#export SPARK_LIBRARY_PATH=/usr/local/soft/hbase-1.1.2/lib

     export SPARK_JAR=/usr/local/soft/spark-1.4.1-bin-hadoop-2.6.0/lib/spark-assembly-1.4.1-hadoop2.6.0.jar

     export PATH=$SPARK_HOME/bin:$PATH

 

4、将spark-assembly-1.4.1-hadoop2.6.0.jar包，拷贝到$HIVE_HOME/lib目录下

 

5、修改hive-site.xml

<property>

<name>hive.metastore.uris</name>

<value>thrift://hadoop-spark01:9083</value>

<description>Thrift URI forthe remote metastore. Used by metastore client to connect to remote metastore.</description>

</property>

 

<property>

<name>hive.server2.thrift.min.worker.threads</name>

<value>5</value>

<description>Minimum number of Thrift worker threads</description>

</property>

 

<property>

<name>hive.server2.thrift.max.worker.threads</name>

<value>500</value>

<description>Maximum number of Thrift worker threads</description>

</property>

 

<property>

<name>hive.server2.thrift.port</name>

<value>10000</value>

<description>Port number of HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT</description>

</property>

 

<property>

<name>hive.server2.thrift.bind.host</name>

<value>hadoop-spark01</value>

<description>Bind host on which to run the HiveServer2 Thrift interface.Can be overridden by setting$HIVE_SERVER2_THRIFT_BIND_HOST</description>

</property>

 <property>

   <name>spark.serializer</name>

<value>org.apache.spark.serializer.KryoSerializer</value>

</property>

<property>

<name>spark.eventLog.enabled</name>

 <value>true</value>

 </property>

 <property>

  <name>spark.eventLog.dir</name>

  <value>hdfs://founder/sparklog/logs</value>

 </property>

 <property>

<name>spark.master</name>

  <value>spark://hadoop-spark01:7077,hadoop-spark02:7077</value>

 </property>

还有这些参数也要配置上

1、hive.exec.local.scratchdir

/opt/hive-1.2/tmp

2、hive.downloaded.resources.dir

/opt/hive-1.2/resources

 

 

 

配置Mysql数据库

1、javax.jdo.option.ConnectionPassword

123456

2、javax.jdo.option.ConnectionURL

jdbc:mysql://hadoop-spark01:3306/hive_db

 

3、javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

 

4、javax.jdo.option.ConnectionUserName

root

 

 

6、启动

启动spark

./start-all.sh

在backup-master节点上

./start-master

 

启动hive

./hive

 

 

------------------------------------------------------------------------------------------------------------------------------------------------------

 

二、使用beeline连接，这个比较使用，因为可以使用jdbc让客户端连接

     首先特么的这个是不用重新编译spark的源码的，他需要支持hive

    1、启动spark

    2、启动thriftserver

       cd $SPARK_HOME/sbin

       ./start-thriftserver.sh --master spark://hadoop-spark01:7077 --executor-memory 1g

 

    3、启动hive metastore

        hive --service metastore > metastore.log 2>&1 &    

 

 

    使用beeline连接

        

  [root@hadoop-spark01 logs]# beeline

    beeline> !connect jdbc:hive2://hadoop-spark01:10000

    0: jdbc:hive2://hadoop-spark01:10000> select count(*) from t_trackinfo;

+------+--+

| _c0  |

+------+--+

| 188  |

+------+--+

1 row selected (16.738 seconds)

 

 

 

 

 

 

 

 

 

 

需要注意的几点：

1、我的hive中的数据是从hbase同步过来的。

2、不需要从新编译hive源码。直接从apache官网上下载就可以了。

3、一般使用的都是thriftserver2这种方式，通过客户端程序通过jdbc操作hive。所以不用编译源码，做好相应的配置就可以了。

 

这些配置已经过时，并且写在spark-defaults.conf文件里面，就可以了

 

SPARK_CLASSPATH was detected (set to '/usr/local/soft/sparkclasspath/mysql-connector-java-5.1.38-bin.jar:/usr/local/soft/sparkcla

sspath/hive-hbase-handler-1.2.1.jar:/usr/local/soft/sparkclasspath/hbase-common-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-client-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-protocol-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-server-1.1.2.jar:/usr/local/soft/sparkclasspath/protobuf-java-2.5.0.jar:/usr/local/soft/sparkclasspath/htrace-core-3.1.0-incubating.jar:/usr/local/soft/sparkclasspath/guava-12.0.1.jar:/usr/local/soft/sparkclasspath/hive-exec-1.2.1.jar').This is deprecated in Spark 1.0+.

 

Please instead use:

 - ./spark-submit with --driver-class-path to augment the driver classpath

 - spark.executor.extraClassPath to augment the executor classpath

 

SPARK_WORKER_INSTANCES was detected (set to '2').

This is deprecated in Spark 1.0+.

 

Please instead use:

 - ./spark-submit with --num-executors to specify the number of executors

 - Or set SPARK_EXECUTOR_INSTANCES

 - spark.executor.instances to configure the number of instances in the spark config.