前一篇文章是Spark SQL的入门篇Spark SQL初探,介绍了一些基础知识和API,可是离我们的日常使用还似乎差了一步之遥。

终结Shark的利用有2个:

1、和Spark程序的集成有诸多限制

2、Hive的优化器不是为Spark而设计的,计算模型的不同,使得Hive的优化器来优化Spark程序遇到了瓶颈。

这里看一下Spark SQL 的基础架构:

Spark1.1公布后会支持Spark SQL CLI 。 Spark SQL的CLI会要求被连接到一个Hive Thrift Server上,来实现类似hive shell的功能。

(ps:眼下git里面的branch-1.0-jdbc。眼下还没有正式release,我測了一下午。发现还是有bug的,耐心等待release吧!)

本着研究的心态,想和Hive环境集成一下。在spark shell里运行hive的语句。

一、编译Spark支持Hive

让Spark支持Hive有2种sbt编译方式:

1、sbt前加变量名

SPARK_HADOOP_VERSION=0.20.2-cdh3u5 SPARK_HIVE=true sbt/sbt assembly

2、改动project/SparkBuild.scala文件

val DEFAULT_HADOOP_VERSION = "0.20.2-cdh3u5"
val DEFAULT_HIVE = true 然后运行sbt/sbt assembly

二、Spark SQL 操作Hive

前置:hive可用,而且在Spark-env.sh下,须要将Hive的conf和Hadoop的conf配到CLASSPATH里。

启动spark-shell

[root@web01 spark]# bin/spark-shell --master spark://10.1.8.210:7077 --driver-class-path /app/hadoop/hive-0.11.0-bin/lib/mysql-connector-java-5.1.13-bin.jar:/app/hadoop/hive-0.11.0-bin/lib/hadoop-lzo-0.4.15.jar

导入HiveContext

scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@7766d31c scala> import hiveContext._
import hiveContext._

hiveContext里提供了一个运行sql的函数 hql(string text)

去hive里show databases. 这里Spark会parse hql 然后生成Query Plan。可是这里不会运行查询,仅仅有调用collect的时候才会运行。

scala> val show_databases = hql("show databases")
14/07/09 19:59:09 INFO storage.BlockManager: Removing broadcast 0
14/07/09 19:59:09 INFO storage.BlockManager: Removing block broadcast_0
14/07/09 19:59:09 INFO parse.ParseDriver: Parsing command: show databases
14/07/09 19:59:09 INFO parse.ParseDriver: Parse Completed
14/07/09 19:59:09 INFO analysis.Analyzer: Max iterations (2) reached for batch MultiInstanceRelations
14/07/09 19:59:09 INFO analysis.Analyzer: Max iterations (2) reached for batch CaseInsensitiveAttributeReferences
14/07/09 19:59:09 INFO analysis.Analyzer: Max iterations (2) reached for batch Check Analysis
14/07/09 19:59:09 INFO storage.MemoryStore: Block broadcast_0 of size 393044 dropped from memory (free 308713881)
14/07/09 19:59:09 INFO broadcast.HttpBroadcast: Deleted broadcast file: /tmp/spark-c29da0f8-c5e3-4fbf-adff-9aa77f9743b2/broadcast_0
14/07/09 19:59:09 INFO sql.SQLContext$$anon$1: Max iterations (2) reached for batch Add exchange
14/07/09 19:59:09 INFO sql.SQLContext$$anon$1: Max iterations (2) reached for batch Prepare Expressions
14/07/09 19:59:09 INFO spark.ContextCleaner: Cleaned broadcast 0
14/07/09 19:59:09 INFO ql.Driver: <PERFLOG method=Driver.run>
14/07/09 19:59:09 INFO ql.Driver: <PERFLOG method=TimeToSubmit>
14/07/09 19:59:09 INFO ql.Driver: <PERFLOG method=compile>
14/07/09 19:59:09 INFO exec.ListSinkOperator: 0 finished. closing...
14/07/09 19:59:09 INFO exec.ListSinkOperator: 0 forwarded 0 rows
14/07/09 19:59:09 INFO ql.Driver: <PERFLOG method=parse>
14/07/09 19:59:09 INFO parse.ParseDriver: Parsing command: show databases
14/07/09 19:59:09 INFO parse.ParseDriver: Parse Completed
14/07/09 19:59:09 INFO ql.Driver: </PERFLOG method=parse start=1404907149927 end=1404907149928 duration=1>
14/07/09 19:59:09 INFO ql.Driver: <PERFLOG method=semanticAnalyze>
14/07/09 19:59:09 INFO ql.Driver: Semantic Analysis Completed
14/07/09 19:59:09 INFO ql.Driver: </PERFLOG method=semanticAnalyze start=1404907149928 end=1404907149977 duration=49>
14/07/09 19:59:09 INFO exec.ListSinkOperator: Initializing Self 0 OP
14/07/09 19:59:09 INFO exec.ListSinkOperator: Operator 0 OP initialized
14/07/09 19:59:09 INFO exec.ListSinkOperator: Initialization Done 0 OP
14/07/09 19:59:09 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
14/07/09 19:59:09 INFO ql.Driver: </PERFLOG method=compile start=1404907149925 end=1404907149980 duration=55>
14/07/09 19:59:09 INFO ql.Driver: <PERFLOG method=Driver.execute>
14/07/09 19:59:09 INFO ql.Driver: Starting command: show databases
14/07/09 19:59:09 INFO ql.Driver: </PERFLOG method=TimeToSubmit start=1404907149925 end=1404907149980 duration=55>
14/07/09 19:59:09 INFO ql.Driver: <PERFLOG method=runTasks>
14/07/09 19:59:09 INFO ql.Driver: <PERFLOG method=task.DDL.Stage-0>
14/07/09 19:59:09 INFO metastore.HiveMetaStore: 0: get_all_databases
14/07/09 19:59:09 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_all_databases
14/07/09 19:59:09 INFO exec.DDLTask: results : 1
14/07/09 19:59:10 INFO ql.Driver: </PERFLOG method=task.DDL.Stage-0 start=1404907149980 end=1404907150032 duration=52>
14/07/09 19:59:10 INFO ql.Driver: </PERFLOG method=runTasks start=1404907149980 end=1404907150032 duration=52>
14/07/09 19:59:10 INFO ql.Driver: </PERFLOG method=Driver.execute start=1404907149980 end=1404907150032 duration=52>
14/07/09 19:59:10 INFO ql.Driver: OK
14/07/09 19:59:10 INFO ql.Driver: <PERFLOG method=releaseLocks>
14/07/09 19:59:10 INFO ql.Driver: </PERFLOG method=releaseLocks start=1404907150033 end=1404907150033 duration=0>
14/07/09 19:59:10 INFO ql.Driver: </PERFLOG method=Driver.run start=1404907149925 end=1404907150033 duration=108>
14/07/09 19:59:10 INFO mapred.FileInputFormat: Total input paths to process : 1
14/07/09 19:59:10 INFO ql.Driver: <PERFLOG method=releaseLocks>
14/07/09 19:59:10 INFO ql.Driver: </PERFLOG method=releaseLocks start=1404907150037 end=1404907150037 duration=0>
show_databases: org.apache.spark.sql.SchemaRDD =
SchemaRDD[16] at RDD at SchemaRDD.scala:100
== Query Plan ==
<Native command: executed by Hive>

运行查询计划:

scala> show_databases.collect()
14/07/09 20:00:44 INFO spark.SparkContext: Starting job: collect at SparkPlan.scala:52
14/07/09 20:00:44 INFO scheduler.DAGScheduler: Got job 2 (collect at SparkPlan.scala:52) with 1 output partitions (allowLocal=false)
14/07/09 20:00:44 INFO scheduler.DAGScheduler: Final stage: Stage 2(collect at SparkPlan.scala:52)
14/07/09 20:00:44 INFO scheduler.DAGScheduler: Parents of final stage: List()
14/07/09 20:00:44 INFO scheduler.DAGScheduler: Missing parents: List()
14/07/09 20:00:44 INFO scheduler.DAGScheduler: Submitting Stage 2 (MappedRDD[20] at map at SparkPlan.scala:52), which has no missing parents
14/07/09 20:00:44 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 2 (MappedRDD[20] at map at SparkPlan.scala:52)
14/07/09 20:00:44 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
14/07/09 20:00:44 INFO scheduler.TaskSetManager: Starting task 2.0:0 as TID 9 on executor 0: web01.dw (PROCESS_LOCAL)
14/07/09 20:00:44 INFO scheduler.TaskSetManager: Serialized task 2.0:0 as 1511 bytes in 0 ms
14/07/09 20:00:45 INFO scheduler.DAGScheduler: Completed ResultTask(2, 0)
14/07/09 20:00:45 INFO scheduler.TaskSetManager: Finished TID 9 in 12 ms on web01.dw (progress: 1/1)
14/07/09 20:00:45 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
14/07/09 20:00:45 INFO scheduler.DAGScheduler: Stage 2 (collect at SparkPlan.scala:52) finished in 0.014 s
14/07/09 20:00:45 INFO spark.SparkContext: Job finished: collect at SparkPlan.scala:52, took 0.020520428 s
res5: Array[org.apache.spark.sql.Row] = Array([default])

返回default数据库。

相同的运行:show tables

scala> hql("show tables").collect()
14/07/09 20:01:28 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
14/07/09 20:01:28 INFO scheduler.DAGScheduler: Stage 3 (collect at SparkPlan.scala:52) finished in 0.013 s
14/07/09 20:01:28 INFO spark.SparkContext: Job finished: collect at SparkPlan.scala:52, took 0.019173851 s
res7: Array[org.apache.spark.sql.Row] = Array([item], [src])

理论上是支持HIVE全部的操作,包含UDF。

PS:遇到的问题:

Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BoneCP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.

解决的方法:就是我上面启动的时候带上sql-connector的路径。

三、总结:

Spark SQL 兼容了Hive的大部分语法和UDF,可是在处理查询计划的时候,使用了Catalyst框架进行优化,优化成适合Spark编程模型的运行计划,使得效率上高出hive非常多。

因为Spark1.1临时还未公布。眼下还存在bug,等到稳定版公布了再继续測试了。

全文完:)

原创文章,转载请注明出自:http://blog.csdn.net/oopsoom/article/details/37603261

Spark SQL with Hive的更多相关文章

  1. spark2.3.0 配置spark sql 操作hive

    spark可以通过读取hive的元数据来兼容hive,读取hive的表数据,然后在spark引擎中进行sql统计分析,从而,通过spark sql与hive结合实现数据分析将成为一种最佳实践.配置步骤 ...

  2. spark sql数据源--hive

    使用的是idea编辑器 spark sql从hive中读取数据的步骤:1.引入hive的jar包 2.将hive-site.xml放到resource下 3.spark sql声明对hive的支持 案 ...

  3. Hive on Spark和Spark sql on Hive,你能分的清楚么

    摘要:结构上Hive On Spark和SparkSQL都是一个翻译层,把一个SQL翻译成分布式可执行的Spark程序. 本文分享自华为云社区<Hive on Spark和Spark sql o ...

  4. Spark SQL读取hive数据时报找不到mysql驱动

    Exception: Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BoneC ...

  5. Spark SQL与Hive on Spark的比较

    简要介绍了SparkSQL与Hive on Spark的区别与联系 一.关于Spark 简介 在Hadoop的整个生态系统中,Spark和MapReduce在同一个层级,即主要解决分布式计算框架的问题 ...

  6. Spark SQL 操作Hive 数据

    Spark 2.0以前版本:val sparkConf = new SparkConf().setAppName("soyo")    val spark = new SparkC ...

  7. spark sql 查询hive表并写入到PG中

    import java.sql.DriverManager import java.util.Properties import com.zhaopin.tools.{DateUtils, TextU ...

  8. spark sql 访问hive数据时找不mysql的解决方法

    我尝试着在classpath中加n入mysql的驱动仍不行 解决方法:在启动的时候加入参数--driver-class中加入mysql 驱动 [hadoop@master spark-1.0.1-bi ...

  9. Spark SQL Hive Support Demo

    前提: 1.spark1.0的包编译时指定支持hive:./make-distribution.sh --hadoop 2.3.0-cdh5.0.0 --with-yarn --with-hive - ...

随机推荐

  1. Mysql数据类型(一)

    介绍 存储引擎决定了表的类型,而表内存放的数据也要有不同的类型,每种数据类型都有自己的宽度,但宽度是可选的 详细参考链接:http://www.runoob.com/mysql/mysql-data- ...

  2. LeetCode Weekly Contest 25

    1. 507. Perfect Number 显然小于2的都不满足(尤其是负数的情况),进一步,显然质数都不满足,所以小于4的数,直接return false. 然后依次暴力枚举判断到sqrt(n), ...

  3. 2.TinkPHP入门----控制器

    1.控制器创建 命名规则:控制器名称+Controller+.class.php, 例如GoodsController.class.php  UserController.class.php 控制器结 ...

  4. git window端工具之sourcetree使用

    https://www.jianshu.com/p/3478e2a214a1

  5. SourceInsight3.5 Space 替换Tab

    # SourceInsight3.5 Space 替换Tab 公司要求所有的缩进都要使用空格,而不是Tab.至于使用Tab,还是Space来进行缩进,这在网上有各种各样的讨论,毕竟使用Tab可以节省很 ...

  6. hdu2121 Ice_cream’s world II 最小树形图(难)

    这题比HDU4009要难一些.做了4009,大概知道了最小树形图的解法.拿到这题,最直接的想法是暴力.n个点试过去,每个都拿来做一次根.最后WA了,估计是超时了.(很多题都是TLE说成WA,用了G++ ...

  7. Linker scripts之SECTIONS

    1 Purpose The linker script describes how the sections in the input files should be mapped into the ...

  8. 我的C++笔记(语句基本结构)

    #include <iostream> using namespace std; int main() { unsigned char c1=24; int year; bool isLe ...

  9. java连接AD域

    import org.springframework.boot.autoconfigure.SpringBootApplication; import java.util.Hashtable; imp ...

  10. 认识图片放大工具PhotoZoom的菜单栏

    使用PhotoZoom能够对数码图片无损放大,备受设计师和业内人员的青睐,它的出现时一场技术的革新,新颖的技术,简单的界面,优化的算法,使得它可以对图片进行放大而没有锯齿,不会失真.本文为您一起来认识 ...