大数据入门到精通12--spark dataframe 注册成hive 的临时表

一、获得最初的数据并形成dataframe

val ny= sc.textFile("data/new_york/")
val header=ny.first
val filterNY =ny.filter(listing=>{
listing.split(",").size==14 && listing!=header
})
val nyMap= filterNY.map(listing=>{
val listingInfo=listing.split(",")
(listingInfo(0).toInt,listingInfo(2),listingInfo(9).toFloat,listingInfo(4))

})
nyMap.take(20).foreach(println)

val nyDF=nyMap.toDF("Room_ID","Room_Type","Price","Neighborhood")

nyDF.show

二、注册成临时表

nyDF.registerTempTable("listings")

//如果不成功可以执行

cp /etc/hive/conf/hive-site.xml /etc/spark/conf

三、通过hql对临时表做select，聚集函数，group by，sort  by  where等等操作

spark.sqlContext.sql("select * from listings limit 10").show

执行sql以后的结果也是dataframe，所以可以用show这些操作

scala> spark.sqlContext.sql("select Neighborhood,count(Price) as sump ,round(avg(Price),2) as avgp from listings where Room_Type!='' group by Neighborhood order by sump desc" ).show
+------------------+-----+------+
| Neighborhood| sump| avgp|
+------------------+-----+------+
| Williamsburg|77973|148.53|
|Bedford-Stuyvesant|54667|108.34|
| Harlem|50551|122.45|
| East Village|43979|196.45|
| Upper West Side|42466| 215.9|
| Bushwick|37670| 87.97|
| Upper East Side|36950|205.24|
| Hell's Kitchen|34202|217.42|
| Crown Heights|28033|106.34|
| Chelsea|25551|258.95|
| Lower East Side|22982|184.73|
| Midtown|22746|286.51|
| East Harlem|22393|133.19|
| Greenpoint|21167|144.98|
| West Village|20370|285.79|
|Washington Heights|18276| 99.08|
| Astoria|15586|110.39|
| Clinton Hill|12244|191.38|
| Flatbush|11430| 96.74|
| Park Slope|11294|173.55|
+------------------+-----+------+
only showing top 20 rows

这些都是hql的基础sql语法，不需要过多的讲解了