es-09-spark集成

es和spark的集成比较简单, 直接使用内部封装的一些方法即可

版本设置说明:

https://www.elastic.co/guide/en/elasticsearch/hadoop/current/requirements.html

maven依赖说明:

https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html

1, maven配置:

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <parent>

        <artifactId>xiaoniubigdata</artifactId>

        <groupId>com.wenbronk</groupId>

        <version>1.0</version>

    </parent>

    <modelVersion>4.0.</modelVersion>

    <artifactId>spark06-es</artifactId>

    <properties>

        <spark.version>2.3.</spark.version>

        <spark.scala.version>2.11</spark.scala.version>

        <scala.version>2.11.</scala.version>

    </properties>

    <dependencies>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-core_${spark.scala.version}</artifactId>

            <version>${spark.version}</version>

            <!--<scope>provided</scope>-->

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-sql_${spark.scala.version}</artifactId>

            <version>${spark.version}</version>

            <!--<scope>provided</scope>-->

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-streaming_${spark.scala.version}</artifactId>

            <version>${spark.version}</version>

            <!--<scope>provided</scope>-->

        </dependency>

        <dependency>

            <groupId>org.elasticsearch</groupId>

            <artifactId>elasticsearch-spark-20_2.</artifactId>

            <version>6.3.</version>

        </dependency>

    </dependencies>

    <build>

        <plugins>

            <plugin>

                <groupId>org.scala-tools</groupId>

                <artifactId>maven-scala-plugin</artifactId>

                <version>2.15.</version>

                <executions>

                    <execution>

                        <goals>

                            <goal>compile</goal>

                            <goal>testCompile</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-deploy-plugin</artifactId>

                <version>2.8.</version>

                <configuration>

                    <skip>true</skip>

                </configuration>

            </plugin>

        </plugins>

    </build>

</project>

2, RDD的使用

1), read

package com.wenbronk.spark.es.rdd

import org.apache.spark.rdd.RDD

import org.apache.spark.sql.SparkSession

import org.apache.spark.{SparkConf, SparkContext}

/**

  * 从es中读取数据

  */

object ReadMain {

  def main(args: Array[String]) = {

//    val sparkconf = new SparkConf().setAppName("read-es").setMaster("local[4]")

//    val spark = new SparkContext(sparkconf)

    val sparkSession = SparkSession.builder()

      .appName("read-es-rdd")

      .master("local[4]")

      .config("es.index.auto.create", true)

      .config("es.nodes", "10.124.147.22")

      .config("es.port", )

      .getOrCreate()

    val spark = sparkSession.sparkContext

    // 自定义query, 导入es包

    import org.elasticsearch.spark._

    // 以array方式读取

    val esreadRdd: RDD[(String, collection.Map[String, AnyRef])] = spark.esRDD("macsearch_fileds/mac",

      """

        |{

        |  "query": {

        |    "match_all": {}

        |  }

        |}

      """.stripMargin)

    val value: RDD[(Option[AnyRef], Int)] = esreadRdd.map(_._2.get("mac")).map(mac => (mac, )).reduceByKey(_ + _)

      .sortBy(_._2)

    val tuples: Array[(Option[AnyRef], Int)] = value.collect()

    tuples.foreach(println)

    esreadRdd.saveAsTextFile("/Users/bronkwen/work/IdeaProjects/xiaoniubigdata/spark06-es/target/json")

    sparkSession.close()

  }

}

2, readJson

package com.wenbronk.spark.es.rdd

import org.apache.spark.sql.SparkSession

import scala.util.parsing.json.JSON

object ReadJsonMain {

  def main(args: Array[String]): Unit = {

    val sparkSession = SparkSession.builder()

      .appName("read-es-rdd")

      .master("local[4]")

      .config("es.index.auto.create", true)

      .config("es.nodes", "10.124.147.22")

      .config("es.port", )

      .getOrCreate()

    val spark = sparkSession.sparkContext

    // 使用json的方式读取, 带查询的

    import org.elasticsearch.spark._

    val esJsonRdd = spark.esJsonRDD("macsearch_fileds/mac",

      """

      {

        "query": {

          "match_all": {}

        }

      }

      """.stripMargin)

    esJsonRdd.map(_._2).saveAsTextFile("/Users/bronkwen/work/IdeaProjects/xiaoniubigdata/spark06-es/target/json")

    sparkSession.close()

  }

}

3, write

package com.wenbronk.spark.es.rdd

import org.apache.spark.rdd.RDD

import org.apache.spark.sql.SparkSession

import org.elasticsearch.spark.rdd.EsSpark

object WriteMain {

  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder()

      .master("local[4]")

      .appName("write-spark-es")

      .config("es.index.auto.create", true)

      .config("es.nodes", "10.124.147.22")

      .config("es.port", )

      .getOrCreate()

    val df: RDD[String] = spark.sparkContext.textFile("/Users/bronkwen/work/IdeaProjects/xiaoniubigdata/spark06-es/target/json")

//    df.map(_.substring())

    import org.elasticsearch.spark._

//    df.rdd.saveToEs("spark/docs")

//    EsSpark.saveToEs(df, "spark/docs")

    EsSpark.saveJsonToEs(df, "spark/json")

    spark.close()

  }

}

4, 写入多个index中

package com.wenbronk.spark.es.rdd

import org.apache.spark.sql.SparkSession

object WriteMultiIndex {

  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder()

      .master("local[4]")

      .appName("es-spark-multiindex")

      .config("es.es.index.auto.create", true)

      .config("es.nodes", "10.124.147.22")

      .config("es.port", )

      .getOrCreate()

    val sc = spark.sparkContext

    val game = Map("media_type"->"game","title" -> "FF VI","year" -> "")

    val book = Map("media_type" -> "book","title" -> "Harry Potter","year" -> "")

    val cd = Map("media_type" -> "music","title" -> "Surfing With The Alien")

    import org.elasticsearch.spark._

    // 可以自定义自己的metadata, 只添加id

    sc.makeRDD(Seq((, game), (, book), (, cd))).saveToEs("my-collection-{media_type}/doc")

    spark.close()

  }

}

2, streaming

1), write

package com.wenbronk.spark.es.stream

import org.apache.spark.streaming.dstream.InputDStream

import org.apache.spark.{SparkConf, SparkContext}

import org.apache.spark.streaming.{Seconds, StreamingContext}

import org.elasticsearch.spark.rdd.EsSpark

import org.elasticsearch.spark.streaming.EsSparkStreaming

import scala.collection.mutable

object WriteStreamingMain {

  def main (args: Array[String]): Unit = {

    val conf = new SparkConf().setAppName("es-spark-streaming-write").setMaster("local[4]")

      conf.set("es.index.auto.create", "true")

      conf.set("es.nodes", "10.124.147.22")

      // 默认端口9200， 不知道怎么设置 Int类型

    val sc = new SparkContext(conf)

    val ssc = new StreamingContext(sc, Seconds())

    val numbers = Map("one" -> , "two" -> , "three" -> )

    val airports = Map("arrival" -> "Otopeni", "SFO" -> "San Fran")

    val rdd = sc.makeRDD(Seq(numbers, airports))

    val microbatches = mutable.Queue(rdd)

    val dstream: InputDStream[Map[String, Any]] = ssc.queueStream(microbatches)

//    import org.elasticsearch.spark.streaming._

//    dstream.saveToEs("sparkstreaming/doc")

//    EsSparkStreaming.saveToEs(dstream, "sparkstreaming/doc")

    // 带有id的

//    EsSparkStreaming.saveToEs(dstream, "spark/docs", Map("es.mapping.id" -> "id"))

    // json格式

    EsSparkStreaming.saveJsonToEs(dstream, "sparkstreaming/json")

    ssc.start()

    ssc.awaitTermination()

  }

}

2, 写入带有meta的, rdd也是用

package com.wenbronk.spark.es.stream

import org.apache.spark.{SparkConf, SparkContext}

import org.apache.spark.streaming.{Seconds, StreamingContext}

import scala.collection.mutable

object WriteStreamMeta {

  def main(args: Array[String]): Unit = {

    val conf = new SparkConf().setAppName("es-spark-streaming-write").setMaster("local[4]")

    conf.set("es.index.auto.create", "true")

    conf.set("es.nodes", "10.124.147.22")

    // 默认端口9200， 不知道怎么设置 Int类型

    val sc = new SparkContext(conf)

    val ssc = new StreamingContext(sc, Seconds())

    val otp = Map("iata" -> "OTP", "name" -> "Otopeni")

    val muc = Map("iata" -> "MUC", "name" -> "Munich")

    val sfo = Map("iata" -> "SFO", "name" -> "San Fran")

    val airportsRDD = sc.makeRDD(Seq((, otp), (, muc), (, sfo)))

    val microbatches = mutable.Queue(airportsRDD)

    import org.elasticsearch.spark.streaming._

    ssc.queueStream(microbatches).saveToEsWithMeta("airports/2015")

    ssc.start()

    ssc.awaitTermination()

  }

  /**

    * 使用多种meta

    */

  def main1(args: Array[String]): Unit = {

    val ID = "id";

    val TTL = "ttl"

    val VERSION = "version"

    val conf = new SparkConf().setAppName("es-spark-streaming-write").setMaster("local[4]")

    val sc = new SparkContext(conf)

    val ssc = new StreamingContext(sc, Seconds())

    val otp = Map("iata" -> "OTP", "name" -> "Otopeni")

    val muc = Map("iata" -> "MUC", "name" -> "Munich")

    val sfo = Map("iata" -> "SFO", "name" -> "San Fran")

    // 定义meta 不需要一对一对应

    val otpMeta = Map(ID -> , TTL -> "3h")

    val mucMeta = Map(ID -> , VERSION -> "")

    val sfoMeta = Map(ID -> )

    val airportsRDD = sc.makeRDD(Seq((otpMeta, otp), (mucMeta, muc), (sfoMeta, sfo)))

    val microbatches = mutable.Queue(airportsRDD)

    import org.elasticsearch.spark.streaming._

    ssc.queueStream(microbatches).saveToEsWithMeta("airports/2015")

    ssc.start()

    ssc.awaitTermination()

  }

}

3, sql的使用

1), read

package com.wenbronk.spark.es.sql

import org.apache.spark.sql.{DataFrame, SparkSession}

object ESSqlReadMain {

  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder()

      .master("local[4]")

      .appName("es-sql-read")

      .config("es.index.auto.create", true)

      // 转换sql为es的DSL

      .config("pushown", true)

      .config("es.nodes", "10.124.147.22")

      .config("es.port", )

      .getOrCreate()

    // 完全查询

//    val df: DataFrame = spark.read.format("es").load("macsearch_fileds/mac")

    import org.elasticsearch.spark.sql._

    val df = spark.esDF("macsearch_fileds/mac",

      """

        |{

        |   "query": {

        |     "match_all": {

        |   }

        |}

      """.stripMargin)

    // 显示下数据

    df.printSchema()

    df.createOrReplaceTempView("macseach_fileds")

    val dfSql: DataFrame = spark.sql(

      """

        select

         mac,

         count(mac) con

        from macseach_fileds

        group by mac

        order by con desc

      """.stripMargin)

    dfSql.show()

    // 存入本地文件中

    import spark.implicits._

    df.write.json("/Users/bronkwen/work/IdeaProjects/xiaoniubigdata/spark06-es/target/sql/json")

    spark.stop()

  }

}

2), write

package com.wenbronk.spark.es.sql

import org.apache.spark.sql.{DataFrame, SparkSession}

import org.elasticsearch.spark.sql.EsSparkSQL

object ESSqlWriteMain {

  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder()

      .master("local[4]")

      .appName("es-sql-write")

      .config("es.index.auto.create", true)

      .config("es.nodes", "10.124.147.22")

      .config("es.port", )

      .getOrCreate()

    import spark.implicits._

    val df: DataFrame = spark.read.format("json").load("/Users/bronkwen/work/IdeaProjects/xiaoniubigdata/spark06-es/target/sql/json")

    df.show()

    // json格式直接写入

//    import org.elasticsearch.spark.sql._

//    df.saveToEs("spark/people")

    EsSparkSQL.saveToEs(df, "spark/people")

    spark.close()

  }

}

4, structStream

对结构化流不太熟悉, 等熟悉了在看

package com.wenbronk.spark.es.structstream

import org.apache.spark.sql.SparkSession

object StructStreamWriteMain {

  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder()

      .appName("structstream-es-write")

      .master("local[4]")

      .config("es.index.auto.create", true)

      .config("es.nodes", "10.124.147.22")

      .config("es.port", )

      .getOrCreate()

    val df = spark.readStream

        .format("json")

      .load("/Users/bronkwen/work/IdeaProjects/xiaoniubigdata/spark06-es/target/json")

    df.writeStream

        .option("checkpointLocation", "/save/location")

        .format("es")

        .start()

    spark.close()

  }

}

es-09-spark集成的更多相关文章

Spark：利用Eclipse构建Spark集成开发环境
前一篇文章“Apache Spark学习:将Spark部署到Hadoop 2.2.0上”介绍了如何使用Maven编译生成可直接运行在Hadoop 2.2.0上的Spark jar包,而本文则在此基础上 ...
spark集成hive遭遇mysql check失败的问题
问题: spark集成hive,启动spark-shell或者spark-sql的时候,报错: INFO MetaStoreDirectSql: MySQL check failed, assumin ...
09.客户端集成IdentityServer
09.客户端集成IdentityServer 新建API的项目 dotnet new webapi --name ClientCredentialApi 在我们上一节课的代码IdentityServe ...
Spark集成
一.Spark 架构与优化器 1.Spark架构 (重点) 2.Spark优化器二.Spark+SQL的API (重点) 1.DataSet简介 2.DataFrame简介 3.RDD与DF/DS的 ...
机器学习 - pycharm, pyspark, spark集成篇
AS WE ALL KNOW,学机器学习的一般都是从python+sklearn开始学,适用于数据量不大的场景(这里就别计较“不大”具体指标是啥了,哈哈) 数据量大了,就需要用到其他技术了,如:spa ...
机器学习 - 开发环境安装pycharm + pyspark + spark集成篇
AS WE ALL KNOW,学机器学习的一般都是从python+sklearn开始学,适用于数据量不大的场景(这里就别计较“不大”具体指标是啥了,哈哈) 数据量大了,就需要用到其他技术了,如:spa ...
spark 集成elasticsearch
pyspark读写elasticsearch依赖elasticsearch-hadoop包,需要首先在这里下载,版本号可以通过自行修改url解决. """ write d ...
Ignite与Spark集成时，ClassNotFoundException问题解决
参考文章:https://apacheignite-fs.readme.io/docs/installation-deployment Spark application deployment mod ...
spark集成hbase与hive数据转换与代码练习
帮一个朋友写个样例,顺便练手啦~一直在做平台的各种事,但是代码后续还要精进啊... import java.util.Date import org.apache.hadoop.hbase.HBase ...
ES 09 - 定制Elasticsearch的分词器 (自定义分词策略)
目录 1 索引的分析 1.1 分析器的组成 1.2 倒排索引的核心原理-normalization 2 ES的默认分词器 3 修改分词器 4 定制分词器 4.1 向索引中添加自定义的分词器 4.2 测 ...

随机推荐

Visual Studio 2017快捷键
作者:tongqingliu 转载请注明出处:http://www.cnblogs.com/liutongqing/p/7048639.html Visual Studio 2017快捷键 Ctrl+ ...
NSUserDefaults 添加与删除
//NSUserDefaults会创建一个plist文件,内部存放一个字典 NSUserDefaults *userDefaults = [NSUserDefaults standardUser ...
Java应用分类
Java应用分类一.应用程序.指在操作系统上直接运行的,不是浏览器,Java环境用本机的,需要在客户端安装,Java环境可以一起安装. 1.GUI图形界面应用程序 ...
C#-VS支持的语言
其中C语言选C++
MCU_数码管常用表
共阴极数码管编码(0---F) unsigned char code table[]={ 0x3f,0x06,0x5d,0x4f, 0x66,0x6d,0x77,0x7c, 0x39,0x5e,0x7 ...
A - Class Statistics
A - Class Statistics Time Limit:3000MS Memory Limit:0KB 64bit IO Format:%lld & %llu Subm ...
headpq
从一个集合中获得最大或者最小的N个元素列表 http://python3-cookbook.readthedocs.io/zh_CN/latest/c01/p04_find_largest_or_sm ...
dot net core 使用 IPC 进程通信
本文告诉大家如何使用dot net core 和其他进程进行通信一般都是使用 WCF 或 remoting 做远程通信,但是 dot net core 不支持 WCF 所以暂时我就只能使用管道通信 ...
MVC 5 Strongly Typed Views(强类型视图)
学习MVC这样久以来,发觉网站上很多MVC的视频或是文章,均是使用Strongly Type views来实现控制器与视图的交互.Insus.NET以前发布的博文中,也大量使用这种方式: <Da ...
Linux系统日志分析与管理(14)
当你的 Linux 系统出现不明原因的问题时,你需要查阅一下系统日志才能够知道系统出了什么问题了,所以说了解系统日志是很重要的事情,系统日志可以记录系统在什么时间.哪个主机.哪个服务.出现了什么信息等 ...

es-09-spark集成

1, maven配置:

2, RDD的使用

2, streaming

3, sql的使用

4, structStream

es-09-spark集成的更多相关文章

随机推荐

热门专题