Spark（五十三）：Spark RPC初尝试使用

基本用法主要掌握一点就行：

master slave模式运用：driver 就是master，executor就是slave。

如果executor要想和driver交互必须拿到driver的EndpointRef，通过driver的EndpointRef来调接口访问。

driver启动时，会在driver中注册一个Endpoint服务，并暴露自己的ip和端口。executor端生成driver的EndpointRef，就主要需要两个参数就行：driver的host(ip)和port。

导入Maven依赖

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.4.0</version>
        </dependency>

定义RPC Server端的ip（localhost）。port(57992)、服务名称(hello-rpc-service)

object HelloRpcSettings {
  val rpcName = "hello-rpc-service"
  val port = 57992
  val hostname="localhost"

  def getName() = {
    rpcName
  }

  def getPort(): Int = {
    port
  }

  def getHostname():String={
    hostname
  }
}

定义RPC的Endpoint类和发送数据类SayHi/SayBye

case class SayHi(msg: String)

case class SayBye(msg: String)

import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEnv}

class HelloEndpoint(override val rpcEnv: RpcEnv) extends RpcEndpoint {
  override def onStart(): Unit = {
    println(rpcEnv.address)
    println("start hello endpoint")
  }

  override def receive: PartialFunction[Any, Unit] = {
    case SayHi(msg) =>
      println(s"receive $msg" )
  }

  override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
    case SayHi(msg) => {
      println(s"receive $msg")
      context.reply(s"hi, $msg")
    }
    case SayBye(msg) => {
      println(s"receive $msg")
      context.reply(s"bye, $msg")
    }
  }

  override def onStop(): Unit = {
    println("stop hello endpoint")
  }
}

定义RPC 服务提供者

import org.apache.spark.{SecurityManager, SparkConf, SparkContext, SparkEnv}
import org.apache.spark.rpc._
import org.apache.spark.sql.SparkSession

object RpcServerTest {
  def main(args: Array[String]): Unit = {
    val conf: SparkConf = new SparkConf()
    val sparkSession = SparkSession.builder().config(conf).master("local[*]").appName("test rpc").getOrCreate()
    val sparkContext: SparkContext = sparkSession.sparkContext
    val sparkEnv: SparkEnv = sparkContext.env

    val rpcEnv = RpcEnv.create(HelloRpcSettings.getName(), HelloRpcSettings.getHostname(), HelloRpcSettings.getHostname(), HelloRpcSettings.getPort(), conf,
      sparkEnv.securityManager, 1, false)

    val helloEndpoint: RpcEndpoint = new HelloEndpoint(rpcEnv)
    rpcEnv.setupEndpoint(HelloRpcSettings.getName(), helloEndpoint)

    rpcEnv.awaitTermination()
  }
}

定义RPC服务使用者

import org.apache.spark.{SecurityManager, SparkConf, SparkContext, SparkEnv}
import org.apache.spark.rpc.{RpcAddress, RpcEndpointRef, RpcEnv, RpcEnvConfig}
import org.apache.spark.sql.{Dataset, Row, SparkSession}

import scala.concurrent.duration.Duration
import scala.concurrent.{Await, Future}

object RpcClientTest {
  def main(args: Array[String]): Unit = {
    val conf: SparkConf = new SparkConf()
    val sparkSession = SparkSession.builder().config(conf).master("local[*]").appName("test rpc").getOrCreate()
    val sparkContext: SparkContext = sparkSession.sparkContext
    val sparkEnv: SparkEnv = sparkContext.env

    val rpcEnv: RpcEnv = RpcEnv.create(HelloRpcSettings.getName(),HelloRpcSettings.getHostname(),HelloRpcSettings.getPort(),conf,sparkEnv.securityManager,false)
    val endPointRef: RpcEndpointRef = rpcEnv.setupEndpointRef(RpcAddress(HelloRpcSettings.getHostname(), HelloRpcSettings.getPort()), HelloRpcSettings.getName())

    import scala.concurrent.ExecutionContext.Implicits.global

    endPointRef.send(SayHi("test send"))

    val future: Future[String] = endPointRef.ask[String](SayHi("neo"))
    future.onComplete {
      case scala.util.Success(value) => println(s"Got the result = $value")
      case scala.util.Failure(e) => println(s"Got error: $e")
    }
    Await.result(future, Duration.apply("30s"))

    val res = endPointRef.askSync[String](SayBye("test askSync"))
    println(res)

    sparkSession.stop()
  }

}

启动RPC 服务提供者

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
// :: INFO SparkContext: Running Spark version 2.4.
// :: WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
// :: INFO SparkContext: Submitted application: test rpc
// :: INFO SecurityManager: Changing view acls to: boco
// :: INFO SecurityManager: Changing modify acls to: boco
// :: INFO SecurityManager: Changing view acls groups to:
// :: INFO SecurityManager: Changing modify acls groups to:
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(boco); groups with view permissions: Set(); users  with modify permissions: Set(boco); groups with modify permissions: Set()
// :: INFO Utils: Successfully started service 'sparkDriver' on port .
// :: INFO SparkEnv: Registering MapOutputTracker
// :: INFO SparkEnv: Registering BlockManagerMaster
// :: INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
// :: INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
// :: INFO DiskBlockManager: Created local directory at C:\Users\boco\AppData\Local\Temp\blockmgr-7128dde8-9c46--bb72-c2161ba65bf7
// :: INFO MemoryStore: MemoryStore started with capacity 901.8 MB
// :: INFO SparkEnv: Registering OutputCommitCoordinator
// :: INFO Utils: Successfully started service 'SparkUI' on port .
// :: INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://DESKTOP-JL4FSCV:4040
// :: INFO Executor: Starting executor ID driver on host localhost
// :: INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port .
// :: INFO NettyBlockTransferService: Server created on DESKTOP-JL4FSCV:
// :: INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
// :: INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, DESKTOP-JL4FSCV, , None)
// :: INFO BlockManagerMasterEndpoint: Registering block manager DESKTOP-JL4FSCV: with 901.8 MB RAM, BlockManagerId(driver, DESKTOP-JL4FSCV, , None)
// :: INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, DESKTOP-JL4FSCV, , None)
// :: INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, DESKTOP-JL4FSCV, , None)
// :: INFO Utils: Successfully started service 'hello-rpc-service' on port .
localhost:
start hello endpoint

启动RPC 服务使用者

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
// :: INFO SparkContext: Running Spark version 2.4.
// :: WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
// :: INFO SparkContext: Submitted application: test rpc
// :: INFO SecurityManager: Changing view acls to: boco
// :: INFO SecurityManager: Changing modify acls to: boco
// :: INFO SecurityManager: Changing view acls groups to:
// :: INFO SecurityManager: Changing modify acls groups to:
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(boco); groups with view permissions: Set(); users  with modify permissions: Set(boco); groups with modify permissions: Set()
// :: INFO Utils: Successfully started service 'sparkDriver' on port .
// :: INFO SparkEnv: Registering MapOutputTracker
// :: INFO SparkEnv: Registering BlockManagerMaster
// :: INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
// :: INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
// :: INFO DiskBlockManager: Created local directory at C:\Users\boco\AppData\Local\Temp\blockmgr-6a0b8e7f-86d2-4bb8-b45c-7c04deabcb91
// :: INFO MemoryStore: MemoryStore started with capacity 901.8 MB
// :: INFO SparkEnv: Registering OutputCommitCoordinator
// :: WARN Utils: Service 'SparkUI' could not bind on port . Attempting port .
// :: INFO Utils: Successfully started service 'SparkUI' on port .
// :: INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://DESKTOP-JL4FSCV:4041
// :: INFO Executor: Starting executor ID driver on host localhost
// :: INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port .
// :: INFO NettyBlockTransferService: Server created on DESKTOP-JL4FSCV:
// :: INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
// :: INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, DESKTOP-JL4FSCV, , None)
// :: INFO BlockManagerMasterEndpoint: Registering block manager DESKTOP-JL4FSCV: with 901.8 MB RAM, BlockManagerId(driver, DESKTOP-JL4FSCV, , None)
// :: INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, DESKTOP-JL4FSCV, , None)
// :: INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, DESKTOP-JL4FSCV, , None)
// :: WARN Utils: Service 'hello-rpc-service' could not bind on port . Attempting port .
// :: INFO Utils: Successfully started service 'hello-rpc-service' on port .
// :: INFO TransportClientFactory: Successfully created connection to localhost/127.0.0.1: after  ms ( ms spent in bootstraps)
bye, test askSync
Got the result = hi, neo
// :: INFO SparkUI: Stopped Spark web UI at http://DESKTOP-JL4FSCV:4041
// :: INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
// :: INFO MemoryStore: MemoryStore cleared
// :: INFO BlockManager: BlockManager stopped
// :: INFO BlockManagerMaster: BlockManagerMaster stopped
// :: INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
// :: INFO SparkContext: Successfully stopped SparkContext
// :: INFO ShutdownHookManager: Shutdown hook called
// :: INFO ShutdownHookManager: Deleting directory 

此时 RPC 服务提供者打印信息如下：

receive test send
receive neo
receive test askSync
// :: WARN TransportChannelHandler: Exception in connection from /127.0.0.1:
java.io.IOException: 远程主机强迫关闭了一个现有的连接。
    at sun.nio.ch.SocketDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:)
    at sun.nio.ch.IOUtil.read(IOUtil.java:)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:)
    at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:)
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:)
    at io.netty.util.concurrent.SingleThreadEventExecutor$.run(SingleThreadEventExecutor.java:)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:)
    at java.lang.Thread.run(Thread.java:)