ScalaPB（1）： using protobuf in akka
任何类型的实例作为消息在两端独立系统的机器之间进行传递时必须经过序列化/反序列化serialize/deserialize处理过程。假设以下场景：在一个网络里有两台连接的服务器，它们分别部署了独立的akka系统。如果我们需要在这两台服务器的akka系统之间进行消息交换的话，所有消息都必须经过序列化/反序列化处理。akka系统对于用户自定义消息类型的默认序列化处理是以java-object serialization 方式进行的。我们上次提过：由于java-object-serialization会把一个java-object的类型信息、实例值、它所包含的其它类型描述信息等都写入序列化的结果里，所以会占据较大空间，传输数据的效率相对就低了。protobuf是binary格式的，基本只包括实例值，所以数据传输效率较高。下面我们就介绍如何在akka系统中使用protobuf序列化。在akka中使用自定义序列化方法包括下面的这些步骤：
1、在.proto文件中对消息类型进行IDL定义
2、用ScalaPB编译IDL文件并产生scala源代码。这些源代码中包括了涉及的消息类型及它们的操作方法
3、在akka程序模块中import产生的classes，然后直接调用这些类型和方法
4、按akka要求编写序列化方法
5、在akka的.conf文件里actor.serializers段落中定义akka的默认serializer
下面的build.sbt文件里描述了程序结构：
lazy val commonSettings = Seq(
name := "AkkaProtobufDemo",
version := "1.0",
scalaVersion := "2.12.6",
) lazy val local = (project in file("."))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11",
"com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"
),
name := "akka-protobuf-demo"
) lazy val remote = (project in file("remote"))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11"
),
name := "remote-system"
).dependsOn(local) PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)
local和remote是两个分开的项目。我们会在这两个项目里分别部署akka系统。注意依赖项中的scalapb.runtime。PB.targets指明了产生源代码的路径。我们还需要在project/scalapb.sbt中指定scalaPB插件：
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.18")
libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.7.1"
我们首先在.proto文件里定义消息：
syntax = "proto3"; // Brought in from scalapb-runtime
import "scalapb/scalapb.proto";
import "google/protobuf/wrappers.proto"; package learn.proto; message Added { int32 nbr1 = ;
int32 nbr2 = ;
} message Subtracted {
int32 nbr1 = ;
int32 nbr2 = ;
} message AddedResult {
int32 nbr1 = ;
int32 nbr2 = ;
int32 result = ;
} message SubtractedResult {
int32 nbr1 = ;
int32 nbr2 = ;
int32 result = ;
}
现在我们先在remote项目里定义一个Calculator actor：
package akka.protobuf.calculator
import akka.actor._
import com.typesafe.config.ConfigFactory
import learn.proto.messages._ class Calculator extends Actor with ActorLogging { override def receive: Receive = {
case Added(a,b) =>
log.info("Calculating %d + %d".format(a, b))
sender() ! AddedResult(a,b,a+b)
case Subtracted(a,b) =>
log.info("Calculating %d - %d".format(a, b))
sender() ! SubtractedResult(a,b,a-b)
} } object Calculator {
def props = Props(new Calculator)
} object CalculatorStarter extends App { val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2552")
.withFallback(ConfigFactory.load()) val calcSystem = ActorSystem("calcSystem",config) calcSystem.actorOf(Calculator.props,"calculator") println("press any key to end program ...") scala.io.StdIn.readLine() calcSystem.terminate() }
运行CalculatorStarter产生一个calculator actor: akka.tcp://calcSystem@127.0.0.1:2552/user/calculator
下面我们在local项目里从端口2551上部署另一个akka系统，然后调用端口2552上部署akka系统的calculator actor：
package akka.protobuf.calcservice
import akka.actor._
import learn.proto.messages._
import scala.concurrent.duration._ class CalcRunner(path: String) extends Actor with ActorLogging {
sendIdentifyRequest() def sendIdentifyRequest(): Unit = {
context.actorSelection(path) ! Identify(path)
import context.dispatcher
context.system.scheduler.scheduleOnce(.seconds, self, ReceiveTimeout)
} def receive = identifying def identifying : Receive = {
case ActorIdentity(calcPath,Some(calcRef)) if (path.equals(calcPath)) =>
log.info("Remote calculator started!")
context.watch(calcRef)
context.become(calculating(calcRef))
case ActorIdentity(_,None) =>
log.info("Remote calculator not found!")
case ReceiveTimeout =>
sendIdentifyRequest()
case s @ _ =>
log.info(s"Remote calculator not ready. [$s]")
} def calculating(calculator: ActorRef) : Receive = {
case (op : Added) => calculator ! op
case (op : Subtracted) => calculator ! op case AddedResult(a,b,r) =>
log.info(s"$a + $b = $r")
case SubtractedResult(a,b,r) =>
log.info(s"$a - $b = $r") case Terminated(calculator) =>
log.info("Remote calculator terminated, restarting ...")
sendIdentifyRequest()
context.become(identifying) case ReceiveTimeout => //nothing
} } object CalcRunner {
def props(path: String) = Props(new CalcRunner(path))
}
这个CalcRunner是一个actor，在程序里首先通过向remote项目中的calculator-actor传送Identify消息以取得具体的ActorRef。然后用这个ActorRef与calculator-actor进行交互。这其中Identify是akka预定消息类型，其它消息都是ScalaPB从.proto文件中产生的。下面是local项目的运算程序：
package akka.protobuf.demo
import akka.actor._
import akka.util.Timeout
import com.typesafe.config.ConfigFactory
import akka.protobuf.calcservice._ import scala.concurrent.duration._
import scala.util._
import learn.proto.messages._ object Main extends App { val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2551")
.withFallback(ConfigFactory.load()) val calcSystem = ActorSystem("calcSystem",config) val calcPath = "akka.tcp://calcSystem@127.0.0.1:2552/user/calculator" val calculator = calcSystem.actorOf(CalcRunner.props(calcPath),"calcRunner") println("Calculator started ...") import calcSystem.dispatcher calcSystem.scheduler.schedule(.second, .second) {
if (Random.nextInt() % == )
calculator ! Added(Random.nextInt(), Random.nextInt())
else
calculator ! Subtracted(Random.nextInt(), Random.nextInt())
} scala.io.StdIn.readLine() }
配置文件application.conf：
akka {
actor {
provider = remote
}
remote {
netty.tcp {
hostname = "127.0.0.1"
}
}
}
先运行remote然后local。注意下面出现的提示：
[akka.serialization.Serialization(akka://calcSystem)] Using the default Java serializer for class [learn.proto.messages.Added] which is not recommended because of performance implications. Use another serializer
下面是protobuf类型的序列化方法：
package akka.protobuf.serializer import akka.serialization.SerializerWithStringManifest
import learn.proto.messages._ class ProtobufSerializer extends SerializerWithStringManifest{ def identifier: Int = override def manifest(o: AnyRef): String = o.getClass.getName
final val AddedManifest = classOf[Added].getName
final val SubtractedManifest = classOf[Subtracted].getName
final val AddedResultManifest = classOf[AddedResult].getName
final val SubtractedResultManifest = classOf[SubtractedResult].getName override def fromBinary(bytes: Array[Byte], manifest: String): AnyRef = { println("inside fromBinary"+manifest) manifest match {
case AddedManifest => Added.parseFrom(bytes)
case SubtractedManifest => Subtracted.parseFrom(bytes)
case AddedResultManifest => AddedResult.parseFrom(bytes)
case SubtractedResultManifest => SubtractedResult.parseFrom(bytes)
}
} override def toBinary(o: AnyRef): Array[Byte] = { println("inside toBinary ")
o match {
case a: Added => a.toByteArray
case s :Subtracted => s.toByteArray
case aR: AddedResult => aR.toByteArray
case sR: SubtractedResult => sR.toByteArray
}
}
}
然后我们需要在application.conf中告诉akka系统使用这些方法：
actor {
serializers {
proto = "akka.protobuf.serializer.ProtobufSerializer"
}
serialization-bindings {
"java.io.Serializable" = none
"com.google.protobuf.Message" = proto
"learn.proto.messages.Added" = proto
"learn.proto.messages.AddedResult" = proto
"learn.proto.messages.Subtracted" = proto
"learn.proto.messages.SubtractedResult" = proto
}
}
现在再重新运行：
[INFO] [// ::02.348] [calcSystem-akka.actor.default-dispatcher-] [akka.tcp://calcSystem@127.0.0.1:2551/user/calcRunner] Remote calculator started!
inside toBinary
inside fromBinarylearn.proto.messages.AddedResult
[INFO] [// ::03.234] [calcSystem-akka.actor.default-dispatcher-] [akka.tcp://calcSystem@127.0.0.1:2551/user/calcRunner] 18 + 38 = 56
inside toBinary
inside fromBinarylearn.proto.messages.AddedResult
[INFO] [// ::04.197] [calcSystem-akka.actor.default-dispatcher-] [akka.tcp://calcSystem@127.0.0.1:2551/user/calcRunner] 22 + 74 = 96
系统使用了自定义的ProtobufferSerializer。
下面是本次示范的完整源代码：
project/scalapb.sbt
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.18")
libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.7.1"
build.sbt
lazy val commonSettings = Seq(
name := "AkkaProtobufDemo",
version := "1.0",
scalaVersion := "2.12.6",
) lazy val local = (project in file("."))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11",
"com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"
),
name := "akka-protobuf-demo"
) lazy val remote = (project in file("remote"))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11"
),
name := "remote-system"
).dependsOn(local) PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)
resources/application.conf
akka {
actor {
provider = remote
}
remote {
netty.tcp {
hostname = "127.0.0.1"
}
}
actor {
serializers {
proto = "akka.protobuf.serializer.ProtobufSerializer"
}
serialization-bindings {
"java.io.Serializable" = none
"com.google.protobuf.Message" = proto
"learn.proto.messages.Added" = proto
"learn.proto.messages.AddedResult" = proto
"learn.proto.messages.Subtracted" = proto
"learn.proto.messages.SubtractedResult" = proto
}
}
}
main/protobuf/messages.proto
syntax = "proto3"; // Brought in from scalapb-runtime
import "scalapb/scalapb.proto";
import "google/protobuf/wrappers.proto"; package learn.proto; message Added { int32 nbr1 = 1;
int32 nbr2 = 2;
} message Subtracted {
int32 nbr1 = 1;
int32 nbr2 = 2;
} message AddedResult {
int32 nbr1 = 1;
int32 nbr2 = 2;
int32 result = 3;
} message SubtractedResult {
int32 nbr1 = 1;
int32 nbr2 = 2;
int32 result = 3;
}
remote/Calculator.scala
package akka.protobuf.calculator
import akka.actor._
import com.typesafe.config.ConfigFactory
import learn.proto.messages._ class Calculator extends Actor with ActorLogging { override def receive: Receive = {
case Added(a,b) =>
log.info("Calculating %d + %d".format(a, b))
sender() ! AddedResult(a,b,a+b)
case Subtracted(a,b) =>
log.info("Calculating %d - %d".format(a, b))
sender() ! SubtractedResult(a,b,a-b)
} } object Calculator {
def props = Props(new Calculator)
} object CalculatorStarter extends App { val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2552")
.withFallback(ConfigFactory.load()) val calcSystem = ActorSystem("calcSystem",config) calcSystem.actorOf(Calculator.props,"calculator") println("press any key to end program ...") scala.io.StdIn.readLine() calcSystem.terminate() }
CalcService.scala
package akka.protobuf.calcservice
import akka.actor._
import learn.proto.messages._
import scala.concurrent.duration._ class CalcRunner(path: String) extends Actor with ActorLogging {
sendIdentifyRequest() def sendIdentifyRequest(): Unit = {
context.actorSelection(path) ! Identify(path)
import context.dispatcher
context.system.scheduler.scheduleOnce(.seconds, self, ReceiveTimeout)
} def receive = identifying def identifying : Receive = {
case ActorIdentity(calcPath,Some(calcRef)) if (path.equals(calcPath)) =>
log.info("Remote calculator started!")
context.watch(calcRef)
context.become(calculating(calcRef))
case ActorIdentity(_,None) =>
log.info("Remote calculator not found!")
case ReceiveTimeout =>
sendIdentifyRequest()
case s @ _ =>
log.info(s"Remote calculator not ready. [$s]")
} def calculating(calculator: ActorRef) : Receive = {
case (op : Added) => calculator ! op
case (op : Subtracted) => calculator ! op case AddedResult(a,b,r) =>
log.info(s"$a + $b = $r")
case SubtractedResult(a,b,r) =>
log.info(s"$a - $b = $r") case Terminated(calculator) =>
log.info("Remote calculator terminated, restarting ...")
sendIdentifyRequest()
context.become(identifying) case ReceiveTimeout => //nothing
} } object CalcRunner {
def props(path: String) = Props(new CalcRunner(path))
}
Main.scala
package akka.protobuf.demo
import akka.actor._
import akka.util.Timeout
import com.typesafe.config.ConfigFactory
import akka.protobuf.calcservice._ import scala.concurrent.duration._
import scala.util._
import learn.proto.messages._ object Main extends App { val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2551")
.withFallback(ConfigFactory.load()) val calcSystem = ActorSystem("calcSystem",config) val calcPath = "akka.tcp://calcSystem@127.0.0.1:2552/user/calculator" val calculator = calcSystem.actorOf(CalcRunner.props(calcPath),"calcRunner") println("Calculator started ...") import calcSystem.dispatcher calcSystem.scheduler.schedule(.second, .second) {
if (Random.nextInt() % == )
calculator ! Added(Random.nextInt(), Random.nextInt())
else
calculator ! Subtracted(Random.nextInt(), Random.nextInt())
} scala.io.StdIn.readLine() }
ProtobufferSerializer.scala
package akka.protobuf.serializer import akka.serialization.SerializerWithStringManifest
import learn.proto.messages._ class ProtobufSerializer extends SerializerWithStringManifest{ def identifier: Int = override def manifest(o: AnyRef): String = o.getClass.getName
final val AddedManifest = classOf[Added].getName
final val SubtractedManifest = classOf[Subtracted].getName
final val AddedResultManifest = classOf[AddedResult].getName
final val SubtractedResultManifest = classOf[SubtractedResult].getName override def fromBinary(bytes: Array[Byte], manifest: String): AnyRef = { println("inside fromBinary"+manifest) manifest match {
case AddedManifest => Added.parseFrom(bytes)
case SubtractedManifest => Subtracted.parseFrom(bytes)
case AddedResultManifest => AddedResult.parseFrom(bytes)
case SubtractedResultManifest => SubtractedResult.parseFrom(bytes)
}
} override def toBinary(o: AnyRef): Array[Byte] = { println("inside toBinary ")
o match {
case a: Added => a.toByteArray
case s :Subtracted => s.toByteArray
case aR: AddedResult => aR.toByteArray
case sR: SubtractedResult => sR.toByteArray
}
}
}
ScalaPB（1）： using protobuf in akka的更多相关文章
- Mina、Netty、Twisted一起学（五）：整合protobuf
protobuf是谷歌的Protocol Buffers的简称,用于结构化数据和字节码之间互相转换(序列化.反序列化),一般应用于网络传输,可支持多种编程语言. protobuf如何使用这里不再介绍, ...
- ScalaPB（0）： 找寻合适的内部系统微服务集成工具
前一段时间我们探讨了SDP的一个基于集群的综合数据平台解决方案,由多种数据库组成,包括:JDBC, Cassandra 及MongoDB.其中Cassandra和MongoDB属于分布式数据库,可以在 ...
- ScalaPB（5）：用akka-stream实现reactive-gRPC
在前面几篇讨论里我们介绍了scala-gRPC的基本功能和使用方法,我们基本确定了选择gRPC作为一种有效的内部系统集成工具,主要因为下面gRPC支持的几种服务模式: .Unary-Call:独立 ...
- ScalaPB（4）： 通用跨系统protobuf数据，sbt设置
我们知道,在集群环境节点之间进行交换的数据必须经过序列化/反序列化处理过程,而在这方面protobuf是一个比较高效.易用的模式.用户首先在.proto文件中用IDL来定义系统中各种需要进行交换的数据 ...
- ScalaPB（2）： 在scala中用gRPC实现微服务
gRPC是google开源提供的一个RPC软件框架,它的特点是极大简化了传统RPC的开发流程和代码量,使用户可以免除许多陷阱并聚焦于实际应用逻辑中.作为一种google的最新RPC解决方案,gRPC具 ...
- ScalaPB（3）： gRPC streaming
接着上期讨论的gRPC unary服务我们跟着介绍gRPC streaming,包括: Server-Streaming, Client-Streaming及Bidirectional-Streami ...
- Akka系列（七）：Actor持久化之Akka persistence
前言.......... 我们在使用Akka时,会经常遇到一些存储Actor内部状态的场景,在系统正常运行的情况下,我们不需要担心什么,但是当系统出错,比如Actor错误需要重启,或者内存溢出,亦或者 ...
- Mina、Netty、Twisted一起学（八）：HTTP服务器
HTTP协议应该是目前使用最多的应用层协议了,用浏览器打开一个网站就是使用HTTP协议进行数据传输. HTTP协议也是基于TCP协议,所以也有服务器和客户端.HTTP客户端一般是浏览器,当然还有可能是 ...
- Mina、Netty、Twisted一起学（十）：线程模型
要想开发一个高性能的TCP服务器,熟悉所使用框架的线程模型非常重要.MINA.Netty.Twisted本身都是高性能的网络框架,如果再搭配上高效率的代码,才能实现一个高大上的服务器.但是如果不了解它 ...
随机推荐
- python/SQLAchemy
python/SQLAchemy SQLAlchemy是Python编程语言下的一款ORM框架,该框架建立在数据库API之上,使用关系对象映射进行数据库操作,简言之便是:将对象转换成SQL,然后使用数 ...
- RTKLIB源码解析（一）——单点定位(pntpos.c)
RTKLIB源码解析(一)--单点定位(pntpos.c) 标签: GNSS RTKLIB 单点定位 [TOC] pntpos int pntpos (const obsd_t *obs, int n ...
- 使用开源数据库客户端DBeaver连接DB2数据库
下载安装 首先进入 官网 选择对应的版本进行安装. 下载下来后,一直惦记next即可完成安装(期间包括选择文件安装路径等操作,可按需修改). 连接db2 打开DBeaver,新建连接-->DBe ...
- Python selenium 三种等待方式详解
1. 强制等待第一种也是最简单粗暴的一种办法就是强制等待sleep(xx),强制让闪电侠等xx时间,不管凹凸曼能不能跟上速度,还是已经提前到了,都必须等xx时间.看代码: # -*- coding: ...
- 列表（list）之三 -如何较为均匀的将任意字符串按指定组数分组，方差最少
当字符串的长度不是份数的整数倍时如何均匀地分割,例如:长度为11的字符串要分割成4份,有很多种分法,比如3, 3, 3, 2(前3个字符一份,中间3个一份,再中间3个一份,最后2个一份)这种是比较均匀 ...
- web攻击和防御措施
1.SQL注入:参照下面的链接 http://www.cnblogs.com/chenhaoyu/p/8758888.html 2.跨网站脚本攻击(Cross Site Scripting, XSS) ...
- kafka知识体系
最近一直在整理kafka相关资料,以构建自己的知识体系. 主要分为五大方面: Kafka设计与原理分析 Kafka配置分析 Kafka运维手册 Kafka编程开发 kafka源码分析
- [LeetCode] Subarray Product Less Than K 子数组乘积小于K
Your are given an array of positive integers nums. Count and print the number of (contiguous) subarr ...
- for-each的坑(Hollis)
直接用代码来说明: public class ForEach { public static void main(String[] args) { List<String> list = ...
- ●BZOJ 3270 博物馆
题链: http://www.lydsy.com/JudgeOnline/problem.php?id=3270题解: 期望DP,高斯消元 本来是定义的关于概率的dp, 但是发现这样定义有很多解释不通 ...