ScalaPB（1）： using protobuf in akka
任何类型的实例作为消息在两端独立系统的机器之间进行传递时必须经过序列化/反序列化serialize/deserialize处理过程。假设以下场景：在一个网络里有两台连接的服务器，它们分别部署了独立的akka系统。如果我们需要在这两台服务器的akka系统之间进行消息交换的话，所有消息都必须经过序列化/反序列化处理。akka系统对于用户自定义消息类型的默认序列化处理是以java-object serialization 方式进行的。我们上次提过：由于java-object-serialization会把一个java-object的类型信息、实例值、它所包含的其它类型描述信息等都写入序列化的结果里，所以会占据较大空间，传输数据的效率相对就低了。protobuf是binary格式的，基本只包括实例值，所以数据传输效率较高。下面我们就介绍如何在akka系统中使用protobuf序列化。在akka中使用自定义序列化方法包括下面的这些步骤：
1、在.proto文件中对消息类型进行IDL定义
2、用ScalaPB编译IDL文件并产生scala源代码。这些源代码中包括了涉及的消息类型及它们的操作方法
3、在akka程序模块中import产生的classes，然后直接调用这些类型和方法
4、按akka要求编写序列化方法
5、在akka的.conf文件里actor.serializers段落中定义akka的默认serializer
下面的build.sbt文件里描述了程序结构：
lazy val commonSettings = Seq(
name := "AkkaProtobufDemo",
version := "1.0",
scalaVersion := "2.12.6",
) lazy val local = (project in file("."))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11",
"com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"
),
name := "akka-protobuf-demo"
) lazy val remote = (project in file("remote"))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11"
),
name := "remote-system"
).dependsOn(local) PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)
local和remote是两个分开的项目。我们会在这两个项目里分别部署akka系统。注意依赖项中的scalapb.runtime。PB.targets指明了产生源代码的路径。我们还需要在project/scalapb.sbt中指定scalaPB插件：
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.18")
libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.7.1"
我们首先在.proto文件里定义消息：
syntax = "proto3"; // Brought in from scalapb-runtime
import "scalapb/scalapb.proto";
import "google/protobuf/wrappers.proto"; package learn.proto; message Added { int32 nbr1 = ;
int32 nbr2 = ;
} message Subtracted {
int32 nbr1 = ;
int32 nbr2 = ;
} message AddedResult {
int32 nbr1 = ;
int32 nbr2 = ;
int32 result = ;
} message SubtractedResult {
int32 nbr1 = ;
int32 nbr2 = ;
int32 result = ;
}
现在我们先在remote项目里定义一个Calculator actor：
package akka.protobuf.calculator
import akka.actor._
import com.typesafe.config.ConfigFactory
import learn.proto.messages._ class Calculator extends Actor with ActorLogging { override def receive: Receive = {
case Added(a,b) =>
log.info("Calculating %d + %d".format(a, b))
sender() ! AddedResult(a,b,a+b)
case Subtracted(a,b) =>
log.info("Calculating %d - %d".format(a, b))
sender() ! SubtractedResult(a,b,a-b)
} } object Calculator {
def props = Props(new Calculator)
} object CalculatorStarter extends App { val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2552")
.withFallback(ConfigFactory.load()) val calcSystem = ActorSystem("calcSystem",config) calcSystem.actorOf(Calculator.props,"calculator") println("press any key to end program ...") scala.io.StdIn.readLine() calcSystem.terminate() }
运行CalculatorStarter产生一个calculator actor: akka.tcp://calcSystem@127.0.0.1:2552/user/calculator
下面我们在local项目里从端口2551上部署另一个akka系统，然后调用端口2552上部署akka系统的calculator actor：
package akka.protobuf.calcservice
import akka.actor._
import learn.proto.messages._
import scala.concurrent.duration._ class CalcRunner(path: String) extends Actor with ActorLogging {
sendIdentifyRequest() def sendIdentifyRequest(): Unit = {
context.actorSelection(path) ! Identify(path)
import context.dispatcher
context.system.scheduler.scheduleOnce(.seconds, self, ReceiveTimeout)
} def receive = identifying def identifying : Receive = {
case ActorIdentity(calcPath,Some(calcRef)) if (path.equals(calcPath)) =>
log.info("Remote calculator started!")
context.watch(calcRef)
context.become(calculating(calcRef))
case ActorIdentity(_,None) =>
log.info("Remote calculator not found!")
case ReceiveTimeout =>
sendIdentifyRequest()
case s @ _ =>
log.info(s"Remote calculator not ready. [$s]")
} def calculating(calculator: ActorRef) : Receive = {
case (op : Added) => calculator ! op
case (op : Subtracted) => calculator ! op case AddedResult(a,b,r) =>
log.info(s"$a + $b = $r")
case SubtractedResult(a,b,r) =>
log.info(s"$a - $b = $r") case Terminated(calculator) =>
log.info("Remote calculator terminated, restarting ...")
sendIdentifyRequest()
context.become(identifying) case ReceiveTimeout => //nothing
} } object CalcRunner {
def props(path: String) = Props(new CalcRunner(path))
}
这个CalcRunner是一个actor，在程序里首先通过向remote项目中的calculator-actor传送Identify消息以取得具体的ActorRef。然后用这个ActorRef与calculator-actor进行交互。这其中Identify是akka预定消息类型，其它消息都是ScalaPB从.proto文件中产生的。下面是local项目的运算程序：
package akka.protobuf.demo
import akka.actor._
import akka.util.Timeout
import com.typesafe.config.ConfigFactory
import akka.protobuf.calcservice._ import scala.concurrent.duration._
import scala.util._
import learn.proto.messages._ object Main extends App { val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2551")
.withFallback(ConfigFactory.load()) val calcSystem = ActorSystem("calcSystem",config) val calcPath = "akka.tcp://calcSystem@127.0.0.1:2552/user/calculator" val calculator = calcSystem.actorOf(CalcRunner.props(calcPath),"calcRunner") println("Calculator started ...") import calcSystem.dispatcher calcSystem.scheduler.schedule(.second, .second) {
if (Random.nextInt() % == )
calculator ! Added(Random.nextInt(), Random.nextInt())
else
calculator ! Subtracted(Random.nextInt(), Random.nextInt())
} scala.io.StdIn.readLine() }
配置文件application.conf：
akka {
actor {
provider = remote
}
remote {
netty.tcp {
hostname = "127.0.0.1"
}
}
}
先运行remote然后local。注意下面出现的提示：
[akka.serialization.Serialization(akka://calcSystem)] Using the default Java serializer for class [learn.proto.messages.Added] which is not recommended because of performance implications. Use another serializer
下面是protobuf类型的序列化方法：
package akka.protobuf.serializer import akka.serialization.SerializerWithStringManifest
import learn.proto.messages._ class ProtobufSerializer extends SerializerWithStringManifest{ def identifier: Int = override def manifest(o: AnyRef): String = o.getClass.getName
final val AddedManifest = classOf[Added].getName
final val SubtractedManifest = classOf[Subtracted].getName
final val AddedResultManifest = classOf[AddedResult].getName
final val SubtractedResultManifest = classOf[SubtractedResult].getName override def fromBinary(bytes: Array[Byte], manifest: String): AnyRef = { println("inside fromBinary"+manifest) manifest match {
case AddedManifest => Added.parseFrom(bytes)
case SubtractedManifest => Subtracted.parseFrom(bytes)
case AddedResultManifest => AddedResult.parseFrom(bytes)
case SubtractedResultManifest => SubtractedResult.parseFrom(bytes)
}
} override def toBinary(o: AnyRef): Array[Byte] = { println("inside toBinary ")
o match {
case a: Added => a.toByteArray
case s :Subtracted => s.toByteArray
case aR: AddedResult => aR.toByteArray
case sR: SubtractedResult => sR.toByteArray
}
}
}
然后我们需要在application.conf中告诉akka系统使用这些方法：
actor {
serializers {
proto = "akka.protobuf.serializer.ProtobufSerializer"
}
serialization-bindings {
"java.io.Serializable" = none
"com.google.protobuf.Message" = proto
"learn.proto.messages.Added" = proto
"learn.proto.messages.AddedResult" = proto
"learn.proto.messages.Subtracted" = proto
"learn.proto.messages.SubtractedResult" = proto
}
}
现在再重新运行：
[INFO] [// ::02.348] [calcSystem-akka.actor.default-dispatcher-] [akka.tcp://calcSystem@127.0.0.1:2551/user/calcRunner] Remote calculator started!
inside toBinary
inside fromBinarylearn.proto.messages.AddedResult
[INFO] [// ::03.234] [calcSystem-akka.actor.default-dispatcher-] [akka.tcp://calcSystem@127.0.0.1:2551/user/calcRunner] 18 + 38 = 56
inside toBinary
inside fromBinarylearn.proto.messages.AddedResult
[INFO] [// ::04.197] [calcSystem-akka.actor.default-dispatcher-] [akka.tcp://calcSystem@127.0.0.1:2551/user/calcRunner] 22 + 74 = 96
系统使用了自定义的ProtobufferSerializer。
下面是本次示范的完整源代码：
project/scalapb.sbt
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.18")
libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.7.1"
build.sbt
lazy val commonSettings = Seq(
name := "AkkaProtobufDemo",
version := "1.0",
scalaVersion := "2.12.6",
) lazy val local = (project in file("."))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11",
"com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"
),
name := "akka-protobuf-demo"
) lazy val remote = (project in file("remote"))
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-remote" % "2.5.11"
),
name := "remote-system"
).dependsOn(local) PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)
resources/application.conf
akka {
actor {
provider = remote
}
remote {
netty.tcp {
hostname = "127.0.0.1"
}
}
actor {
serializers {
proto = "akka.protobuf.serializer.ProtobufSerializer"
}
serialization-bindings {
"java.io.Serializable" = none
"com.google.protobuf.Message" = proto
"learn.proto.messages.Added" = proto
"learn.proto.messages.AddedResult" = proto
"learn.proto.messages.Subtracted" = proto
"learn.proto.messages.SubtractedResult" = proto
}
}
}
main/protobuf/messages.proto
syntax = "proto3"; // Brought in from scalapb-runtime
import "scalapb/scalapb.proto";
import "google/protobuf/wrappers.proto"; package learn.proto; message Added { int32 nbr1 = 1;
int32 nbr2 = 2;
} message Subtracted {
int32 nbr1 = 1;
int32 nbr2 = 2;
} message AddedResult {
int32 nbr1 = 1;
int32 nbr2 = 2;
int32 result = 3;
} message SubtractedResult {
int32 nbr1 = 1;
int32 nbr2 = 2;
int32 result = 3;
}
remote/Calculator.scala
package akka.protobuf.calculator
import akka.actor._
import com.typesafe.config.ConfigFactory
import learn.proto.messages._ class Calculator extends Actor with ActorLogging { override def receive: Receive = {
case Added(a,b) =>
log.info("Calculating %d + %d".format(a, b))
sender() ! AddedResult(a,b,a+b)
case Subtracted(a,b) =>
log.info("Calculating %d - %d".format(a, b))
sender() ! SubtractedResult(a,b,a-b)
} } object Calculator {
def props = Props(new Calculator)
} object CalculatorStarter extends App { val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2552")
.withFallback(ConfigFactory.load()) val calcSystem = ActorSystem("calcSystem",config) calcSystem.actorOf(Calculator.props,"calculator") println("press any key to end program ...") scala.io.StdIn.readLine() calcSystem.terminate() }
CalcService.scala
package akka.protobuf.calcservice
import akka.actor._
import learn.proto.messages._
import scala.concurrent.duration._ class CalcRunner(path: String) extends Actor with ActorLogging {
sendIdentifyRequest() def sendIdentifyRequest(): Unit = {
context.actorSelection(path) ! Identify(path)
import context.dispatcher
context.system.scheduler.scheduleOnce(.seconds, self, ReceiveTimeout)
} def receive = identifying def identifying : Receive = {
case ActorIdentity(calcPath,Some(calcRef)) if (path.equals(calcPath)) =>
log.info("Remote calculator started!")
context.watch(calcRef)
context.become(calculating(calcRef))
case ActorIdentity(_,None) =>
log.info("Remote calculator not found!")
case ReceiveTimeout =>
sendIdentifyRequest()
case s @ _ =>
log.info(s"Remote calculator not ready. [$s]")
} def calculating(calculator: ActorRef) : Receive = {
case (op : Added) => calculator ! op
case (op : Subtracted) => calculator ! op case AddedResult(a,b,r) =>
log.info(s"$a + $b = $r")
case SubtractedResult(a,b,r) =>
log.info(s"$a - $b = $r") case Terminated(calculator) =>
log.info("Remote calculator terminated, restarting ...")
sendIdentifyRequest()
context.become(identifying) case ReceiveTimeout => //nothing
} } object CalcRunner {
def props(path: String) = Props(new CalcRunner(path))
}
Main.scala
package akka.protobuf.demo
import akka.actor._
import akka.util.Timeout
import com.typesafe.config.ConfigFactory
import akka.protobuf.calcservice._ import scala.concurrent.duration._
import scala.util._
import learn.proto.messages._ object Main extends App { val config = ConfigFactory.parseString("akka.remote.netty.tcp.port=2551")
.withFallback(ConfigFactory.load()) val calcSystem = ActorSystem("calcSystem",config) val calcPath = "akka.tcp://calcSystem@127.0.0.1:2552/user/calculator" val calculator = calcSystem.actorOf(CalcRunner.props(calcPath),"calcRunner") println("Calculator started ...") import calcSystem.dispatcher calcSystem.scheduler.schedule(.second, .second) {
if (Random.nextInt() % == )
calculator ! Added(Random.nextInt(), Random.nextInt())
else
calculator ! Subtracted(Random.nextInt(), Random.nextInt())
} scala.io.StdIn.readLine() }
ProtobufferSerializer.scala
package akka.protobuf.serializer import akka.serialization.SerializerWithStringManifest
import learn.proto.messages._ class ProtobufSerializer extends SerializerWithStringManifest{ def identifier: Int = override def manifest(o: AnyRef): String = o.getClass.getName
final val AddedManifest = classOf[Added].getName
final val SubtractedManifest = classOf[Subtracted].getName
final val AddedResultManifest = classOf[AddedResult].getName
final val SubtractedResultManifest = classOf[SubtractedResult].getName override def fromBinary(bytes: Array[Byte], manifest: String): AnyRef = { println("inside fromBinary"+manifest) manifest match {
case AddedManifest => Added.parseFrom(bytes)
case SubtractedManifest => Subtracted.parseFrom(bytes)
case AddedResultManifest => AddedResult.parseFrom(bytes)
case SubtractedResultManifest => SubtractedResult.parseFrom(bytes)
}
} override def toBinary(o: AnyRef): Array[Byte] = { println("inside toBinary ")
o match {
case a: Added => a.toByteArray
case s :Subtracted => s.toByteArray
case aR: AddedResult => aR.toByteArray
case sR: SubtractedResult => sR.toByteArray
}
}
}
ScalaPB（1）： using protobuf in akka的更多相关文章
- Mina、Netty、Twisted一起学（五）：整合protobuf
protobuf是谷歌的Protocol Buffers的简称,用于结构化数据和字节码之间互相转换(序列化.反序列化),一般应用于网络传输,可支持多种编程语言. protobuf如何使用这里不再介绍, ...
- ScalaPB（0）： 找寻合适的内部系统微服务集成工具
前一段时间我们探讨了SDP的一个基于集群的综合数据平台解决方案,由多种数据库组成,包括:JDBC, Cassandra 及MongoDB.其中Cassandra和MongoDB属于分布式数据库,可以在 ...
- ScalaPB（5）：用akka-stream实现reactive-gRPC
在前面几篇讨论里我们介绍了scala-gRPC的基本功能和使用方法,我们基本确定了选择gRPC作为一种有效的内部系统集成工具,主要因为下面gRPC支持的几种服务模式: .Unary-Call:独立 ...
- ScalaPB（4）： 通用跨系统protobuf数据，sbt设置
我们知道,在集群环境节点之间进行交换的数据必须经过序列化/反序列化处理过程,而在这方面protobuf是一个比较高效.易用的模式.用户首先在.proto文件中用IDL来定义系统中各种需要进行交换的数据 ...
- ScalaPB（2）： 在scala中用gRPC实现微服务
gRPC是google开源提供的一个RPC软件框架,它的特点是极大简化了传统RPC的开发流程和代码量,使用户可以免除许多陷阱并聚焦于实际应用逻辑中.作为一种google的最新RPC解决方案,gRPC具 ...
- ScalaPB（3）： gRPC streaming
接着上期讨论的gRPC unary服务我们跟着介绍gRPC streaming,包括: Server-Streaming, Client-Streaming及Bidirectional-Streami ...
- Akka系列（七）：Actor持久化之Akka persistence
前言.......... 我们在使用Akka时,会经常遇到一些存储Actor内部状态的场景,在系统正常运行的情况下,我们不需要担心什么,但是当系统出错,比如Actor错误需要重启,或者内存溢出,亦或者 ...
- Mina、Netty、Twisted一起学（八）：HTTP服务器
HTTP协议应该是目前使用最多的应用层协议了,用浏览器打开一个网站就是使用HTTP协议进行数据传输. HTTP协议也是基于TCP协议,所以也有服务器和客户端.HTTP客户端一般是浏览器,当然还有可能是 ...
- Mina、Netty、Twisted一起学（十）：线程模型
要想开发一个高性能的TCP服务器,熟悉所使用框架的线程模型非常重要.MINA.Netty.Twisted本身都是高性能的网络框架,如果再搭配上高效率的代码,才能实现一个高大上的服务器.但是如果不了解它 ...
随机推荐
- Python 爬虫性能相关
性能相关 在编写爬虫时,性能的消耗主要在IO请求中,当单进程单线程模式下请求URL时必然会引起等待,从而使得请求整体变慢. import requests def fetch_async(url): ...
- Python生成随机验证码
Python生成随机验证码,需要使用PIL模块. 安装: pip3 install pillow 基本使用 1.创建图片 from PIL import Image img = Image.new(m ...
- monogodb3.4安装修改，权限设置
下载地址:https://www.mongodb.com/download-center#community 这里的方法只对应3.4,别的有没有效果请自行判断. 下载后按默认下一步. 默认安装地址 ...
- scrapy中的response
初始化参数 class scrapy.http.Response( url[, status=200, headers, body, flags ] ) 其他成员 url status headers ...
- PHP实现统计在线人数功能示例
本文实例讲述了PHP实现统计在线人数的方法.分享给大家供大家参考,具体如下: 我记得ASP里面统计在线人数用application 这个对象就可以了.PHP怎么设计? PHP对session对象的封装 ...
- MySQL高可用架构之MHA 原理与实践
MHA简介 关于MHA MHA(Master HA)是一款开源的MySQL的高可用程序,它为MySQL主从复制架构提供了automating master failover 功能.MHA在监控到mas ...
- 列表(list)之一定义 添加 删除 排序 反转 索引等其他操作
1.定义: 创建一个列表,只要把逗号分隔的不同的数据项使用方括号括起来即可,序列中的每个元素都分配一个数字 - 它的位置,或索引,第一个索引是0,第二个索引是1,依此类推. # 列表list1=[&q ...
- Headless Chrome：服务端渲染JS站点的一个方案【中篇】【翻译】
接上篇 防止重新渲染 其实说不对客户端代码做任何修改是忽悠人的.在我们的Express 应用中,通过Puppteer加载页面,提供给客户端响应,但是这个过程是有一些问题的. js脚本在服务端的Head ...
- NoClassDefFoundError: org/apache/commons/lang3/StringUtils
出错信息: 2014-2-5 21:38:05 org.apache.catalina.core.StandardContext filterStart严重: Exception starting f ...
- SQL学习之查询
查询实例: 1.创建数据库并使用: create database school; use school; 2.创建表并插入内容: create table student( Sno char(9) ...