akka-stream原则上是一种推式(push-model)的数据流。push-model和pull-model的区别在于它们解决问题倾向性:push模式面向高效的数据流下游(fast-downstream-subscriber),pull model倾向高效的上游(fast-upstream-publisher)。现实中速度同等的上下游并不多见,不匹配的上下游速度最终造成数据丢失。如果下游的subscriber无法及时接收由publisher向下游推送的全部数据,那么无论有多大的缓冲区,最终会造成溢出丢失数据。如果上游的publisher无法及时满足下游subscriber的数据读取需求会加长下游的等待状态造成超时甚至会使遗失下游请求遗失。对于akka-stream这种push模式的数据流,因为超速推送数据会造成数据丢失,所以必须想办法控制publisher产生数据的速度。因为akka-stream已经在上下游环节全部实现了Reactive-Streams-Specification,所以上下游之间可以进行互动,这样就可以在akka-stream里由下游通知上游自身可接收数据的状态来控制上游数据流速,即所谓的压力缓冲backpressure了。akka-stream的backpressure使用了缓冲区buffer来成批预存及补充数据,这样可以提高数据传输效率。另外,如果用async进行数据流的并行运算的话上游就不必理会下游反应,可以把数据推进buffer然后立即继续处理下一个数据元素。所以async运算模式的buffering就不可或缺了。akka-stream可以通过以下几种方式来设定异步运算使用的缓冲大小:

1、在配置文件中设定默认buffer:

akka.stream.materializer.max-input-buffer-size = 

2、在ActorMaterializerSetting中宏观层面上设定:

val materializer = ActorMaterializer(
ActorMaterializerSettings(system)
.withInputBuffer(
initialSize = ,
maxSize = ))

3、通过Attribute属性设定。因为Atrribute保持了层级关系,所以通过Attribute设定的inputbuffer也延续了属性继承:

import Attributes._
val nestedSource =
Source.single()
.map(_ + )
.named("nestedSource") // Wrap, no inputBuffer set val nestedFlow =
Flow[Int].filter(_ != )
.via(Flow[Int].map(_ - ).withAttributes(inputBuffer(, ))) // override
.named("nestedFlow") // Wrap, no inputBuffer set val nestedSink =
nestedFlow.to(Sink.fold()(_ + _)) // wire an atomic sink to the nestedFlow
.withAttributes(name("nestedSink") and inputBuffer(, )) // override

在上面的示例里nestdSource继承了Materializer全局inputBuffer属性;nestedSink重写了属性;nestedFlow先是继承了nestedSink的设定然后又重写了自己的inputBuffer属性。我们可以用addAttribute来新添加Attribute:

  val flow = Flow[Int].map(_ * ).async.addAttributes(Attributes.inputBuffer(,))
val (_,fut) = flow.runWith(Source( to ),Sink.foreach(println))
fut.andThen{case _ => sys.terminate()}

上面定义这些inputBuffer包括了起始值和最大值,主要应用在backpressure。所以,理论上inputBuffer可以设成一个字节(initial=1,max=1),因为有了backpressure就不用担心数据溢出,但这样会影响数据流传输效率。所以akka-stream默认的缓冲区长度为16字节。所以aka-stream的backpressure是batching backpressure。

由于akka-stream是push模式的,我们还可以用buffer来控制包括Source,Flow这些上游环节推送的数据:

  val source = Source( to ).buffer(,OverflowStrategy.dropTail)
val sum = source.runFold()((acc,i) => i + acc)
sum.map(println) //.andThen{case _ => sys.terminate()} val flow = Flow[Int].map(_ * ).buffer(,OverflowStrategy.dropNew)
val (_,fut) = flow.runWith(Source( to ),Sink.fold(){(acc,a) => acc + a})
fut.map(println).andThen{case _ => sys.terminate()}

上游所设buffer对publisher过快产生的数据可以采用溢出处理策略OverflowStrategy。上面用Attribute添加的inputBuffer默认了OverflowStrategy.backpressure,其它OverflowStrategy选项如下:

object OverflowStrategy {
/**
* If the buffer is full when a new element arrives, drops the oldest element from the buffer to make space for
* the new element.
*/
def dropHead: OverflowStrategy = DropHead /**
* If the buffer is full when a new element arrives, drops the youngest element from the buffer to make space for
* the new element.
*/
def dropTail: OverflowStrategy = DropTail /**
* If the buffer is full when a new element arrives, drops all the buffered elements to make space for the new element.
*/
def dropBuffer: OverflowStrategy = DropBuffer /**
* If the buffer is full when a new element arrives, drops the new element.
*/
def dropNew: OverflowStrategy = DropNew /**
* If the buffer is full when a new element is available this strategy backpressures the upstream publisher until
* space becomes available in the buffer.
*/
def backpressure: OverflowStrategy = Backpressure /**
* If the buffer is full when a new element is available this strategy completes the stream with failure.
*/
def fail: OverflowStrategy = Fail
}

当akka-stream需要与外界系统进行数据交换时就无法避免数据流上下游速率不匹配的问题了。如果外界系统不支持Reactive-Stream标准,就会发生数据丢失现象。对此akka-stream提供了具体的解决方法:如果外界系统是在上游过快产生数据可以用conflate函数用Seq这样的集合把数据传到下游。如果下游能及时读取则Seq(Item)中的Item正是上游推送的数据元素,否则Seq(i1,i2,i3...)就代表上游在下游再次读取时间段内产生的数据。因为Seq可以是无限大,所以理论上可以避免数据丢失。下面是这个函数的定义:

 /**
* Allows a faster upstream to progress independently of a slower subscriber by conflating elements into a summary
* until the subscriber is ready to accept them. For example a conflate step might average incoming numbers if the
* upstream publisher is faster.
*
* This version of conflate allows to derive a seed from the first element and change the aggregated type to be
* different than the input type. See [[FlowOps.conflate]] for a simpler version that does not change types.
*
* This element only rolls up elements if the upstream is faster, but if the downstream is faster it will not
* duplicate elements.
*
* Adheres to the [[ActorAttributes.SupervisionStrategy]] attribute.
*
* '''Emits when''' downstream stops backpressuring and there is a conflated element available
*
* '''Backpressures when''' never
*
* '''Completes when''' upstream completes
*
* '''Cancels when''' downstream cancels
*
* @param seed Provides the first state for a conflated value using the first unconsumed element as a start
* @param aggregate Takes the currently aggregated value and the current pending element to produce a new aggregate
*
* See also [[FlowOps.conflate]], [[FlowOps.limit]], [[FlowOps.limitWeighted]] [[FlowOps.batch]] [[FlowOps.batchWeighted]]
*/
def conflateWithSeed[S](seed: Out ⇒ S)(aggregate: (S, Out) ⇒ S): Repr[S] =
via(Batch(1L, ConstantFun.zeroLong, seed, aggregate).withAttributes(DefaultAttributes.conflate))

下面是conflateWithSeed函数用例:

import akka.actor._
import akka.stream._
import akka.stream.scaladsl._
import scala.concurrent.duration._
object StreamDemo1 extends App { implicit val sys = ActorSystem("streamSys")
implicit val ec = sys.dispatcher
implicit val mat = ActorMaterializer(
ActorMaterializerSettings(sys)
.withInputBuffer(,)
) case class Tick() RunnableGraph.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._ // this is the asynchronous stage in this graph
val zipper = b.add(ZipWith[Tick, Seq[String], Seq[String]]((tick, count) => count).async)
// this slows down the pipeline by 3 seconds
Source.tick(initialDelay = .seconds, interval = .seconds, Tick()) ~> zipper.in0
// faster producer with all elements passed inside a Seq
Source.tick(initialDelay = .second, interval = .second, "item")
.conflateWithSeed(Seq(_)) { (acc,elem) => acc :+ elem } ~> zipper.in1 zipper.out ~> Sink.foreach(println)
ClosedShape
}).run() }

在上面这个例子里我们用ZipWith其中一个低速的输入端来控制整个管道的速率。这时我们会发现输出端Seq长度代表ZipWith消耗数据的延迟间隔。注意:前面3个输出好像没有延迟,这是akka-stream 预读prefetch造成的。因为我们设定了InputBuffer(Initial=1,max=1),第一个数据被预读当作及时消耗了。

如果没有实现Reactive-Stream标准的外界系统上游producer速率过慢,有可能造成下游超时,akka-stream提供了expand函数来解决这个问题:

 /**
* Allows a faster downstream to progress independently of a slower publisher by extrapolating elements from an older
* element until new element comes from the upstream. For example an expand step might repeat the last element for
* the subscriber until it receives an update from upstream.
*
* This element will never "drop" upstream elements as all elements go through at least one extrapolation step.
* This means that if the upstream is actually faster than the upstream it will be backpressured by the downstream
* subscriber.
*
* Expand does not support [[akka.stream.Supervision.Restart]] and [[akka.stream.Supervision.Resume]].
* Exceptions from the `seed` or `extrapolate` functions will complete the stream with failure.
*
* '''Emits when''' downstream stops backpressuring
*
* '''Backpressures when''' downstream backpressures or iterator runs empty
*
* '''Completes when''' upstream completes
*
* '''Cancels when''' downstream cancels
*
* @param seed Provides the first state for extrapolation using the first unconsumed element
* @param extrapolate Takes the current extrapolation state to produce an output element and the next extrapolation
* state.
*/
def expand[U](extrapolate: Out ⇒ Iterator[U]): Repr[U] = via(new Expand(extrapolate))

当上游无法及时发送下游请求的数据时我们可以用expand推送一个固定的数据元素来临时满足下游的要求:

 val lastFlow = Flow[Double]
.expand(Iterator.continually(_))

Akka(20): Stream:压力缓冲-Batching backpressure and buffering的更多相关文章

  1. Akka(20): Stream:异步运算,压力缓冲-Async, batching backpressure and buffering

    akka-stream原则上是一种推式(push-model)的数据流.push-model和pull-model的区别在于它们解决问题倾向性:push模式面向高效的数据流下游(fast-downst ...

  2. Java入门 - 语言基础 - 20.Stream和File和IO

    原文地址:http://www.work100.net/training/java-stream-file-io.html 更多教程:光束云 - 免费课程 Stream和File和IO 序号 文内章节 ...

  3. 双缓冲技术(Double Buffering)(1、简介和源代码部分)

    这一节实在是有些长,翻译完后统计了一下,快到2w字了.考虑到阅读的方便和网络的速度,打算把这节分为5个部分,第一部分为双缓冲技术的一个 简介和所有的代码,如果能够看懂代码,不用看译文也就可以了.第二部 ...

  4. Akka Stream文档翻译:Quick Start Guide: Reactive Tweets

    Quick Start Guide: Reactive Tweets 快速入门指南: Reactive Tweets (reactive tweets 大概可以理解为“响应式推文”,在此可以测试下GF ...

  5. Akka(25): Stream:对接外部系统-Integration

    在现实应用中akka-stream往往需要集成其它的外部系统形成完整的应用.这些外部系统可能是akka系列系统或者其它类型的系统.所以,akka-stream必须提供一些函数和方法来实现与各种不同类型 ...

  6. C语言 流缓冲 Stream Buffering

    From : https://www.gnu.org/software/libc/manual/html_node/Stream-Buffering.html 译者:李秋豪 12.20 流缓冲 通常情 ...

  7. C语言 流缓冲

    **From : https://www.gnu.org/software/libc/manual/html_node/Stream-Buffering.html** 12.20 流缓冲 通常情况下, ...

  8. Stream初步认识(一)

    Stream初步认识(一)测试 简介 Stream 是 Java8 中处理集合的关键抽象概念,它可以指定你希望对 集合进行的操作,可以执行非常复杂的查找.过滤和映射数据等操作. 使用Stream AP ...

  9. java8之stream

    lambda表达式是stream的基础,初学者建议先学习lambda表达式,http://www.cnblogs.com/andywithu/p/7357069.html 1.初识stream 先来一 ...

随机推荐

  1. js&jquery跨域详解jsonp,jquery并发大量请求丢失回调bug

    URL  说明 是否允许通信 http://www.a.com/a.js http://www.a.com/b.js 同一域名下 允许 http://www.a.com/lab/a.js http:/ ...

  2. 基于 WebRTC 技术的实时通信服务开发实践

    随着直播的发展,直播实时互动性变得日益重要.又拍云在 WebRTC 的基础上,凭借多年的开发经验,结合当下实际情况,开发 UPRTC 系统,解决了网络延时.并发量大.客户端解码能力差等问题. WebR ...

  3. JMeter 监控和记录&常用功能

    使用https连接时,如果对应站点的CA 证书错误,会直接报连接不到服务器的错误,org.apache.commons.httpclient.NoHttpResponseException,把错误证书 ...

  4. (转)mybatis常用jdbcType数据类型

    1 MyBatis 通过包含的jdbcType类型 BIT FLOAT CHAR TIMESTAMP OTHER UNDEFINED TINYINT REAL VARCHAR BINARY BLOB ...

  5. CRM权限管理

    CRM权限管理 一.概念 权限管理就是管理用户对于资源的操作.本 CRM 系统的权限(也称作资源)是基于角色操作权限来实现的,即RBAC(Role-Based Access Control,基于角色的 ...

  6. 64位linux系统通过编译安装apache+…

    二.安装php 上传php压缩包 例如:php-5.2.3.tar.gz 移动 mv php-5.2.3.tar.gz /usr/local/src 进入 cd /usr/local/src 解压 t ...

  7. 转:SpringMVC浅谈

    因为项目文案需要,于是乎翻阅spring相关资料.顿觉该篇不错详尽易懂,特转载之. 转载出处: http://blog.csdn.net/gane_cheng/article/details/5278 ...

  8. 【python密码学编程】8.使用换位加密法加密

    替代加密法:用其他字符替代原有字符 换位加密法:搞乱字符顺序 [换位加密法]需要一个密钥 仅允许非商业转载,转载请注明出处

  9. MarkDown入门指南

    标题 标题是每篇文章必备而且最常用的格式. 在Markdown中,如果想将一段文字定义为标题,只需要在这段文字前面加上 #,再在 # 后加一个空格即可.还可增加二.三.四.五.六级标题,总共六级,只需 ...

  10. android学习ViewPager的简单使用

    使用ViewPager需要引入android.support.v4.View.ViewPager这样的jar包,谷歌公司为解决当前版本碎片化的问题,提供的兼容的包.主要目的就是解决向下兼容问题. 1, ...