Akka Stream connect to multiple sinks

问题

I have implemented a custom component in akka stream which takes elements as input, groups and merges them based on a key and sends them out through one of a dozen outlets. You can think of this component as a kind of GroupBy component which does not partition the flow into subflows, but actual flows. In addition to partitioning incoming elements, it merges them into one element, i.e. there is some buffering happening inside the component such that 1 element in does not necessarily mean 1 element out through an outlet.

Below is a simplified implementation of said component.

class CustomGroupBy[A,B](k: Int, f: A => Int) extends GraphStage[FlowShape[B, B]] {

  val in = Inlet[A]("CustomGroupBy.in")
  val outs = (0 until k).map(i => Outlet[B](s"CustomGroupBy.$i.out"))

  override val shape = new AmorphousShape(scala.collection.immutable.Seq(in), outs)

  /* ... */
}

I now what to connect each outlet of that component to a different Sink and combine the materialized value of all these sinks.

I have tried a few things with the graph DSL, but have not quite managed to get it working. Would anyone be so kind as to provide me with a snippet to do that or point me in the right direction?

Thanks in advance!

回答1:

You most likely want the built-in broadcast stage. Example usage can be found here:

val bcast = builder.add(Broadcast[Int](2))

in ~> f1 ~> bcast ~> f2 ~> merge ~> f3 ~> out
            bcast ~> f4 ~> merge

回答2:

You probably want the akka.stream.scaladsl.Partition[T](outputPorts: Int, partitioner: T ⇒ Int) stage.

EDIT:

To connect all the ports, and keep the materialized values, you have to give your stages as parameters to the GraphDSL.create method.

This allows you to define a combiner for the materialized values, and add the stages to your GraphDSLBuilder, as parameters to the last argument. Note that this overloaded create method does not take a varargs parameter, so it may not be possible to have 14 different stages treated that way.

Assuming some names for your stages, here is how I would implement it, in the case of 3 outputs:

val runnable = RunnableGraph.fromGraph(
  GraphDSL.create(
    source, customGroupBy, sink1, sink2, sink3)(combiner) {  //the combiner is the function to combine the materialized values
      implicit b => //this is the builder, needed as implicit to make the connections 
      (src, cgb, s1, s2, s3) => //here are the stages added to the builder
      import GraphDSL.Implicits._

      src.out ~> cgb.in
      List(s1, s2, s3).map(_.in).zip(cgb.outlets).foreach{
        case (in, out) => in ~> out
      }

      ClosedShape
    }
  )
)

Remember that if you don't need one of the stages' materialized value, you can just add it inside the DSL by doing val cgb = b.add(customGroupBy)

来源：https://stackoverflow.com/questions/43119221/akka-stream-connect-to-multiple-sinks

标签

scala

akka

akka-stream

reactive-streams