BroadcastHub filtering based on “resource” the connected client is working on?

Question

I am writing a pure websocket web application, meaning that prior to the websocket upgrade there is no user/client step, more specifically: Authentication request goes over websockets as does the rest of the communication

There is/are:

• Exactly ONE websocket endpoint on /api/ws
• Multiple clients connected to that endpoint
• Multiple projects for multiple clients

Now, not each client has access to each project - the access control for that is implemented on the server side (ofc) and has nothing to do with websockets per se.

My problem is, that I want to allow collaboration, meaning that N clients can work on 1 project together.

Now if one of these clients modifies something, I want to notify all the other clients that are working on THAT project.

This is especially important, because atm I am the only one working on this and testing it and this is major oversight on my side, because right now:

if Client A connects to Project X and Client B connects to Proejct Y, if any of them update something in their respective project, the other one gets notified of those changes.

Now my WebsocketController is rather simple, I basically have this:

private val fanIn = MergeHub.source[AllowedWSMessage].to(sink).run()
private val fanOut = source.toMat(BroadcastHub.sink[AllowedWSMessage])(Keep.right).run()

def handle: WebSocket = WebSocket.accept[AllowedWSMessage, AllowedWSMessage]
{
  _ => Flow.fromSinkAndSource(fanIn, fanOut)
}

Now from my understanding what I would need is either

1) Multiple websocket endpoints per project, like /api/{project_identifier}/ws

(X)OR

2) Some means of splitting the WebSocket connections/the connected clients based on the project they are working.

As I would prefer not to go route 1) I'll share my thoughts on 2):

The problem I don't see a workaround for now, is that I may easily create some collection on the server side, where I store which user is working on which project at any given moment (like, if they choose/switch a project, the client sends that to the server and that one stores this information)

BUT I still have that one fanOut, so this would not solve my problem in regards to the WebSocket/AkkaStreams.

Is there some magic (filtering) to be invoked on BroadcastHub that does what I want?

edit: Sharing my whole websocket logic here now, after trying but failing to apply the good hints of @James Roper:

 class WebSocketController @Inject()(implicit cc: ControllerComponents, ec: ExecutionContext, system: ActorSystem, mat: Materializer) extends AbstractController(cc)

{ val logger: Logger = Logger(this.getClass())

type WebSocketMessage = Array[Byte]

import scala.concurrent.duration._

val tickingSource: Source[WebSocketMessage, Cancellable] =
  Source.tick(initialDelay = 1 second, interval = 10 seconds, tick = NotUsed)
    .map(_ => Wrapper().withKeepAlive(KeepAlive()).toByteArray)

private val generalActor = system.actorOf(Props
{
  new myActor(system, "generalActor")
}, "generalActor")

private val serverMessageSource = Source
  .queue[WebSocketMessage](10, OverflowStrategy.backpressure)
  .mapMaterializedValue
  { queue => generalActor ! InitTunnel(queue) }

private val sink: Sink[WebSocketMessage, NotUsed] = Sink.actorRefWithAck(generalActor, InternalMessages.Init(), InternalMessages.Acknowledged(), InternalMessages.Completed())
private val source: Source[WebSocketMessage, Cancellable] = tickingSource.merge(serverMessageSource)

private val fanIn = MergeHub.source[WebSocketMessage].to(sink).run()
private val fanOut = source.toMat(BroadcastHub.sink[WebSocketMessage])(Keep.right).run()

// TODO switch to WebSocket.acceptOrResult
def handle: WebSocket = WebSocket.accept[WebSocketMessage, WebSocketMessage]
  {
    //_ => createFlow()
    _ => Flow.fromSinkAndSource(fanIn, fanOut)
  }

private val projectHubs = TrieMap.empty[String, (Sink[WebSocketMessage, NotUsed], Source[WebSocketMessage, NotUsed])]

private def buildProjectHub(projectName: String) =
{
  logger.info(s"building projectHub for $projectName")

  val projectActor = system.actorOf(Props
  {
    new myActor(system, s"${projectName}Actor")
  }, s"${projectName}Actor")

  val projectServerMessageSource = Source
    .queue[WebSocketMessage](10, OverflowStrategy.backpressure)
    .mapMaterializedValue
    { queue => projectActor ! InitTunnel(queue) }

  val projectSink: Sink[WebSocketMessage, NotUsed] = Sink.actorRefWithAck(projectActor, InternalMessages.Init(), InternalMessages.Acknowledged(), InternalMessages.Completed())
  val projectSource: Source[WebSocketMessage, Cancellable] = tickingSource.merge(projectServerMessageSource)

  val projectFanIn = MergeHub.source[WebSocketMessage].to(projectSink).run()
  val projectFanOut = projectSource.toMat(BroadcastHub.sink[WebSocketMessage])(Keep.right).run()

  (projectFanIn, projectFanOut)
}

private def getProjectHub(userName: String, projectName: String): Flow[WebSocketMessage, WebSocketMessage, NotUsed] =
{
  logger.info(s"trying to get projectHub for $projectName")

  val (sink, source) = projectHubs.getOrElseUpdate(projectName, {
    buildProjectHub(projectName)
  })

  Flow.fromSinkAndSourceCoupled(sink, source)
}

private def extractUserAndProject(msg: WebSocketMessage): (String, String) =
{
  Wrapper.parseFrom(msg).`type` match
  {
    case m: MessageType =>
      val message = m.value
      (message.userName, message.projectName)
    case _ => ("", "")
  }
}

private def createFlow(): Flow[WebSocketMessage, WebSocketMessage, NotUsed] =
{
  // broadcast source and sink for demux/muxing multiple chat rooms in this one flow
  // They'll be provided later when we materialize the flow
  var broadcastSource: Source[WebSocketMessage, NotUsed] = null
  var mergeSink: Sink[WebSocketMessage, NotUsed] = null

  Flow[WebSocketMessage].map
  {
    m: WebSocketMessage =>
    val msg = Wrapper.parseFrom(m)
    logger.warn(s"client sent project related message: ${msg.toString}");
    m
  }.map
    {
      case isProjectRelated if !extractUserAndProject(isProjectRelated)._2.isEmpty =>
        val (userName, projectName) = extractUserAndProject(isProjectRelated)

        logger.info(s"userName: $userName, projectName: $projectName")
        val projectFlow = getProjectHub(userName, projectName)

        broadcastSource.filter
        {
          msg =>
            val (_, project) = extractUserAndProject(msg)
            logger.info(s"$project == $projectName")
            (project == projectName)
        }
          .via(projectFlow)
          .runWith(mergeSink)

        isProjectRelated

      case other =>
      {
        logger.info("other")
        other
      }
    } via {
      Flow.fromSinkAndSourceCoupledMat(BroadcastHub.sink[WebSocketMessage], MergeHub.source[WebSocketMessage])
      {
        (source, sink) =>
          broadcastSource = source
          mergeSink = sink

          source.filter(extractUserAndProject(_)._2.isEmpty)
            .map
            { x => logger.info("Non project related stuff"); x }
            .via(Flow.fromSinkAndSource(fanIn, fanOut))
            .runWith(sink)

          NotUsed
      }
    }
}

}

Solution/Idea how I understood it:

1) We have a "wrapper flow" where we have a broadcastSource and mergeSink that are null until we materialize them in the outer } via { block

2) In that "wrapper flow" we map each element to inspect it.

I) In case it's project related, we

a) get/create an own subflow for the project b) filter the elements based on the project name c) let those that pass the filter be consumed by the sub/project-flow so that everyone that is connected to the project gets that element

II) In case it's not project related, we just pass it on

3) Our wrapper flow is going by a "on demand" materialized flow, and in the via where it is materialized we let the elements that are not project related be distributed to all connected web socket clients.

To summarize: We have a "wrapper flow" for the websocket connection that either goes via a projectFlow or a generalFlow, depending on the message/element it is working in.

My problem now is (and it seems to be trivial, yet I am struggling somehow) that EVERY message should go into the myActor (atm) and there should be messages coming out from there as well (see serverMesssageSource and source)

But the above code is creating non-deterministic results, e.g. one client sends 2 messages, but there are 4 being handled (according to logs and results the server sends back), sometimes messages are suddenly lost on their way from the controller to the actor.

I can't explain that, but if I leave it just with _ => Flow.fromSinkAndSource(fanIn, fanOut) everyone gets everything, but at least if there is only one client it does exactly what is expected (obviously :))

Answer 1

I would actually recommend using Play's socket.io support. This offers namespaces, which from what I can tell from your description make it straight forward to implement exactly what you want - each namespace is its own independently managed flow, but all namespaces go down the same WebSocket. I wrote a blog post about why you might choose to use socket.io today.

If you don't want to use socket.io, I have an example here (this uses socket.io, but doesn't use socket.io namespaces, so could easily be adapted to run on straight WebSockets) which shows a multi chat room protocol - it feeds messages into a BroadcastHub, and then there is one subscription to the hub for each chat room that the user is currently a part of (for you, it would be one subscription for each project). Each of those subscriptions filter the messages from the hub to include only the messages for that subscriptions chat room, and then feed the messages into that chatrooms MergeHub.

The highlighted code here is not specific to socket.io at all, if you can adapt the WebSocket connection to be a flow of ChatEvent, you can use this as is:

https://github.com/playframework/play-socket.io/blob/c113e74a4d9b435814df1ccdc885029c397d9179/samples/scala/multi-room-chat/app/chat/ChatEngine.scala#L84-L125

To address your requirement to direct non project specific messages through a broadcast channel that everyone connects to, first, create that channel:

val generalFlow = {
  val (sink, source) = MergeHub.source[NonProjectSpecificEvent]
    .toMat(BroadcastHub.sink[NonProjectSpecificEvent])(Keep.both).run
  Flow.fromSinkAndSourceCoupled(sink, source)
}

Then, when the broadcast sink/source for each connected WebSocket connects, attach it (this is from the chat example:

} via {
  Flow.fromSinkAndSourceCoupledMat(BroadcastHub.sink[YourEvent], MergeHub.source[YourEvent]) { (source, sink) =>
    broadcastSource = source
    mergeSink = sink

    source.filter(_.isInstanceOf[NonProjectSpecificEvent])
      .via(generalFlow)
      .runWith(sink)

    NotUsed
  }
}