JDBC with Webflux - how to dispatch to container thread

Question

I am working on a small proof-of-concept with webflux. In one part of my application I would like to communicate with a database (via JDBC) which is blocking and is not a good fit for reactor. Nevertheless for this proof-of-concept, I am thinking of following trick:

1. Define a dedicated thread pool (let's call it DBThreadPool) as ExecutorService with a fixed number of threads that equal JDBC connection pool size. Wrap that pool in Reactor Scheduler (named dbScheduller)

2. Wrap blocking call (like described in Reactor doc) and schedule it on dbScheduller:

   public Flux<DbUser> allUsers() {
       return Mono.fromCallable(() -> <jdbcQueryHere>)
           .flatMapIterable(Function.identity())
           .log("DB-OPER").subscribeOn(dbScheduller);
   }
   

3. As soon as the query is done, I would like to process the returned Flux<DbUser> with some reactive operators but perform them in a container thread so I can release the DB thread. According to Reactor doc it could be done via publishOn method:

   public Mono<ServerResponse> allUsers(ServerRequest request) {
       return ServerResponse.ok()
       .contentType(APPLICATION_STREAM_JSON)
       .body(
           usersDao.allUsers()
           .publishOn(<netty Thread pool as scheduller>)
           .map(<some function>)
           .filter(<some predicate>), 
       DbUser.class);
   }
   

Do you think this is a viable approach (until we get non-blocking drivers to databases)?

How to get access to container (netty) thread pool so one of its threads can be utilised to finalise (post-process data from DB and write to response) a HTTP request?

I know I can use DB thread by itself (by omitting publishOn) for finalising a HTTP request, but I would like to release it as soon as possible (so it can be reused by another request that also needs access to DB) and leave rest of the work (which potentially can be time consuming) to a netty managed thread (it can be the one that originally executed my handler method).

Answer 1

Do you think this is a viable approach (until we get non-blocking drivers to databases)?

I personally do think this is a valid approach. BUT many others (most of them know much more about reactive programming than I do) doubt this is a good idea.

Your alternatives:

• write a separate service for your blocking DB calls and call it using WebClient

  Pro: clean separation of blocking and non-blocking code.

  Con: One more service boundary including the network.

• write a separate service for your blocking DB calls including its own user interface.

  Pro: No inter service communication.

  Con: Reduces the percentage of your application that is actually reactive.

• if you want to keep the blocking code in this service, this is the correct way to do it.

How to get access to container (netty) thread pool?

If you are just rendering your response you don't need to, because that happens on the netty thread automagically.

If you have actual transformations to do, there is nothing ready-made to do this. I guess one could use some Netty API and implement a Scheduler based on that. But I don't know enough about Netty to help with that.

And to answer the follow-up question I kind of asked myself:

But isn't that a gigantic omission?

The publishOn methods weren't really intended for this. Their use-case is more along use in e.g. Swing applications where you have at the end of a pipeline the requirement to use a certain thread or thread pool.

My personal feeling is Reactor should have something like this, but the developers are currently not convinced that this is a good idea. So I suggest if you think it really makes sense for your application: open a ticket and be prepared to argue your point. Either you succeed or we both learn why it is a bad idea :-)

Anything else?

Glad you asked if you're doing blocking stuff on a thread pool in an otherwise reactive pipeline you should think how to handle the scenario when the thread pool (or the respective resource) is overloaded i.e. it processes events slower than they get produced.