Data Consistency Across Microservices

Question

While each microservice generally will have its own data - certain entities are required to be consistent across multiple services.

For such data consistency requiremen

Answer 1

Same problem here. We have data in different microservices and there are cases where one service needs to know if there is a specific entity in another microservice. We do not want the services to call each others to complete a request because this adds response time and multipies downtimes. Also it adds a nightmare of coupling depth. The client should not decide about business logic and data validation/consistency either. We also do not want central services like "Saga Controllers" to provide for consistency between services.

So we use a Kafka message bus to inform observing services of state changes in "upstream" services. We try very hard not to miss or ignore any messages even in error conditions and we use Martin Fowler's "tolerant reader" pattern to couple as loosely as possible. Still sometimes services are changed and after the change they might need information from other services that they might have emitted on the bus before but they are now gone (even Kafka cannot store forever).

We decided for now that each Service be split in a pure and decoupled web service (RESTful) that does the actual work and a separate Connector-Service which listens to the Bus and may also call other services. This Connector runs in the background. It is only triggered by bus messages. It then will try to add data to the main service via REST calls. If the service responds with a consistency error, the connector will try to repair this by fetching the needed data from the upstream service and inject it as needed. (We cannot afford batch-jobs to "synchronize" data en block, so we just fetch what we need). If there are better ideas, we are always open, but "pull" or "just change data model" is not what we consider feasible...