Use cases of the Workflow Engine

Question

I\'d like to know about specific problems you - the SO reader - have solved using Workflow Engines and what libraries/frameworks you used if you didn\'t roll your own. I\'d

Answer 1

I'm one of the authors of Cadence Workflow Engine we developed at Uber. The difference between Cadence and the majority of the existing workflow engines is that it is developer focused and is extremely flexible and scalable (to tens of thousands updates per second and up to billions of open workflows). The workflows are written as object oriented programs and the engine ensures that the state of the workflow objects including thread stacks and local variables is fully preserved in case of host failures.

What problems have you used workflow engines to solve? Cadence is used for practically any backend application that lives beyond a single request reply. Examples of usage are:

• Distributed CRON jobs
• Managing ML/Data pipelines
• Reacting to business events. For example trip events at Uber. The workflow can accumulate state based on events received and execute activities when necessary.
• Services Deployment to Mesos/ Kubernetes
• CI Pipeline implementation
• Ensuring that multiple service calls complete when a request is received. Including SAGA pattern implementation
• Managing human worker tasks (similar to Amazon MTurk)
• Media processing
• Customer Support Ticket Routing
• Order processing
• Testing service similar to ChaosMonkey

and many others

The other set of use cases is based on porting existing workflow engines to run on Cadence. Practically any existing engine workflow specification language can be ported to run on Cadence. There are multiple internal Uber systems that were ported. This way a single backend service can power multiple domain specific workflow systems.

What libraries/frameworks did you use?

Cadence is a self contained service written in Go with Go and Java client side libraries. The only external dependency is storage. Cassandra and SQL databases are supported.

Cadence also support asynchronous cross region (using AWS terminology) replication.

When did a simpler State Machine/Task Management like system suffice?

Inside Uber the Cadence service is managed by our team. So the overhead of building any custom state machine/task management is always higher than using Cadence. Outside the company the service and storage for it need to be set up. If you already have an SQL database the service deployment is trivial through a docker image. The docker is also used to run a local Cadence service for development on a personal computer or laptop.