Use cases of the Workflow Engine

Question

I\'d like to know about specific problems you - the SO reader - have solved using Workflow Engines and what libraries/frameworks you used if you didn\'t roll your own. I\'d

Answer 1

I'm biased as well, as I am the main author of StonePath.

I have developed workflow applications for the U.S. State Department, the Geneva Centre for Humanitarian Demining, several fortune 500 clients, and most recently the Washington DC Public School System. Every time I have seen a 'workflow engine' that tried to be the one master reference for business processes, I have seen an organization fighting itself to work around the tool. This may be due to fact that these solutions have always been vendor/product driven, and then end up with a tactical team of 'consultants' constantly feeding the app... but because of this, I tend to react negatively when I hear the benefits of process-based tools that promise to 'centralize the workflow definitions in one place and make them repeatable'.

That said, I very much like Ruote - I have been following that project for some time and should I need that kind of solution, it will be the next tool I'll be willing to try. StonePath has a very different purpose than ruote - where Ruote is useful to Ruby in general, StonePath is aimed at Rails, the web framework written in Ruby. Where Ruote is about long-lived business processes and their associated definitions, StonePath is about managing State-based workflow and tasking. Frankly, I think the distinction from the outside looking in might be subtle - many times the same kinds of business processes can be represented either way - the state-and-task-based model tends to map to my mental model though.

Let me describe the highlights of a state-based workflow. In short, imagine a workflow revolving around the processing of something like a mortgage loan or a passport renewal. As the document moves 'around the office', it travels from state to state. Imagine if you are responsible for the document, and your boss asked you every few hours for a status update, and wanted a brief answer... you'd say things like "It is in data entry"... "We are checking the applicant's credentials now"... "we are awaiting quality review"... "We are done"... and so on. These are the states in a state-based workflow. We move from state to state via transitions - like "approve", "apply", kickback", "deny", and so on. these tend to be action verbs. Things like this are modeled all the time in software as a state machine.

The next part of a state/task-based workflow is the creation of tasks. A Task is a unit of work, typically with a due date and handling instructions, that connects a work item (the loan application or passport renewal, for instance), to a users "in box". Tasks can happen in parallel with each other or sequentialy, and we can create tasks automatically when we enter states, create tasks manually as people realize work needs to get done, and require tasks be complete before we can move onto a new state. All of this kind of behavior is optional, and part of the workflow definition.

The rabbit hole can go a lot deeper than this, and I wrote an article about it for Issue #4 of PragPub, the Pragmatic Programmer's Magazine. Check out the reo link above for an updated PDF of that article.

In working with StonePath the last few months, I have found that the state based model maps really well to restful web architectures - in particular, the tasks and state transitions map nicely as nested resources. Expect to see future writing from me on this subject.

Answer 2

I'm one of the authors of Cadence Workflow Engine we developed at Uber. The difference between Cadence and the majority of the existing workflow engines is that it is developer focused and is extremely flexible and scalable (to tens of thousands updates per second and up to billions of open workflows). The workflows are written as object oriented programs and the engine ensures that the state of the workflow objects including thread stacks and local variables is fully preserved in case of host failures.

What problems have you used workflow engines to solve? Cadence is used for practically any backend application that lives beyond a single request reply. Examples of usage are:

• Distributed CRON jobs
• Managing ML/Data pipelines
• Reacting to business events. For example trip events at Uber. The workflow can accumulate state based on events received and execute activities when necessary.
• Services Deployment to Mesos/ Kubernetes
• CI Pipeline implementation
• Ensuring that multiple service calls complete when a request is received. Including SAGA pattern implementation
• Managing human worker tasks (similar to Amazon MTurk)
• Media processing
• Customer Support Ticket Routing
• Order processing
• Testing service similar to ChaosMonkey

and many others

The other set of use cases is based on porting existing workflow engines to run on Cadence. Practically any existing engine workflow specification language can be ported to run on Cadence. There are multiple internal Uber systems that were ported. This way a single backend service can power multiple domain specific workflow systems.

What libraries/frameworks did you use?

Cadence is a self contained service written in Go with Go and Java client side libraries. The only external dependency is storage. Cassandra and SQL databases are supported.

Cadence also support asynchronous cross region (using AWS terminology) replication.

When did a simpler State Machine/Task Management like system suffice?

Inside Uber the Cadence service is managed by our team. So the overhead of building any custom state machine/task management is always higher than using Cadence. Outside the company the service and storage for it need to be set up. If you already have an SQL database the service deployment is trivial through a docker image. The docker is also used to run a local Cadence service for development on a personal computer or laptop.