The Missing Step to Automate your Processes with Software
Steven Sinofsky's take on how hard it is to automate processes with software
Automation is really hard to do well - and the more automation you do, the more brittle it becomes
Steven Sinofsky’s Hardcore Software blog recently commented on how hard it is to automate processes with software. And he has some good examples. I agree with his insight that every process looks simpler to automate from the outside, than from the inside.
He rightly points out that it isn’t the main-line process - the “happy path” as process people would say - isn’t necessarily hard to automate. It’s all the endless exceptions which a person would just handle, but a program needs to be told what to do in each case. His examples - an invoicing system he built when younger, build systems for software, medical diagnoses - are all good ones.
Even better was “scheduling meetings”:
My favorite example of automation is scheduling meetings. I never met an important big boss with multiple administrative assistants who didn't tell me scheduling meetings is easy and asked why they still need to have admins or why doesn't Outlook solve this.
The least empathetic managers I knew had no idea how hard it was to schedule meetings. There is literally no standard process. Everyone who ever told me it was standard ("I [big boss] always let people use my blocks of time MW 10-12") simply wasn't thinking about how the organization contorted around their rules and how every reschedule was a cascade of scrambling and frustration. Problem solved.
It’s a good example in that we can all relate to how many exception cases are, and how much harder this problem gets the more people are involved - both finding a free time for the meeting, and knowing which people might be okay to lose to a scheduling conflict, and whether the meeting is online or in person or both, and whether food is needed and drink. And whether additional people can be added to the invite. Does his or her boss need to be included? are there holidays in some locales that are not holidays in others? time zones?
On top of the complexity of all the exceptions and escalations, there is the problem of discovering all of them in advance, rather than experiencing them as failures. And on top of that, the solution built with pure automation - even AI-automation - is brittle. It will likely fail. And if you leverage LLMs you can avoid a traditional “error” message, but you’ll have the LLM assert an incorrect answer. As Sinofsky says:
The problem then becomes not just the error rate [..] but that the tools are currently designed to always produce a fine sounding answer rather than say “I don’t know.” This works fine or even better than fine for generating on the fly words or images for basic use cases. But in practice that is because those uses do not have financial, physical, or societal costs. So even the most basic of agents and automation will be seen to require oversight.
In short, brittle. And the kind of failures that are hidden from you in little bits of code or in LLM responses (asserting an incorrect answer with confidence), are the hardest to detect. In a six-sigma view of the world a problem is rated for Severity, Frequency, and Detectability. A difficult to detect problem can be as bad as one that happens frequently, or one that is quite severe. A problem that is both severe and hard to detect is really bad (think structural problems in bridges, perhaps). A problem that is severe, frequent, and hard to detect is obviously even worse.
For most of the software world, there’s been no real escape from this trade-off of automation and brittleness. The more you automate, the more efficient you are. But, the more brittle your process is, and potentially the more likely to break or need to be replaced.
In the software world, one example of this is integration. You can integrate two separate systems really well with APIs. However, if those APIs change, you break the integration (brittle), it doesn’t just “figure out what to do if it changed” like a human would. And then there were places where no APIs were available - too “fuzzy” for the integration software to tackle. leading to strange terms like the “bio-link” for integration (human swivel chair). And the industry had an answer to this in the 2015-2022 era, burning hot and bright: RPA.
Trust me, the Robotic Process Automation (RPA) world is 100% built on this tradeoff. It attempts to automate that which couldn’t previously be automated - the human interactions with software that aren’t backed by clean APIs. Many previous generations of software failed to automate this work in a way that was acceptable to the business. And the approach did resolve some of the most basic brittleness issues at the beginning of the curve - in other words, it pushed the line of “where is it reasonable to stop automating” quite a bit further.
However, the more RPA-style automation you have, the more brittle the solution is likely to be. Each automation is, in and of itself, a basket of exceptions and escalations and re-tries. Each one can be broken by a change to the user interface, to networking protocols, to security protocols, to a security credentials. Each one can be broken by software being retired or brought online.
And none of these automations really make sense, in a visceral way, to the business. That is not to say that business users can’t record automations - they can. But six months later can they explain what they did and why, and why they didn’t do these other things - mostly no. And if a stranger has to pick up the automations you’ve built in RPA, and you haven’t done process and requirements documentation and documented your automations (code), they are really stuck. The time to debug and fix will multiply, and it may be faster to redevelop the automation than to fix it.
Is it AI to the rescue?
Our software industry is endlessly creative while not fundamentally changing the approach. AI was to fix these brittleness issues: it would notice the name of a field changed to a word similar to the one it wanted. Or that the field looked the same but had moved to a different place on the screen. It might help route the work through the sequence of automations. And AI can maybe read that messy handwriting that previously needed someone to read it manually (escalation/exception path).
But it is still fundamentally brittle in that a small change can break the process - and when it breaks, it may break just for that one instance, or for everything of a certain type or use case, or it may break in all respects on a common element.
So AI helps - and RPA helps - but they don’t get you to 100% automation with sustainability, because the point of brittleness will be reached as you strive for more automation, and because the solution isn’t easily understood by business or technical personnel alike. And in many cases those failures will happen (with AI) in undetectable ways.
There is an answer. And it is a process. But typically software engineers miss a critical step.
What Sinofsky describes doing - and what most software engineers or programmers would do - is to move from “I understand the process” straight to automating that process with software (and yes, if you automate it with AI, it is still automation, and it is still software - it isn’t magic!). Let me tell you about what’s missing.
The Missing Step: Modeling the process
There’s an important step being left out. Modeling the process. Now, the software people will tell you that you’re wrong, they captured and documented the requirements. I’m here to tell you that requirements are not (usually) documented as a process model. So usually, when someone says requirements, it will have everything and anything but the process.
It turns out process orchestration is an area with great available research, led by PhD’s who have spent their careers advancing the art of process orchestration. Since we’re talking about automating with software, however, let me cut to the chase and let you look at my homework.
In the 2000’s, a bunch of smart people who understand things like UML, BPEL, and other modeling notations, got together and designed a new model and notation for business processes: BPMN (Business Process Model and Notation)12. Wikipedia gives an excellent resource or jumping off point.
BPMN is a standard notation - for representing business processes - which you can draw by hand on a whiteboard or in your notes. Unlike a typical flowchart notation, BPMN is a standard that is unambiguously defined, so that the resulting process can be determined to be “sound” (or not). And a sound process can be executed by an interpreter (process orchestration engine).
An example (from Camunda’s website) is shown below. Some processes may be much simpler - and some much more complex.
For more complicated processes, they may include sub-processes (any one of these rectangular boxes may in fact implement a sub-process). Sub-processes can be re-used in various higher level processes, allowing for business process re-use at a level of abstraction that works - a concept familiar to software engineers.
Decision gateways can create a single choice output, while splits (parallel gateways) generate parallel pathways along output lines. Join gateways can gather these threads back together.
Why capture the process model in BPMN?
This question almost answers itself. While no model is correct, some are useful. BPMN models are useful.
Processes are the algorithms that run your business. Process models capture those algorithms.
Process models help us expose the various ways our thinking about a process has been inexact or imprecise. They expose the ambiguities in our thought process. This is important, because this is how we run our business.
Process models are a way for us to build a common understanding between various business stakeholders about what the process really is. This is important, because this is how we run our business.
Process models are also a way for us to build a common understanding between various IT stakeholders about what the process really is. This is important, because this is how we build systems and software to run our business.
BPMN itself becomes the lingua franca of process in your organization, and the process models become the source of truth for how your business works. This is important for your business and its processes to be resilient over time.
Finally, these process models are executable. Meaning, the diagram (model) of your process, can be executed faithfully - a form of WYSIWYG (What You See Is What You Get) for business processes.
I was fortunate to work with Lombardi Software3 in the early ‘00s, where we built the first software system for natively running business processes from the BPMN diagram itself. Previously we built solutions where the process model was entirely abstract. Being able to show our customers their business process, and then run it, was a game changer.
I still wouldn’t use BPMN to model your meeting-scheduling process. But you *can* use BPMN to model all kinds of processes in your business, including your software builds (CI/CD and DevOps), your sales process, your order to fulfillment process, your returns (neé reverse logistics!) process.
How does BPMN and process orchestration lead to successful outcomes?
Taking it back to the top - what does modeling - even executing the process - with a BPMN process orchestration engine - have to do with automating a process with software? I’ve implemented thousands of successful projects myself and through BP3 over the last 20 years, using process modeling and orchestration engines to provide the scaffolding into which all the automation (or AI automation) can be implemented to its full effect.
Many intractable problems (according to clients, software partners, and competitors) were solved well with this approach - and solved sustainably by our team.
What does sustainably mean? It means that we have had clients running our solutions for 17 years or longer (going back to Lombardi). These systems typically get retired not because they don’t work, but because some underlying software library has reached end of support and that forces a re-think of the solution. It means that these solutions aren’t brittle. It means that humans understand how they work and why. These systems were designed to be understood by humans *and* machines.
So if you want to automate processes with software in your business, start by looking at real process orchestration software, and do your homework on the work that has come before you in this area. I’d recommend taking a look at Camunda4, for example.
Putting it all together
In summary, if you want to use software to automate a process, please do the following:
Form an understanding of the process and any written requirements you can lay hands on.
Create a process model using BPMN - the formal language will help, and there are great tools for building these process models.
Socialize your process model(s) with all the stakeholders, walk through it with them for feedback and things that might be missed. Ask what happens when things go wrong here, or there. When no one responds to a decision needed. When too much time elapses, etc. Ask about all the corner conditions.
Also use this as an opportunity to understand who is responsible for each part of the process, and what systems and information support their work.
Now circle back with opportunities for improvement (if any). If we could make it better what would it look like?
Now you’re ready to implement automation, AI, decisions, and user experiences in the context of your business process. And the process that runs the business now is the same as the process in the BPMN diagram. That’s pretty valuable.
And that, my friends, is how you tackle the hard problem of automating a business process with software.
One More Thing…
On October 28th, the Austin Technology Council will be hosting the C-Suite Summit - an event not to be missed if you are an executive leader in an Austin tech company or in a bigger firm heading up their Austin office.
When: Monday, October 28th, 2024 from 1pm to 6pm
Where: St. Edwards University- Ragsdale Center Ballroom. 3001 South Congress Ave, Austin, TX, USA.
Our keynote speaker Razat Gaurav, CEO of Planview, is fantastic, as is our closing keynote speaker - John Nehme, the President/CEO of Allies Against Slavery. This has always been a reunion of sorts for a who’s who in Austin tech, and a chance to be inspired by what is possible. I will see you there!
Okay, Another thing…
This coming week I’ll be joining BP3 at CamundaCon in New York City. This is the second time Camunda has hosted this event in the USA. I encourage anyone reading this to register for CamundaCon either in person or on the livestream - it’s a great way to learn about process orchestration and how it can be the framework for creating value in your organization.
Register here: https://www.camundacon.com
Especially excited to see the presentation by Charter Communications and BP3 on the journey with process orchestration and mission critical application development. It is a great story of partnership, collaboration, and culture that you won’t want to miss!
Expect to see more on this channel from me after the conference.
Jakob Freund and Bernd Rücker’s excellent “Real Life BPMN” is one recommendation
Another classic is Bruce Silver’s “BPMN Method and Style”
I joined Lombardi Software when there were ~35 people about a year after they nearly went out of business. At exit Lombardi was ~220 employees. When I left, I was running the technical services teams across Europe and North America, who were responsible for building solutions around our clients’ business processes. It was very gratifying work.
Camunda entered the market in 2013 with on open source licensed process orchestration engine. BP3 has been partnering with Camunda since that time, and I believe implemented the first Camunda implementation in 2013 at Ontario Teachers Pension Plan. Currently BP3 are a platinum certified partner and award-winning partner over the last 4 years.