To improve your process modeling, you must measure it.

You can’t improve something which you cannot measure.  This is a basic tenet in six-sigma and many other process improvement methodologies.    Yet few organizations even consider trying to objectively measure any aspect of the quality of the process models which they implement at the core of their process improvement initiatives.   Fortunately, this is an area where the use of a BPM platform offers significant advantages over other technical approaches.  The fundamental difference between a BPM solution and a traditional code solution is the existence of an explicit representation of the business process, including the steps, their sequencing, branching and looping, worker assignments and external interfaces.  This process representation makes it possible to reason about the characteristics of a BPM solution in ways that are much more difficult with other technologies.

Before going further, we need to clarify what we mean by process quality and what can be measured.  There are many aspects to process quality and not all can be objectively measured.

• One critical aspect of process quality is whether the process will execute on the platform.  Poorly defined processes can have deadlocks or unreachable steps which prevent complete execution.  Since the failure conditions are specific to a specific execution engine, these measures are best left to the manufacturer of the tool.  Most products offer some degree of validation, and this will not be considered further.
• Once we have established that a process will execute, the next most important aspect of quality is the degree to which it solves the intended problem and satisfies the business needs.  Given the variety and complexity of businesses, we cannot expect to directly measure this. It will require subjective human analysis for the foreseeable future.   We can however measure process characteristics which do facilitate that human analysis.  The rest of this article will focus on that.

Over the past fifteen years there has been much academic research on various process metrics.  One aspect which has received significant attention and produced useful results is metrics related to the understandability of process models.   One of the chief benefits of BPM over traditional coding is that the primary artifact is the Process Model, and that it describes concepts from the business domain, rather than programming constructs.  If the business subject matter experts can examine and analyze the process model, they can directly validate its correctness rather than trying to validate by testing inputs and outputs.  This eliminates several translation steps between the requirements and the executable solution, greatly reducing the risk of errors introduced by incorrect understanding.  However, this presumes that the business expert correctly and completely understands the process model.

On the surface, BPMN is straightforward and fairly simple.   Non- technical business people can achieve a basic level of understanding with a few hours of training and a few days of practice.  In reality, BPMN is a rich, complex notation and process models are often quite complex.  The ability to objectively measure characteristics which have been demonstrated to impact understandability is of great assistance in producing models which will be correctly understood by the business experts who are not process modeling experts.

We find it useful to separate two types of quality measures:  Overall measures of a process as a whole and localized modeling practices defined in one or a small number of elements in the model.  A complete discussion of all of the metrics we have used is beyond the scope of an article such as this, however, we will illustrate the concepts and point you towards more information.  First, we will describe a few of the most significant overall measures.   The simplest measure is the overall size, defined as the total number of Activities, Events and Gateways in a model.  If a model has too many elements, it cannot be viewed all at once and is harder to understand.  Research suggests an upper limit in the 30-40 total elements range.  However, it should be obvious that not all models with the same number of elements are equally complex.  Models with a lot of branching and looping are inherently more difficult to understand, as there are many different possible execution paths.   For any given size, the simplest model is a straight path with no gateways.  We have used several measures including the total number of possible execution paths, the amount of parallelism, the percentage of elements in loops, the average and maximum number branches defined in each gateway and the length of the longest path.   All of these produce a numeric measure.  Experiments can establish suitable ranges for these metrics.  A common approach is to give test subjects a period of time to study a model then ask questions about it.  The correct and incorrect answers can be scored and compared to various metrics.

The local practices which we discuss next are more subjective.  They are a result of interviews with modeling experts.   The BPMN standard includes a number of constructs which we feel should not have been included as they reduce understandability without adding any additional important capabilities.  These can be grouped broadly into two categories: implicit behaviors and overloaded elements.  As an example of the first, we believe that all branching should be in gateways.  BPMN allows multiple Sequence Flows in or out of Activities, which can provide the same branching as Gateways.  However, these are much harder to understand, especially for the non-expert.  The BPMN standard includes a particularly opaque example with 3 Sequence Flows out of an Activity and a mix of conditions on the Sequence Flows.  BPMN also includes several elements which support combining multiple, independent elements into a single one.  Multiple Events allow a single symbol to indicate that multiple Events can occur at a location in the model.  The notation does not communicate any information about the types of those events, which can be mixed.  Fortunately, not all products support these, but if yours does, do not use it.  Obviously, these recommendations are based on subjective judgements, and other experts may disagree with some of them.  Our over-riding principle in evaluating a process model is that the simplest and clearest model which will do the job correctly is the best.

We have been using the techniques described here for nearly ten years in our practice and have found that the results of this analysis align well with the judgements of experienced process modelers.  They have demonstrated usefulness in several ways.

• Most fundamentally, they help us produce process models which are correctly understood by all stakeholders.
• They serve as a valuable training aid for new modelers, providing consistent feedback and evaluation.
• For large teams, they improve the consistency so that processes from all modelers follow the same practices.
• They help scale a scarce resource, expert process modelers.

This has been a very short and incomplete introduction to BPM quality measurement.  Our goal is not to show you what to measure or how, but to open your mind to the idea that it is both possible and useful to apply metrics to you BPM development projects.  For readers who want to dive deeper into this topic, we recommend starting with Jan Mendling’s dissertation (available along with his more recent research at www.mendling.com).  It can be a challenging journey, but is worth the effort.