A lot has been written about Big Data in the last few months. Most companies are trying to make sense of the large amount of data they have to serve as an input to both strategic and operational decisions. In this article, I’ll discuss a couple of reasons for the obvious link between BPM and Big Data and how they can co-exist with BPM implementations helping drive adoption of Big Data.
Most BPM platforms have built-in capabilities to collect process parameters and key process related data during process execution. Most also provide some form of rudimentary to sophisticated analysis of the data, perhaps not run time, (yet), but on a post facto basis.
Process owners use this data to gain insight into the process, for example:
- Add/remove resources for certain activities
- Staff resources based on time of day
- Study the impact of adding a new activity
- Study the impact of changing a business rule
So, BPM vendors are no strangers to using analytics on process data, which is one of the key reasons for moving to a BPM platform. Process related data is used for predictive analysis, to improve the current process and assess the impact of changes to the process over time. Most of these activities are done periodically on process data collected over time.
With multiple processes running and large amounts of data collected, in some cases a separate analytics engine attached to process related data might provide more sophisticated insight than the default analytics provided by the vendor. Again, sophistication of analytics varies across vendors.
All organizations have multiple applications, with many applications sharing common data elements, normally the same Master data, for example, Customer and Vendor information. Many times these sets of Master data are not synchronized across systems; they are missing pieces or are simply wrong, leading to erroneous reporting at the aggregate level. Very often, reconciling data across application siloes becomes a nightmare and businesses are left wondering about data quality.
BPM processes can be used to:
- Extract relevant data from multiple applications
- Process the extracted data using BPM processes to improve data quality, i.e. check for missing/erroneous data, update missing/erroneous data, and approve modified data
- Port to a central data store
Analytics software can then be run on the central store with clean data. This could be either the Reporting and Analytic capabilities provided by the BPM Tool itself, or third party tooling, depending on the level of sophistication required.
In some cases BPM can also be used to feed corrected data back to the “Core” applications after running and verifying reports/dashboards generated via the analytics engine. This may prove to be the more cost effective way, rather than make changes to old legacy applications. As we all know, a single change, if not done correctly, can cause impact on other areas of the application, especially in cases where the application has outlived its useful life and is on the Replacement list. We can also use BPM to pull data in from social media sites and analyze it
This synergy between BPM and Big Data is often easy to implement in small to medium size, growing companies where there may be only one or two “Core” applications in use. Due to rapid business growth, they choose BPM tools for both flexibility and shorter implementation cycles due to “no/low code” approach. In many cases the BPM vendor brings in the Analytics vendor and both work together on the solution, with the BPM vendor providing integration capabilities. A more important reason for BPM success in these organizations is that organization politics have not reached the peak. With growing businesses, teams feel they can grow too, as opposed to more stable companies where people tend to protect their turf.
In larger companies where applications have been allowed to grow unchecked with no real synergy or integration among them, (think global companies, multiple IT departments, business unit silos) this could cause a big drain on IT resources and costs. For these companies, a better strategy would be to focus on the “Core “ applications of record for purposes of analytics, then slowly replace/aggregate disparate applications into either the BPM tool itself or retire unneeded applications.
