Process Mining Technologies

“The unlike is joined together, and from differences results the most beautiful harmony.”
– Heraclitus 

Business and government leaders are hearing more and more about process mining. So, what is process mining? If you’ve asked yourself that question, you’ve come to the right place. This article defines process mining and how it works, describes the “right” way to adopt it within your organization, suggests where the industry is heading with some bleeding-edge possibilities, and concludes with the assertion that process technologies will become ubiquitous. Welcome to the growing world of process mining. 

What Is Process Mining and How Does It Work?

Process mining is a powerful artificial intelligence (AI) technique with expansive business and national security use cases. In a business context, process mining algorithms accurately discover processes and in a fraction of the time compared to traditional process mapping methodologies. In a national security context, process mining enables bleeding-edge analyses to fill intelligence gaps.

The goal of process mining is to turn event data into insights and actions [1]. It resides at the intersection of two vast and interdisciplinary fields – data science and management science (see Figure 1). It helps organizations make better decisions and better use of scarce resources while simultaneously producing additional value.

As the name implies, process mining AI applies human thought-like processing to data and surface Markov or Bayesian models of decision-making processes from various event input formats. In a typical business context, the technique analyzes log data generated from literally any management information system, and it learns latent processes from these data with no a priori knowledge about the processes themselves. It is platform agnostic and highly extensible. It works for any industry, company, information system or process [2].

Depending upon organization structure and management styles, business process mining may either complement existing business process management efforts or substitute for such efforts. Quick return on investment (ROI) use cases enable better decision making and help organizations save on costs and re-purpose savings into prioritized capabilities. Deeper and more impactful use cases increase organizational agility and enhance competitive advantage [2].

The first three pieces of information listed below are needed to discover processes; the additional features enrich the analysis:

• Case ID: An identifier that represents a specific execution of a process.
• Activity: One of several steps performed within a process. For example, web-based eCommerce process activities might include “add item to cart” and “initiate checkout.”
• Time stamp: This orders the activities within each case and enables sophisticated modeling.
• Person or department attribute: When the person or department responsible for process steps are known, social nets of interactivity between process activities may be displayed to reveal important insights and troubleshoot bottle necks (optional).
• Cost attribute: If event costs are encoded, then business expenses associated with processes may be understood (optional). 

Figure 2 contains a trivial example. Assume process cases 1 through 3 are time ordered and all contain the same activities, labeled “A” through “E.” In Case 1, the activities happen in natural order. In Case 2, Activity “C” precedes “B.” Finally, in Case 3, Activity “D” is repeated before concluding with Activity “E.” The discovered model accounts for these process variations. Real-world processes are significantly more complicated.

Machine readable process models, which are frequently in BPMN format, come baked with descriptive statistics, transition probabilities and capacity estimates. These data allow organizations to quickly understand their processes, simulate change assumptions, target improvements without guesswork, re-measure upgraded ecosystems and report on improvement savings [2]. 

The Right Way to Adopt Process Mining

Digital transformation is a big buzzword these days and for good reason. Organizations are feeling existential competition from disruptive startups, as well as from firms such as Amazon that are exploring new operational territory [3].

Recently, a prospective client asked me the following question, which really sets up the perfect use case for process mining: “If you’re trying to automate thousands of decisions, how would you pick out the first five, for example, to go after? If you do automate a decision, you get a result and you can know whether the automation was beneficial and, ideally, by how much. But what if you wanted to triage from the front end and decide which decisions to tackle first with analytics/automation? What criteria could I be using to triage?”

Process mining is a nascent enabling technology for forward-thinking organizations with appropriate top-down mandates (to this last point, the CEO or C-suite sponsor is a critical requirement for every major project – analytics or otherwise). Assuming this prerequisite is met, the right answer to the question requires a comprehensive process inventory in order to prioritize improvement and automation efforts. It is a potentially valuable project. There are several approaches.

The typical way to approach a project like this is to corral program managers into conference rooms and systematically document processes using a mapping methodology popular with Lean Six Sigma, for example. This may also involve asking employees to manually tell you where pain points reside, as well as shadowing employees at their desks and observing daily habits. While this is an effective way to understand organizational processes, it takes longer, consumes more employee time, misses process deviations and re-work, is more likely to contain measurement errors, and limits follow-on analyses of value.

The emerging technique employs a multisystem analysis, which consolidates or “knits” event logs together for end-to-end and drill-down process views. The company’s enterprise resource management, human resources and customer relationship management systems may all be mined to derive deep and actionable business intelligence. At this technological stage, a comprehensive process mining solution is still a “gray-box” project, which necessitates client-side expertise to assemble the process diagrams and interpret the process ecosystem. Once mapped end-to-end, however, front-end triage and automation prioritization becomes a data-driven task for finding quick-wins. And, as desired by business leaders, speedy ecosystem re-measurement after process improvements captures ROI rather precisely.

Other options include a stepwise process mining approach, which tackles known process pain points first and then deepens adoption thereafter. Finally, a hybrid approach may also make sense where process mining complements established continuous process improvement efforts.

Once a process inventory is complete, an organization should rate them across meaningful dimensions to understand where improvement opportunities reside. Process activities that get prioritized tend to:

• involve manual steps (copy/paste; CSV file transfers; scanning and uploading),
• have many transactions (lots of data flowing through the process with many decisions),
• have relatively stable and well-understood clusters of decisions (Reject, Approve, Return),
• have potentially asymmetrical impacts on company brand, customer experience or the bottom-line (call center or hospitality front desk experience),
• be online forms with user-experience issues (as evidenced by process re-work or longer-than-desired average form completion time), and
• involve people doing things. 

These are considerations that, if automated or improved from a process perspective, tend to deliver the best value. Automating process steps that involve these types of dynamics may save labor costs, reduce interdepartment pass-through errors and re-work, improve customer experience, speed operations and generally improve business outcomes.

Beyond the scope of my prospective client’s immediate needs, process mining enables other valuable applications such as:

Data-driven key performance indicators (KPI). Process logs enriched with data elements such as department, geographic location and even employee identifier provide end-to-end process views by slice, which may be used in performance evaluations to drive the organization positively.

Continuous process conformance monitoring. It is critical that certain process steps are followed precisely – otherwise people go to jail. For example, in a regulatory compliance or audit environment, process mining tools reduce risk and drive quality by monitoring process activity over fixed process models and sending intelligent alerts when process conformance deviations occur.

Optimized systems of intelligence. Does your team know where your corporate bottlenecks are? Analysis-ready process ecosystems contain data-driven and directionally accurate simulation input parameters that help organizations ask “what if” questions, refine operations and gain competitive advantage.

The Future of Process Mining

Processes are usually considered intentional with profit-maximizing businesses or mission-driven government agencies. However, processes are also emergent and self-organizing. Academics suggest that processes underlie all phenomena [4]. Indeed, their writings assert that everything is ultimately connected with everything else. Process, connectedness and ever-present change have enraptured mankind for millennia. This article began with a quote from the Greek philosopher, Heraclitus, which encapsulates the natural beauty of process interactions and emergent harmony from disparate data.

Bringing these concepts together, future process technologies will analyze disparate and ever-changing data streams, identify process signals, auto-discover process models and elucidate naturally occurring complex process ecosystems.

Many other types of data may potentially be mined, in addition to management information systems logs. Nearly any data source that chronicles events is usable; these data may be obtained from cybersecurity logs, daily financial regulatory filing data, in extremis Internet of Things networks, social media posts, dark websites, etc. [5].

Poorly understood or opaque processes are national security and transnational crime-fighting intelligence gaps to be mitigated. Machine readable process models of societies, governments, holistic corporate ecosystems, critical infrastructure sector participants and other processes of interest enable numerous applications:

Information warfare (IW). The information age presents vivid new security challenges related to information weaponry. For instance, Russia sowed confusion and distrust during the 2016 U.S. presidential election quite effectively with micro-targeted information and disinformation campaigns. Enhanced with reflexive control, Russia’s IW technique combines models of decision-making processes with vectors designed to exploit process weaknesses – meticulously introducing into human or machine processes data that inclines the adversary toward taking an action that favors the attacker. Process technologies enable techniques for understanding societal, cultural, political, military and critical infrastructure process weaknesses, at scale. Algorithmic process models discovered from varieties of data augment other IW capabilities and give the United States actionable, data-driven intelligence to steel against IW attack vectors [5]. 

Anti-IW risk management standard and framework. Security conscious organizations should consider a comprehensive IW risk management program. A comprehensive checklist with suggested IW remediation [6] educates the organization, develops crisis and public relation plans and many other prudent actions to reduce IW risk. For example, a Red Team process analysis creates machine readable process models from an enemy’s vantage and then improves realistic threat emulation training. Machine process insights are useful for identifying process weak points, such as systems near capacity that could be subjected to distributed denial-of-service (DDoS) attacks or semantic attacks that closely simulate normal operations so as to avoid detection. Finally, process insights into human operations are useful to thwart social engineering attacks [5]. An up-to-date process inventory, as described earlier in this article, is an extremely valuable asset to help protect organizations from IW.

Anti-fragility. Trade-offs exist between organizational complexity and the ability to weather upheaval [7, 8]. Critical infrastructure sector participants have a vested interest to continually manage fragility risk and monitor process ecosystems. For example, process mining financial regulatory data and market event logs suggest hardening strategies and provide insight into sector human operations as well as machine decision-making processes [9].

Organization charts. Troll farms, human trafficking, environmental crime organizations, drug cartels and regional gangs may all be understood and monitored with process technologies. Scaled into production, the solution elucidates and monitors many hundreds of potential domestic and international criminal organizational processes. Importantly, it is highly desirable to have a technical solution that can mine noisy open source data. Using data “in the wild” or available for nominal purchase, nonintuitive organizational charts are constructed to understand relationships within and between opaque organizations. Such models illuminate hidden relationships, internal organization dynamics and external interactions. 

Conclusion

Processes are ubiquitous, and the technologies to exploit or monitor processes of interest must become ubiquitous, as well. I started More Cowbell Unlimited to help America remain a beacon of hope and strength on the world stage. Peer adversaries, non-state actors and eroding competitive advantage necessitate focused attention on dominating all aspects of AI – including process mining.

References and Sources

1. van der Aalst, Wil M. P., 2016, “Process Mining: Data Science in Action,” 2nd ed., New York: Springer.
2. Bicknell, John W., 2019, “Process Dominance: The Capability Nobody Is Talking About,” https://morecowbellunlimited.com/wp-content/uploads/Process-Mining-in-DoD-Context-White-Paper.pdf (May 16, 2019).
3. Koplowitz, Rob, 2019, “Process Mining: Your Compass for Digital Transformation,” https://www.forrester.com/report/Process+Mining+Your+Compass+For+Digital+Transformation/-/E-RES145997# (May 16, 2019).
4. Whitehead, Alfred North, 1979, “Process and Reality,” 2nd edition, New York: Free Press.
5. Bicknell, John W, and Werner G Krebs, 2019(b), “Process Mining: The Missing Piece in Information Warfare,”ResearchGate, February, https://doi.org/10.13140/RG.2.2.23584.94722/1.
6. Bicknell, John W, and Werner G Krebs, 2019(a), “FOCAL Information Warfare Defense Standard,” ResearchGate, June, https://www.doi.org/10.13140/RG.2.2.12672.07687.
7. Brafman, Ori, and Rod A. Beckstrom, 2006, “The Starfish and the Spider: The Unstoppable Power of Leaderless Organizations,” Penguin Books, https://www.amazon.com/Starfish-Spider-Unstoppable-Leaderless-Organizations/dp/1591841836 (May 16, 2019).
8. Taleb, Nassim Nicholas., 2014, “Antifragile: Things That Gain from Disorder,” reprint edition, New York: Random House Trade Paperbacks.
9. Bicknell, John W., and Werner G. Krebs, 2019(c), “Protecting Critical Infrastructures: Financial Services,” https://morecowbellunlimited.com/wp-content/uploads/Protecting-Critical-Infrastructures-Financial-Services.pdf (May 15, 2019).