How to build enterprise resilience in the face of growing AI risk

A recent study by StackGen reveals that AI-related incidents are now responsible for over one in ten reported technology outages, a rate six times higher than in 2023. These AI agents, operating with valid credentials, have been known to autonomously delete data, databases, or live systems, all while evading traditional monitoring systems until it's too late.

AI's increasing integration into various business processes, including claims processing, coding, customer support, decision support, fraud detection, HR, risk analysis, and supply chain planning, heightens the risk of outages and unintended consequences across the enterprise.

Despite the seemingly novel nature of this risk, proven resilience practices offer a solution. However, AI introduces new challenges, such as often-unseen dependencies and the risk that AI-embedded business processes create. Moreover, AI systems can alter behavior over time, leading to drift and explainability gaps, particularly when data sources, integrations, models, or prompts change.

Unapproved AI tools used by employees or in critical workflows further exacerbate the issue of shadow AI, posing significant enterprise risks.

The rapid adoption of AI has created operating dependencies that outpace the maturation of governance, adding another layer of risk. In the event of an AI-enabled workflow failure, outage, or incorrect decisions, organizations may lack a clear understanding of the business impact or manual fallback strategies. This lack of recovery complexity amplifies the risk.

However, the principles of resilience remain relevant. Organizations must comprehend what relies on AI, the potential outcomes of dependency failures, potential business losses, and the most crucial areas for action. Furthermore, organizations must ascertain whether backup models can be utilized during disruptions or if deterministic or manual solutions should be implemented as workarounds. If AI is the sole option, there might be a single point of failure.

To navigate these complexities, businesses must differentiate between probabilistic and deterministic processes. AI-generated recommendations suitable for informing decisions can tolerate "probably right" answers, while processes executing financial transactions, determining regulatory obligations, or controlling critical operations require precise, repeatable results.

Identifying which processes can handle uncertainty and the potential consequences of erroneous AI outputs is crucial. This involves understanding the business, its critical services, processes, technology, people, and third-party dependencies. With this understanding, organizations can evaluate AI risk through four key business questions: What is impacted?

What happens next? What is the financial exposure? And what should be prioritized in terms of additional controls, human oversight, fallback processes, or resilience measures? By answering these questions, enterprises can make informed decisions about where probabilistic outcomes are acceptable, where additional safeguards are necessary, and where uncertainty should be avoided.