Explainable Causal Reinforcement Learning for circular manufacturing supply chains with inverse simulation verification

During a deep dive into reinforcement learning (RL) optimization, the author stumbled upon a critical problem: an RL agent that performed well in simulation but failed in real-world scenarios. This failure was due to correlation-driven overfitting, where the agent learned statistical shortcuts instead of understanding the actual causal relationships within the supply chain.

This experience prompted the author to explore causal inference, counterfactual reasoning, and ultimately, Explainable Causal Reinforcement Learning (XCRL).

The author discovered that combining causal graphs with RL, and verifying the learned policies through inverse simulation, could produce agents that are not only more robust but also auditable. In circular manufacturing, auditability is vital as material flows loop back, regulations are tightening, and every decision has downstream environmental consequences. Therefore, ensuring that the decision-making processes are transparent and explainable is crucial.

The circular manufacturing environment differs significantly from standard linear supply chains. In circular supply chains, decisions about remanufacturing units impact the availability of cores for future quarters, affecting new production economics, carbon accounting, compliance, and overall sustainability. Standard model-free RL treats the environment as a black box Markov Decision Process (MDP), learning from correlations in observed trajectories without understanding the underlying causal structure.

This approach is risky in closed-loop systems where understanding the consequences of decisions is essential. This is where Causal Reinforcement Learning (CRL) comes in.

CRL addresses these challenges by embedding a structural causal model (SCM) within the learning loop. This allows the agent to reason over interventions rather than mere observations. The author presents a minimal structural causal model (SCM) for a circular manufacturing node, detailing nodes such as core inflow, remanufacturing capacity, new production, demand, and recycled mass.

The SCM outlines how these nodes interact, enabling the computation of interventional distributions (P(Y | do(A=a))) that serve as the reward signal for the RL agent. This ensures that the RL agent is not only maximizing observed rewards but also maximizing counterfactually robust rewards.

One key aspect of XCRL is explainability. By leveraging causal attribution methods, the author found that decomposing learned policies into causal pathways provides far more meaningful explanations than traditional feature importance measures. The author implemented causal path attribution using the SCM, estimating the effect of actions on target variables through specific causal pathways.

This approach provides auditable insights into the decision-making process, helping stakeholders understand why a particular action was chosen and how it influenced various aspects of the supply chain.