October 4, 2024

The Critical Role of Model Explainability in Mitigating AI Risks and Ensuring Compliance

Understanding the Importance of Model Explainability

Model explainability is a key aspect of modern artificial intelligence (AI) systems, especially when it comes to determining potential biases and ensuring that the system is functioning as intended. Understanding how a model arrives at its decisions is not only critical for the model’s integrity but also legally mandated, particularly by regulations such as the European Union’s AI Act. The AI Act requires a transparent and explainable AI system to safeguard against issues like bias, fairness, and discrimination, particularly when dealing with sensitive matters such as healthcare, financial services, or justice.

Figure 1. Incidence of AI failure modes (n=106), Source: Holweg, Younger, and Wen (2022)

AI incidents are increasingly becoming a concern for organizations worldwide. A survey conducted by Holweg, Younger, and Wen (2022) in The Reputational Risks of AI provides insights into this. In their study of 106 companies, they identified key areas where AI systems failed:

50% of failures were related to privacy concerns, including issues with data protection and cybersecurity.
31% were related to bias, where AI systems were found to make unfair or discriminatory decisions.
14% of failures were due to replication issues, where results could not be reliably reproduced.
5% of respondents indicated that the failure was unclear, showing the complexity in diagnosing AI problems.

Understanding model explainability is crucial for addressing these issues. For instance, when an AI system fails due to bias, it’s essential to investigate whether the failure arose from the data itself or from the decision-making process within the model. The same can be said for privacy-related issues, where ensuring that the model is only using the required data in a secure manner needs to be part of the explainability process.

As AI systems grow more powerful, their complexity often increases, making them harder to interpret. That’s where explainability frameworks come into play, helping organizations mitigate risks and maintain control over their AI systems. Holweg, M., Younger, R., & Wen, Y. (2022) shed light on these risks and emphasize the necessity for transparency. Their article, The reputational risks of AI, published in California Management Review Insights, highlights the importance of explainability as a critical component for safeguarding AI systems from reputational and operational harm.

In the following chapters, we will dive deeper into the methods of explainability across different AI applications, starting with tabular data.

Explaining Tabular Data in AI Models

When it comes to machine learning models, particularly in finance, tabular data plays a central role in decision-making processes such as credit risk scoring. Models used in this context assess an individual’s likelihood of defaulting on a loan, and transparency in how these decisions are made is crucial. One effective tool for explaining such models is SHAP (SHapley Additive exPlanations), which helps illustrate the influence of individual features on the model’s output.

Figure 2. SHAP Decision Plot for Credit Risk Model

Figure 2 is an example of a SHAP decision plot for a credit risk model. The y-axis lists various features used by the model, such as grade, loan term, homeownership status, and interest rate—all factors that influence the model’s prediction of whether someone might default on a loan. The x-axis represents the probability of default, with values ranging from 0 (low risk) to 1 (high risk). Each line traces the contribution of individual features to the model’s prediction for a specific instance.

In this plot, the lines show how features push the prediction toward a higher or lower probability of default. For instance, a high interest rate might increase the likelihood of default, while being a homeowner could lower it. By visually displaying how much each feature contributes to a decision, this type of explanation improves transparency in model-driven decisions.

In a credit risk scenario, this kind of interpretability is invaluable. Financial institutions can better understand why a loan was approved or denied, helping to ensure the model is both accurate and fair. Additionally, explaining model outputs to clients or regulators becomes easier when you can pinpoint which features influenced the final decision.

Counterfactual Explanations

In addition to SHAP plots, another important method for explaining AI models is through counterfactual explanations. While SHAP shows the contribution of various features, counterfactual explanations focus on the “what if” aspect of model decisions. They answer questions like, “What would need to change for a different outcome to occur?”

For instance, imagine a scenario where a customer’s loan application is denied. A counterfactual explanation might reveal that if the applicant’s credit score were just 20 points higher, they would have been approved for the loan. This type of explanation doesn’t just clarify what led to the decision—it also provides actionable insights into what could change to affect the outcome.

These explanations offer substantial benefits, especially in finance. They help customers understand the reasons behind a decision and offer guidance on how to improve their chances in the future. For instance, by knowing that increasing their credit score or reducing their outstanding loans could improve their risk profile, customers are given concrete steps to improve their financial health.

Moreover, counterfactual explanations are increasingly important for complying with regulations that require transparency in decision-making processes. Financial institutions often face scrutiny from regulators, and counterfactuals can help demonstrate that decisions are made fairly and transparently. Additionally, these explanations are a valuable tool for identifying any biases or dependencies within the model, such as undue reliance on factors like zip code or income, which could lead to unfair outcomes.

Overall, both SHAP decision plots and counterfactual explanations are essential tools for improving the transparency and trustworthiness of machine learning models, particularly in high-stakes domains like credit risk assessment. They allow financial institutions to better explain their models’ predictions, comply with regulations, and build trust with customers.

In the following chapter, we will explore explainability in the context of image classification models, which pose their own set of unique challenges.

The Importance of Explainability in Image Classification

In AI systems, particularly in image classification tasks, explainability is just as crucial as in any other domain, perhaps even more so given the often opaque decision-making processes behind complex image recognition models. While models for tabular data often rely on numerical inputs like credit scores or loan terms, image classification systems process vast amounts of visual data. Without explainability mechanisms, it can be difficult, if not impossible, to determine how the model arrived at its final decision.

One of the most famous examples of explainability in image classification comes from a study by Ribeiro, Singh, and Guestrin (2016). The experiment involved classifying images of huskies and wolves, and the model made a critical mistake—classifying a husky as a wolf. Upon closer inspection, the error wasn’t due to the characteristics of the husky itself but to irrelevant background information.

Figure 3. Husky vs. Wolf Misclassification Example.

(a) is the original image, where a husky is incorrectly classified as a wolf.

(b) is the explanation of the model’s decision. Interestingly, the model didn’t focus on the husky’s features (such as its fur or face) to make its classification. Instead, it paid more attention to the snowy background.

This misclassification demonstrates a significant problem: the model wasn’t learning to identify wolves or huskies based on their distinguishing characteristics. Instead, it relied on the snow as a key feature for identifying wolves, which, while possibly correlated with images of wolves, is clearly an irrelevant factor when it comes to accurately identifying animals. This serves as a cautionary tale of why explainability is vital for image classification models, as it can uncover biases that may be hiding in plain sight.

Non-compliance with the AI Act

This type of failure also illustrates a breach of transparency and accuracy requirements as outlined by the EU AI Act. The Act emphasizes the importance of ensuring that AI systems meet performance standards and make decisions that are consistent with their intended purpose. In this case, the model’s failure to correctly identify the husky is a clear example of non-compliance. An AI system that incorrectly classifies images based on background noise rather than actual subject features could have severe unintended consequences in high-stakes areas like medical imaging, security, or autonomous vehicles.

Moreover, explainability mechanisms allow for immediate diagnosis of why the model failed. By showing that the model’s decision was based on snow rather than animal features, developers can retrain the model to focus on relevant aspects, such as the animal’s fur pattern, ears, or eyes, thereby improving its accuracy and compliance with regulatory requirements.

This example underscores the need for explainability in image classification tasks. Without it, models could make decisions based on irrelevant factors, leading to erroneous outcomes and violating regulatory standards.

In the following chapter, we will discuss explainability challenges for large language models (LLMs) and the role of Retrieval-Augmented Generation (RAG) in maintaining explainability.

Enhancing Explainability in LLMs with RAG and CoT

Large Language Models (LLMs) have become an integral part of modern AI applications, but their complexity often makes it difficult to fully understand or control how they generate responses. One of the main challenges with LLMs is that they can sometimes produce hallucinations, or responses that are not grounded in factual or real-world information. To mitigate these risks and enhance transparency, a framework known as RAG (Retrieval-Augmented Generation) has emerged as a solution.

Figure 4. Basic Structure of a Retrieval-Augmented Generation (RAG) Framework

The RAG framework integrates LLMs with external data retrieval mechanisms to ground the model’s responses in relevant, factual information. The figure above illustrates how this process works:

A question is submitted by a user to the AI system.
The RAG model parses the question and sends it to a content database to retrieve relevant information (step 3).
Once the relevant documents or data are retrieved, they are combined with the user’s question (step 4) and then passed to the LLM.
The LLM processes this combination to generate a more accurate and contextually relevant response (step 5).
Finally, the answer is provided to the user (step 6).

By incorporating external data retrieval into the process, RAG ensures that the model’s responses are grounded in real, verifiable information. This helps avoid common issues with LLMs, such as producing inaccurate or overly generic answers, and improves the overall transparency of the system.

Explainability with RAG

RAG enhances explainability by explicitly tying the model’s outputs to factual sources. In a traditional LLM, it’s difficult to trace the reasoning behind a particular answer, leading to concerns about whether the model is producing reliable or supported results. However, with RAG, the model is designed to retrieve relevant documents or information before generating its response. This grounding of answers in external data improves transparency and makes it easier for users to trace the source of the information.

For instance, if an LLM is answering a question about a complex legal or medical topic, RAG ensures that the model looks at appropriate documents or datasets to form its answer. This approach not only increases the trustworthiness of the response but also allows for verifiability—a user can check the sources used to generate the final answer, which is critical for ensuring compliance with regulatory standards like the EU AI Act.

Chain of Thought Prompting

Another method to improve explainability in LLMs is through Chain of Thought (CoT) prompting, which encourages the model to reason through its answers step by step. In this approach, the model is instructed to break down its thought process into smaller, sequential steps, allowing users to understand the reasoning behind each part of the response.

For example, if an LLM is asked to solve a mathematical problem, CoT prompting ensures that the model explains how it arrived at each intermediate result, instead of just giving the final answer. This approach not only enhances explainability but also helps prevent errors in the final output, as each step can be verified or corrected.

While CoT and RAG can be used independently, they also complement each other. RAG helps ensure that the information provided is factually correct, while CoT prompting provides insight into the model’s reasoning process. Together, they create a more transparent and interpretable AI system that users can trust.

Incorporating mechanisms like RAG and Chain of Thought prompting into LLM systems is essential for maintaining control and explainability. These techniques help address some of the key challenges in LLMs, such as hallucinations and unverifiable responses, by grounding outputs in factual data and revealing the model’s internal reasoning. As LLMs continue to be deployed in high-stakes domains like law, medicine, and finance, ensuring explainability through frameworks like RAG will become increasingly critical for both regulatory compliance and user trust.

Ensure Trust and Transparency in AI Through Explainability

Explainability is more than just a technical feature; it is a crucial element in ensuring trust, accountability, and compliance in AI systems. As AI technologies increasingly influence high-stakes domains such as finance, healthcare, and governance, understanding how these systems make decisions becomes essential.

Throughout this post, we’ve explored how explainability helps us uncover the inner workings of various types of AI models, from credit risk scoring in tabular data models to image classification systems and large language models (LLMs). In each case, explainability tools—such as SHAP decision plots, counterfactual explanations, and RAG (Retrieval-Augmented Generation)—have proven to be invaluable for enhancing transparency, ensuring model fairness, and providing clear justifications for decisions.

The importance of explainability extends beyond developers and regulators—it also benefits the end users of AI systems. When AI systems provide clear, understandable reasoning behind their decisions, users can gain greater trust in the technology. Whether it’s a customer wanting to understand why their loan was rejected, a doctor relying on AI to assist with a medical diagnosis, or an organization deploying AI for automated processes, explainability helps bridge the gap between technical complexity and real-world accountability.

Finally, transparency is not just a best practice—it is increasingly becoming a regulatory requirement, particularly under frameworks such as the EU AI Act. Ensuring that AI systems are explainable is critical for maintaining compliance and mitigating risks. It helps organizations proactively manage AI’s societal and ethical implications, avoiding potential biases or unintended harm.

In summary, as AI continues to evolve and become more integral to our lives, ensuring that these systems are explainable and transparent is not optional—it is essential. By providing both users and developers with insight into how AI models operate, we can create more reliable, trustworthy systems that align with societal values and regulatory standards.

Reference:

Cho, A., Kim, G. C., Karpekov, A., Helbling, A., Wang, Z. J., Lee, S., … & Chau, D. H. (2024). Transformer Explainer: Interactive Learning of Text-Generative Models. arXiv preprint arXiv:2408.04619.
DW, G. D. A. (2019). DARPA’s explainable artificial intelligence program. AI Mag, 40(2), 44.
Holweg, M., Younger, R., & Wen, Y. (2022). The reputational risks of AI. California Management Review Insights.
Mazzini, G., & Bagni, F. (2023). Considerations on the regulation of AI systems in the financial sector by the AI Act. Frontiers in Artificial Intelligence, 6, 1277544.
Molnar, C. (2020). Interpretable machine learning. Lulu. com.
Panigutti, C., Hamon, R., Hupont, I., Fernandez Llorca, D., Fano Yela, D., Junklewitz, H., … & Gomez, E. (2023, June). The role of explainable AI in the context of the AI Act. In Proceedings of the 2023 ACM conference on fairness, accountability, and transparency(pp. 1139-1150).
Quinn, H. O. B., Sedky, M., Francis, J., & Streeton, M. (2024). A Literature Review of Explainable Tabular Data.
Rai, D., Zhou, Y., Feng, S., Saparov, A., & Yao, Z. (2024). A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646.
Wu, X., Zhao, H., Zhu, Y., Shi, Y., Yang, F., Liu, T., … & Liu, N. (2024). Usable XAI: 10 strategies towards exploiting explainability in the LLM era. arXiv preprint arXiv:2403.08946.
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., … & Du, M. (2024). Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2), 1-38.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). ” Why should i trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining(pp. 1135-1144).
Jager, J. (2021, August 6). Question answering through retrieval augmented generation. Medium. https://jettro.dev/question-answering-through-retrieval-augmented-generation-9b54806c214e
DeSimone, H., & Leon-Espinosa, M. (2024, March). Explainable AI: The Quest for Transparency in Business and Beyond. In 2024 7th International Conference on Information and Computer Technologies (ICICT)(pp. 532-538). IEEE.
Nisevic, M., Cuypers, A., & De Bruyne, J. (2024, June). Explainable AI: Can the AI Act and the GDPR go out for a date?. In 2024 International Joint Conference on Neural Networks (IJCNN)(pp. 1-8). IEEE.
Sithakoul, S., Meftah, S., & Feutry, C. (2024, July). BEExAI: Benchmark to Evaluate Explainable AI. In World Conference on Explainable Artificial Intelligence(pp. 445-468). Cham: Springer Nature Switzerland.

About author

Dr. Valentas Gružauskas is an expert in AI governance, conformity assessment, and ethical AI applications. As an Associate Professor at Vilnius University and CEO of AI Conformity & Research Consulting, he leads initiatives in AI regulation and ISO standards. He actively participates in AI governance through the Horizon Europe Programme and various standardization committees. Dr. Gružauskas has made significant contributions to fields like machine learning, deep learning, and agent-based modeling, with a focus on AI ethics and data depersonalization. His work is widely published in esteemed journals, reflecting his impact and recognition in the AI community.

AICR Consulting