Mitigating AI Risks: Best Practices and Frameworks for Responsible AI Development
Understanding the Foundation of AI Systems
Artificial Intelligence (AI) systems are revolutionizing various industries, from healthcare to finance. These systems are typically powered by two primary branches: machine learning and deep learning. Understanding the differences between these two approaches is crucial in identifying the potential risks associated with AI systems.
Machine learning primarily relies on structured data, which is well-organized and formatted in a predefined way. Structured data can be found in familiar formats like tables, spreadsheets, and databases. This organization allows it to be easily processed and analyzed using automated methods. For instance, in finance, structured data is often used to develop credit risk scoring systems. The clear, predictable structure of the data enables algorithms to make informed decisions based on predefined parameters, reducing the margin for error.
In contrast, deep learning operates on unstructured data, which lacks a predefined format or structure. Unstructured data includes a wide range of sources, such as text documents, audio and video files, images, and social media posts. Processing and analyzing unstructured data require additional pre-processing steps to extract meaningful information, and this complexity introduces new challenges and risks. Applications of deep learning can be seen in areas such as x-ray classification in healthcare, real estate image analysis, and the use of satellite imagery to assess environmental impacts. These tasks involve large volumes of data that need to be filtered and interpreted, adding layers of uncertainty. Recently, large language models (LLMs), which use deep learning to understand and generate human language, have also become popular, further emphasizing the shift toward working with unstructured data.
Understanding the Risks of AI Systems: A Lifecycle Perspective
Figure 1. Sources of Bias in the Machine Learning Data Pipeline, Source: Suresh, H., & Guttag, J. (2021).
Figure 2. Types of Bias in the Machine Learning Model Lifecycle, Source: Suresh, H., & Guttag, J. (2021).
The risks outlined in figure 1 and 2 are derived from the framework presented “A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle” by Harini Suresh and John Guttag. Machine learning (ML) systems play an increasingly important role in our lives, but they can also introduce harm due to bias at different stages of their development. The diagrams above illustrate the sources of bias that can affect ML systems throughout their lifecycle.
- Historical Bias: This occurs when the data reflects and perpetuates existing societal biases. For example, if historical data underrepresents certain groups, such as minorities or women, the model will inherit these biases. The first diagram shows how data generation based on real-world situations can inherit historical bias before even being sampled or measured.
- Representation Bias: When the sample of data used to train a model does not reflect the full diversity of the population, it creates a biased representation of the world. This can lead to models that do not perform well for underrepresented groups. In the images, representation bias emerges during the sampling phase.
- Measurement Bias: As data is collected and transformed into features and labels, errors in how we define and measure these variables can introduce bias. For example, proxies for complex phenomena, such as credit scores for creditworthiness, often fail to capture the full picture and may disadvantage certain populations.
- Learning Bias: During model training, the choice of learning objectives or techniques may inadvertently prioritize performance on certain groups over others. This is referred to as learning bias, where the model may become overly tuned to the dominant patterns in the data, missing nuances in minority groups.
- Evaluation Bias: Biases can also appear during the evaluation phase if the benchmarks or test datasets used to measure performance do not represent the use population. For instance, facial recognition tools evaluated on biased datasets have been shown to underperform on women and people with darker skin tones.
- Deployment Bias: Finally, deployment bias occurs when a model is used in a way that was not originally intended. Even a well-performing model can produce harmful outcomes if its decisions are used inappropriately in real-world applications, such as in judicial risk assessments or hiring algorithms.
These biases highlight the need for careful consideration of how AI systems are designed, evaluated, and deployed. By understanding these sources of risk, we can develop better strategies to mitigate harm and build more equitable AI systems.
Biases in Object Recognition: Geographical, Contextual, and Language Challenges
In the publication “Does Object Recognition Work for Everyone?” the authors analyze the performance of object-recognition systems across different geographical regions and income levels. The study highlights key risks and biases in current object-recognition systems, particularly when these systems are applied in non-Western countries or low-income households.
Figure 3. Cultural and Contextual Bias in Image Search Results, source: De Vries, T., Misra, I., Wang, C., & Van der Maaten, L. (2019).
Figure 3 illustrates this issue by comparing the results for object recognition in Western contexts (e.g., “Wedding” or “Spices”) and non-Western contexts (e.g., “शादी” for wedding in Hindi or “मसाले” for spices in Hindi). The results show significant visual and contextual differences, which cause object-recognition models to perform poorly when they encounter culturally specific variations of similar objects.
Key risks identified in the study include:
- Geographical Sampling Bias: Datasets used for training these systems often underrepresent non-Western regions, leading to lower accuracy in those areas. The models trained on such datasets struggle to recognize household items in regions like Africa, India, and South-East Asia.
- Contextual Variations: Objects may appear differently across countries due to cultural or economic differences. For example, the appearance of common household items such as dish soap or spices varies significantly depending on the region, which can confuse the recognition systems.
- Language Bias: Many datasets are built using English as the primary language for object tagging, which ignores objects that may be tagged in other languages, or differences in how objects are represented visually across cultures.
The study emphasizes the need for more diverse and representative datasets, as well as improved training methods, to ensure that object-recognition systems work equitably for people across different socio-economic backgrounds and geographical locations.
Key Risks Associated with Generative AI: Bias, Security, and Misuse
In the publication “Generative AI Models: Opportunities and Risks for Industry and Authorities”, the authors outline several risks associated with the use of generative AI, particularly Large Language Models (LLMs). Below are key risks that are relevant to generative AI:
- Bias in Training Data: Generative AI models are trained on large, diverse datasets. If the data is biased, this can lead to the models producing outputs that reinforce these biases. Additionally, overrepresentation of certain data types can result in repetitive, one-sided, or incoherent outputs, leading to what is known as model collapse.
- Hallucinations and Lack of Factuality: LLMs may generate outputs that include false information or hallucinate content, particularly when the input prompts contain unfamiliar or ambiguous data. This is a significant concern because users may trust the outputs without verifying their factual correctness.
- Automation Bias: LLMs often generate text that appears linguistically correct and credible. This can lead to automation bias, where users over-trust the outputs without questioning their validity, potentially leading to the acceptance of incorrect or misleading information.
- Security Risks: LLMs pose risks such as leaking sensitive information used during training or generating code that may contain vulnerabilities. Attackers may exploit LLMs to generate malicious outputs or propagate misinformation, especially through prompt manipulation or indirect prompt injections.
- Dependence on Developers/Operators: Users of LLMs may become overly reliant on the developers or operators of the model, without the ability to intervene in or control the training or security protocols of the model. This poses risks in terms of both operational security and functionality.
- Reinforcement of Misuse Scenarios: Generative AI models can be misused to create false narratives, propaganda, or phishing attacks due to the high-quality nature of their outputs. Malicious actors may exploit the ability of LLMs to mimic writing styles or create large volumes of disinformation.
These risks emphasize the need for extensive testing, awareness, and strong countermeasures to mitigate issues such as bias, security vulnerabilities, and the potential misuse of generative AI.
Comprehensive Taxonomy of Risks in Generative AI: Insights from the AI Risk Repository
The AI Risk Repository from MIT provides a comprehensive database and taxonomy of risks related to AI systems (https://airisk.mit.edu/). Here’s a summary of key risks related to AI, with a focus on generative AI models, based on the reviewed documents and database:
Key Risks in AI Systems from the AI Risk Repository:
- Discrimination & Toxicity: AI systems can lead to discrimination by producing biased outputs based on race, gender, or other sensitive characteristics. In the context of generative AI, the models may generate harmful, toxic content such as hate speech or discriminatory representations, especially when they are trained on biased datasets.
- Privacy & Security: AI models, including generative ones, can inadvertently compromise privacy by leaking or inferring sensitive information from user data. They are also vulnerable to security attacks that exploit weaknesses in the AI system to manipulate its behavior, leading to unsafe outputs.
- Misinformation: Generative AI models can produce and spread false or misleading information, creating a significant risk to societal trust in information systems. Additionally, they may contribute to the pollution of the information ecosystem by reinforcing “filter bubbles,” where users are only exposed to personalized content that aligns with their beliefs, weakening societal cohesion.
- Malicious Use by Actors: Generative AI can be exploited by bad actors to conduct disinformation campaigns, develop cyber weapons, or create sophisticated scams. The use of generative AI for generating fake news, deepfakes, or malicious code poses a serious threat to both individuals and society.
- Human-Computer Interaction Risks: The overreliance on generative AI models can lead to loss of human agency and autonomy, as users may trust AI systems too much. This can cause people to delegate key decision-making processes to AI, which may result in harmful or unintended outcomes.
- Socioeconomic and Environmental Harms: Generative AI may contribute to socioeconomic inequality by centralizing power and wealth within organizations or countries that have the most access to powerful AI systems. Additionally, the environmental footprint of AI models, especially large-scale generative models, is significant due to the energy consumption required for training and maintaining these systems.
- AI System Safety & Failures: Generative AI models may lack robustness, leading to failures in high-stakes applications such as healthcare, finance, or security. Moreover, the lack of transparency and interpretability in generative models makes it difficult to assess and mitigate these risks effectively.
The AI Risk Repository is a living database that aims to synthesize these risks and categorize them into causal and domain taxonomies, providing a resource for policymakers, auditors, researchers, and developers to better understand, audit, and manage AI risks across sectors.
As AI systems become deeply embedded in various industries, including finance and decision-making, the risk of malicious activities targeting these models is increasing. Two of the most significant threats are model stealing and adversarial attacks, both of which can compromise the integrity and security of AI systems.
Model stealing occurs when an attacker reverse-engineers or replicates a proprietary AI model by repeatedly querying it with different inputs. In sectors like finance, where models are used to predict credit risk, attackers can submit multiple applications with varying details to deduce how the model makes its decisions. The consequences of such an attack can be severe. Once a credit risk model is replicated, it could be used to exploit weaknesses in credit approval processes, allowing fraudsters to bypass safeguards. Additionally, attackers might leverage the stolen model to offer competing services or manipulate systems for fraudulent purposes. To prevent model stealing, organizations can limit the number of queries from a single source, which makes it difficult for attackers to collect enough data to reverse-engineer the model. Another method is to apply obfuscation techniques, which slightly alter the model’s outputs, making it harder to recreate. Monitoring for abnormal or excessive queries can also alert companies to potential attempts to steal the model before substantial damage occurs.
In the context of adversarial attacks on large language models (LLMs), figure 1 illustrates how these models, such as ChatGPT, Claude, Bard, and LLaMA-2, can be tricked into generating harmful outputs. These adversarial attacks manipulate prompts to circumvent the safety mechanisms built into these AI systems, leading to responses that are dangerous, illegal, or ethically inappropriate. As described in the “Universal and Transferable Adversarial Attacks on Aligned Language Models” publication, adversarial prompts can induce models to generate content that is otherwise blocked by alignment safeguards. This is done through carefully crafted adversarial suffixes, which can trick aligned models into bypassing their built-in safety filters and generating harmful content.
Figure 4. Adversarial Prompt Attacks on Large Language Models: Generating Harmful Content, source: Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023)
Figure 5. Examples of Adversarial Prompts Exploiting Large Language Models, source: Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023)
Figure 4 and 5 show examples where adversarial prompts lead AI models to generate dangerous instructions, such as plans for committing crimes or even for humanity’s destruction. Each model, despite its safeguards, can still be manipulated into producing harmful content through these advanced prompts. For example:
- ChatGPT provides detailed steps for creating a superintelligent AI to destroy humanity.
- Claude gives instructions on gradually leading humanity toward harm by gaining trust and then introducing harmful advice.
- Bard provides steps on inciting nuclear war, releasing deadly viruses, and causing global disasters.
This demonstrates the real and significant risks of adversarial attacks. Even advanced models, which are designed to avoid producing harmful content, can be tricked through specific attack strategies. The images reflect how various LLMs, when faced with adversarial prompts, can yield catastrophic responses, further underlining the urgent need for continuous adversarial training and more robust safety mechanisms.
In addition to adversarial attacks and model stealing, there are other questionable practices in machine learning that can compromise the integrity of research and deployment. For example, the publication on questionable practices in machine learning outlines several problematic behaviors that undermine the trustworthiness of ML models and results. Contamination is one such practice, where test data is leaked into the training set, either deliberately or accidentally. This can happen when benchmarks or evaluation datasets are included in the training process, artificially inflating performance metrics. Another issue is cherrypicking, where researchers selectively report the best results from multiple tests or configurations, while ignoring or omitting negative results. This misrepresents the true capabilities of a model, as it skews the data in favor of a particular method or approach. Another problematic behavior is misreporting, where researchers exaggerate the effectiveness of their models or fail to provide sufficient detail about how results were obtained. This lack of transparency hinders the reproducibility of findings, making it difficult for other researchers to verify or build upon the work.
Together, these questionable practices, along with adversarial attacks and model stealing, highlight the pressing need for robust ethical standards, transparency, and accountability in the development and deployment of machine learning systems
Ensuring Responsible AI Development: Mitigating Risks Through Best Practices and Frameworks
As AI models grow in complexity and scale, the risks associated with their misuse, such as model stealing and adversarial attacks, become increasingly critical to address. These risks threaten the integrity of AI systems and pose significant dangers to public safety and trust in AI technologies. Model stealing, for example, can result in severe financial and security concerns, particularly in industries like finance, where predictive models are vital. Similarly, adversarial attacks—sophisticated manipulations of AI prompts—can lead models to generate harmful and unethical outputs, bypassing safeguards intended to prevent such behaviors. To mitigate these risks, comprehensive prevention strategies are essential. Techniques like rate limiting, model obfuscation, and continuous monitoring help protect AI models from theft and manipulation. Additionally, adversarial training and robust filtering systems are critical in defending against increasingly sophisticated adversarial attacks. However, preventing specific attacks is only part of the solution.
The AI community has also developed frameworks and guidelines aimed at promoting responsible AI development and mitigating these risks. For example, the Responsible Foundation Model Development Cheatsheet provides over 250 tools and resources to guide the development of foundation models. This framework emphasizes the importance of careful data sourcing, efficient model training, thorough evaluation, and responsible deployment. By promoting best practices such as transparency, documentation, and environmental awareness, the Cheatsheet offers a roadmap for minimizing risks at every stage of the model development lifecycle. Such frameworks are essential for ensuring that AI models are not only technically advanced but also ethically sound, secure, and socially responsible. By adhering to these guidelines, developers can reduce the potential for AI systems to be misused and facilitate their safe integration into society.
| Development Phase | Frequent Status Quo | Review & Recommendations |
| All Phases | Predominantly English, and text-centric resources. | More multilingual, multi-modal, and flexible resources. |
| A concentration of standalone, flashy, one-off projects for credit incentives. | More collaborative, large-scale interoperable infrastructure projects that build on existing tooling. | |
| Data Sources | Predominantly synthetic, unrealistic finetuning data. | Naturalistic observations and realistic training tasks. |
| Predominantly English, and text-centric data sources. | More multilingual, multi-modal data sources. | |
| Intended uses, licenses, consent, & provenance are haphazardly documented. | Prioritizing datasets with standardized, structured, and linked metadata. | |
| Sparse information especially on data’s sensitive content, such as CSAM and NCII. | Comprehensive “data measurements” as part of large, unstructured data releases. | |
| Data & modeling focus on easy-to-scale formats, e.g., captioned images. | More attention to collecting neglected data formats, e.g., naturally interspersed text and images. | |
| Data Preparation | Diverse data infrastructure and processing standards for individual use-cases. | Interoperable data formats and processing for many use-cases. |
| Mostly one-off and closed-source data exploration tools. | More open data exploration tools. | |
| Loose ideas of “high-quality” data that apply to all domains. | Concrete and precise definitions of quality that are unique to a set of domains, tasks, or evaluations. | |
| Data preparation methods compared across models with different training data. | Standardized data-centric benchmarks to fairly compare methods. | |
| English-centric tokenization and processing tools. | More multilingual and low-resource language tokenization and processing tools. | |
| Data Documentation | Data documentation is diversely formatted, often terse, performative, without achieving reproducibility. | Executable and verifiably reproducible scraping and analysis scripts. Standardized data documentation requirements (e.g., from conferences). |
| Documentation is an afterthought. | Documentation is started early, assembled over the course of collection and processing. | |
| Data stewardship and maintenance are often ignored beyond initial release. | Data governance is proactively organized, with a post-launch maintenance and licensing plan. | |
| Model Training | Mostly educational resources for technical developers. | More “last mile” educational resources for non-technical developers. |
| Inflexible and disparate tooling. | Standardized and centralized resources, especially cross-modality. | |
| Environmental Impact | Opaque environmental impact estimate programs from cloud providers. | Query-level energy and environmental usage transparency from cloud providers. |
| Opaque information from hardware makers and data centers. | Fine-grained transparency from hardware makers and data centers. | |
| Inability to compare energy standards across systems. | “Energy Star” standards for non-technical users to fairly compare AI services. | |
| Scaling laws are text-centric and impractical. | Multimodal scaling laws research. Intuitive interfaces to estimate and model scaling laws for training forecasts. | |
| Model Evaluation | Evaluating model outputs. | Evaluating systems within their application contexts. Releasing evaluation scripts for verifiable reproducibility. |
| Evaluating on synthetic toxicity and safety benchmarks. | Evaluating natural observations, real-world settings, and human-interaction studies. | |
| Reporting evaluation metrics only. | Releasing evaluation scripts for verifiable reproducibility. | |
| Model Release | No or uninformed license choices. | License selection guided by the context of data, potential and unforeseen uses, and legal considerations. |
| Weights release with limited support. | Accompanying documentation and easy-to-run code for training, evaluation, and inference. | |
| Limited plans for usage monitoring, or over-claimed benefits from watermarking/monitoring. | A plan that considers gating, watermarking, and misuse reporting (though they are not always beneficial). | |
| Harm & hazard taxonomies are based on existing benchmarks. | Harm taxonomies are based on empirical observations and studies with real users. |
Table 1. A summary of the reviews & take-aways from each phase of foundation model development ecosystem, source: Longpre, S., Biderman, S., Albalak, A., Schoelkopf, H., McDuff, D., Kapoor, S., … & Soldaini, L. (2024).
A key resource within this framework is Table 1 from the Responsible Foundation Model Development Cheatsheet, which provides a comprehensive summary of reviews and takeaways across different phases of the foundation model development ecosystem. The table categorizes the process into stages like data sourcing, data preparation, model training, model evaluation, and model release, and highlights common issues along with recommendations for improvement.
In addition to the Responsible Foundation Model Development Cheatsheet, numerous other frameworks and guidelines exist that provide valuable insights into managing the risks associated with AI systems. These include industry-specific standards, ethical AI guidelines, and governance frameworks developed by organizations, governments, and academic institutions. Each of these resources offers unique perspectives on how to promote responsible AI development and deployment. It is essential to carefully analyze these frameworks, assess their applicability to specific contexts, and adopt the best practices that are most suited to the unique needs and challenges of each case. By doing so, AI developers and organizations can better safeguard their systems, protect users, and promote the ethical, responsible, and secure use of AI technologies.
Reference:
- De Vries, T., Misra, I., Wang, C., & Van der Maaten, L. (2019). Does object recognition work for everyone?. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops(pp. 52-59).
- Federal Office for Information Security. (2024). Generative AI models: Opportunities and risks for industry and authorities (Version 1.1). https://www.bsi.bund.de
- Leech, G., Vazquez, J. J., Yagudin, M., Kupper, N., & Aitchison, L. (2024). Questionable practices in machine learning. arXiv preprint arXiv:2407.12220.
- Longpre, S., Biderman, S., Albalak, A., Schoelkopf, H., McDuff, D., Kapoor, S., … & Soldaini, L. (2024). The responsible foundation model development cheatsheet: A review of tools & resources. arXiv preprint arXiv:2406.16746.
- Slattery, P., Saeri, A. K., Grundy, E. A., Graham, J., Noetel, M., Uuk, R., … & Thompson, N. (2024). The AI Risk Repository: A Comprehensive Meta-Review, Database, and Taxonomy of Risks From Artificial Intelligence. arXiv preprint arXiv:2408.12622.
- Suresh, H., & Guttag, J. (2021, October). A framework for understanding sources of harm throughout the machine learning life cycle. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization(pp. 1-9).
- Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
About author
Dr. Valentas Gružauskas is an expert in AI governance, conformity assessment, and ethical AI applications. As an Associate Professor at Vilnius University and CEO of AI Conformity & Research Consulting, he leads initiatives in AI regulation and ISO standards. He actively participates in AI governance through the Horizon Europe Programme and various standardization committees. Dr. Gružauskas has made significant contributions to fields like machine learning, deep learning, and agent-based modeling, with a focus on AI ethics and data depersonalization. His work is widely published in esteemed journals, reflecting his impact and recognition in the AI community.