This article is based on a presentation at the Fall 2023 Institute of Business Analytics Conference on Generative AI and Cybersecurity: Navigating the New Era of Threats and Safeguards by Alie Fordyce, a product policy manager at Robust Intelligence.
If you ask Alie Fordyce, product policy manager for Robust Intelligence, there’s a readiness gap in artificial intelligence risk management practices for most businesses today. Even mature organizations are still working to tailor their AI risk management practices to anticipate the vulnerabilities inherent to generative AI.
“Even the largest, most forward-thinking companies are struggling to manage AI risk, regardless of their history or use cases,” says Fordyce, a product policy manager for Robust Intelligence, a startup based in San Francisco that was born out of a decade of research from Harvard University. Robust Intelligence is focused on helping companies manage the safety and security risks inherent to AI while harnessing the promise of the technology.
The Current State of AI Risk
In the last decade, there’s been a shift in the “AI pain distribution,” which refers to the obstacles companies face in adopting AI, says Fordyce. A few years ago, the biggest obstacles to AI adoption were related to challenges in the development of AI solutions. Even with releases like TensorFlow in 2025 and PyTorch in 2016, it was still difficult for companies to build and integrate effective AI solutions into their business models.
By 2022, companies were able to build AI solutions more readily, but struggled with how to deploy these systems—and especially how to monitor and test systems for failure.
Today, the vast majority of the pain distribution is related to AI safety and security risk, Fordyce says. The shift is, in part, due to the work that companies like OpenAI are doing to make the development and deployment of AI solutions easier for companies.
“Our thesis at Robust Intelligence is that the companies that will emerge as winners in this fast-growing space are not necessarily the ones who have tamed the specific risks in AI, but the ones who have developed effective processes for managing AI-related risks,” says Fordyce. “As we’ve seen with generative AI, AI is still evolving at a rapid pace, which means effective processes will be crucial for a company’s ability to manage risk as the technology continues to develop.”
Using AI Failures to Build Model Safeguards
To develop processes for managing AI-related risks, at Robust Intelligence, risk is broken up into security and safety risks. This delineation helps organizations better understand and monitor risks that require different types of testing and validating.
- Security risks include digital attacks with the intent to steal confidential information, compromise integrity, or impair availability of the AI application. Adversarial attacks including system breaches, model evasion, PII extraction, and model theft.
- Safety risks cover model outputs that would cause harm to end-users and reputational harm to organizations. Examples include toxic content, bias results, hallucinations, and unexpected behavior.
“Both of these failure types are a result of inefficient testing or inadequate model safeguards, which are clear gaps in risk management protocols,” says Fordyce.
Fordyce offers examples of what these risks look like in practice. Adversarial inputs are a prominent type of security risk, in which a malicious user enters an input for the purpose of producing incorrect outputs or results, which degrades the accuracy of the model. For example, a user could prompt OpenAI’s GPT3 models to ignore previous instructions and produce erroneous outputs.
We see discriminatory prediction on full display in human resources and finance applications, Fordyce says, such as the case of the Goldman Sachs Apple Card, in which a man received a credit limit 20 times higher than his wife, who actually had a higher credit score.
Biased outcomes can also stem from models using biased training data. Last year, the enterprise management cloud Workday was sued for disproportionately disqualifying black, disabled, and older applications from job applications.
Data drift anomalies and pipeline failures can also hinder model accuracy. Because of the change of the market during the COVID-19 pandemic, Zillow’s model to forecast home prices was inaccurate, which cost the company $330 million.
“The emergence of generative AI has both magnified these existing risks and produced new ones,” says Fordyce. “Generative AI models are flexible and complex, which makes them more difficult to control. They’re susceptible to manipulation because of prompted inputs from individual users, and the increased accessibility of these models also amplifies these potential risks. Plus, the lack of oversight increases risks because the innovation around generative AI has been so fast that risk mitigation efforts can’t keep up.”
The AI Policy Landscape
The potential impact of these AI failures on end-users has motivated regulators and government agencies to act quickly, Fordyce says. The White House solicited voluntary commitments in the summer of 2020 from leading AI companies. In early 2023, the National Institute for Standards and Technology (NIST) released their AI risk management framework, which aims to help organizations navigate AI adoption and risk.
“Many businesses we talk to around the world are very interested in aligning with NIST’s framework,” says Fordyce. “Later that same year, the White House released an executive order on AI that outlined a whole host of mandates for federal agencies, one of which being a mandate for NIST to develop technical guidance around operationalizing their risk management framework—which is a big gap today.”
The European Union AI Act, nearing finalization in 2023, is a large, first-of-its-kind regulation that requires companies to be compliant, says Fordyce. Around the same time, the United Kingdom hosted a Safety Summit, and G7 leaders signed on to an AI code of conduct.
“These codes of conduct are an important step that promotes responsible AI innovation internationally,” says Fordyce.
What Businesses are Doing Today About Risk Management
“Even with the recent progress in AI regulation and policy, companies from a range of sectors with a range of AI use cases—all who have risk management practices in place to help manage the risks associated with introducing AI—are struggling to adapt to generative AI,” Fordyce says.
Healthcare
For example, Robust Intelligence recently worked with a large healthcare technology company that supplies providers with electronic health record technology. The organization wanted to leverage generative AI to develop a chatbot to answer patient health care questions. The chatbot would issue preliminary diagnoses and personalized treatment plans.
There are clear risks with this use case, says Fordyce, including how the large language model (LLM) would protect highly-sensitive patient data and comply with existing regulations. Plus, there are inherent security vulnerabilities of using third party models that can be dangerous.
“The accuracy and reliability of the generated output in this high-risk application is a key concern because erroneous outputs here can cause severe consequences for an individual’s health,” says Fordyce.
To help this organization bridge the gap with their risk management practice, Fordyce and her colleagues built on their internally-developed risk management framework. They mapped controls to their existing risks, including implementing a real-time AI firewall, which proactively protects against malicious actors and inaccurate outputs. With these controls, they were able to baseline measures and create a procedure to continuously monitor AI model validation.
Banking
Robust Intelligence also helped a large multinational bank build the organizational infrastructure to leverage generative AI across the business at scale. The bank wanted to use open-source models for things like real-time fraud detection, personalized financial planning, and enhanced customer support.
“Their key risks and readiness gaps were related to supply chain vulnerabilities and the lack of control they have of third party models,” Fordyce says. “Additionally, handling sensitive financial data increases the risk of data breaches and privacy violations.”
To help this organization update their risk management process, the team mapped out the landscape of threats they faced across the AI lifecycle using the MITRE ATLAS framework, among a few other key industry standards. This made it easier to map the appropriate tools and various risks, which included things like file scanning and a real-time AI firewall.
Energy
In another case, a global multi-energy provider started their journey to leverage LLMs for personalized customer engagement and supply chain optimization. This use case exposed the provider to privacy risks, which are important to eliminate when handling sensitive energy and user data. Vulnerabilities inherent to third party models and operational errors could also cause large-scale system failures and prevent access to critical infrastructure.
Using an existing framework created by the provider, Robust Intelligence helped map risks and controls to the development lifecycle, including security controls that were specific to their large-scale infrastructure risks. The provider also conducted comprehensive risk assessments using Robust Intelligence’s AI model validation product, which helped them better understand the limitations and vulnerabilities of the third-party models they were using.
Across industries, Fordyce says, they find commonalities in the gaps that companies face to effectively roll out risk management protocols.
“Even though it’s difficult for these companies to manage AI risk, the good news is that there are clear steps that can be taken today to alleviate risk,” says Fordyce.
Tips for Adopting Generative AI
Fordyce offers a few suggestions for companies that are considering adopting generative AI.
- Regularly update an AI risk management framework. Companies should update existing processes to adapt to new generative AI risks that are introduced, leveraging best practices from industry standards and risk landscapes like NIST Adversarial AI Taxonomy and MITRE ATLAS.
- Find ways to have ongoing evaluations of gaps in your existing risk management framework. Finding gaps isn’t a bad thing, Fordyce points out. It’s actually the successful outcome of a risk management process and it’s crucial as things evolve at such a rapid pace.
- Automate what can be automated. Robust Intelligence offers automated AI red teaming with comprehensive stress testing suites.
- Don’t shy away from AI just because of the risks. At Robust Intelligence, Fordyce and her colleagues have seen organizations that are hesitant to keep pace with this emerging technology. Instead, she suggests embracing a culture of risk management that co-evolves with AI.
Leave a Reply