This article is based on a presentation at IBA’s Analytics Conference on Cybersecurity given by Hyrum Anderson, Distinguished Engineer at Robust Intelligence.
If you ask Dr. Hyrum Anderson, Artificial Intelligence (AI) is eating the world.
AI capabilities are being adopted at a rapid pace. According to Anderson, Distinguished Engineer at the machine learning integrity platform company Robust Intelligence, a recent survey by McKinsey of over 2000 executives showed that nearly half had adopted AI in at least one function within their organization. This may be an indication that companies are delivering on their AI ambitions: an earlier analysis of earnings reports by CB Insights showed 200+ mentions of AI in earnings calls.
“We’re seeing a near doubling in revenue from machine learning software every 18-24 months,” Anderson said. “If you do the math, that’s a six-times increase in five years. The growth in the market for AI may only be outpaced by its hype factor.”
While AI is in some sense eating the world, it also regularly fails. Its unique failure modes produce risks that are often not well considered, Anderson said, which makes managing that risk one of the biggest challenges for companies that want to adopt and embrace AI in a safe way. “Today, for most companies it’s largely unmanaged risk”, says Anderson. In his session during the 2022 Conference on Analytics for Cybersecurity, sponsored by the Institute for Business Analytics at the Kelley School of Business, Anderson detailed these risks and ways to manage them.
What’s the risk?
There are a number of potential risks associated with AI, Anderson said, including unintentional model failures, intentional failure modes, and vulnerabilities in machine-learning models. Unintentional model failures may be found when machine learning is biased, when there are broken data pipelines, or when the model is presented with a corner case. Unmanaged, these risks can have cause financial and reputational harm, and provide bad outcomes for consumers.
One example of unintentional failure mode: the New York State Department of Financial Services investigated Goldman Sachs, the operator of Apple cards, for an event in which a husband was granted a credit limit 20 times higher than his wife’s. The two had no difference in asset status, so the model had failed to produce a fair outcome, Anderson said. Although courts found no intentional bias in the organization, it presented a shameful outcome for the woman in question, and presented unnecessary public theater for the company.
Since AI can behave poorly under some conditions, this presents a vulnerability that can be exploited by an attacker. In the case of these intentional failure modes, a bug is discovered in a model cannot be easily patched: it is impossible for the machine to “unlearn” a vulnerability. Instead, one must design models that are robust from the outset. Since these failure modes can be exploited by adversaries, it is certainly good practice to discover and mitigate them before deployment, Anderson said.
To demonstrate the vulnerabilities in machine-learning models that can be exposed by users and adversaries, Anderson gave the example of the Twitter chatbot introduced by Microsoft in 2016, named Tay, which was intended to impersonate a “spunky teenage personality.” The bot ended up becoming offensive and inappropriate, and generated false content. Six years later, Meta experienced a similar problem when pranksters caused its “Blender Bot 3” to make anti-Semitic statements and repeat conspiracy theories.
“The point here is that these attacks can be very low sophistication,” Anderson said. “Those attacking these systems do not necessarily have a lot of machine learning knowledge. But the bottom line is that low sophistication does not imply low impact.”
How to manage machine-learning risks
Anderson emphasized the importance of managing AI risk not only to avoid reputational damage, but to protect individual people.
“Operational risk is about managing, deploying, and maintaining models that function correctly,” Anderson said. “Security risks consider a motivated adversary. And responsible AI risk consider safe and fair outcomes for models that can impact people’s lives.”
Anderson described an ongoing risk management framework (RMF) created by the National Institute of Standards and Technology (NIST) that includes governing, mapping, measuring, and managing AI risks. Describing a non-binding voluntary process, getting started with the RMF may begin by mapping the failure modes present all along the machine learning model lifecycle to business impact. In the measuring process, the identified risks are assessed and tracked, and in management, these risks are prioritized and acted upon based on their projected impact.
Governance, the centerpiece in the NIST’s AI Risk Management Framework, influences each of the steps. Governance, Anderson said, ensures that there is a culture of risk management in place within organizations and provides a foundation for working iteratively through the rest of the risk management process. This culture changes security from a check-the-box exercise that might imply that one is “done” to a set of rituals that ensures that AI risk management is an ongoing activity, part of everyone’s everyday role.
Signs that AI governance is universally increasing, Anderson said, include the rise of AI “Red Teams.” AI Red Teams conduct 3rd party, internal testing in partnership with organizations to ensure that machine learning models are secure, unbiased, and performant.
Anderson also described an acceleration in the pace of AI regulation, with governing bodies like the European Union creating the AI Act to provide requirements across industries that they implement risk-based policies to manage their AI systems.
The bottom line, Anderson said, is that adopting AI means adopting AI risk.
“There are too few companies today who are making serious investments to manage that risk,” Anderson said. “There needs to be a lot more research on how to do this effectively.”
The good news? There’s plenty of room for growth—and new talent—in the world of AI.
“AI risk management is a big space,” Anderson said. “This new frontier welcomes contributions from computer science, from information security, business, and law.”
For more information about securing AI against failure modes, Anderson has co-authored a book with Microsoft researcher Ram Shankar Siva Kumar for a general audience entitled Not with a Bug, But with a Sticker: Attacks on Machine Learning Systems and What To Do About Them. All author proceeds will be donated to charities Black in AI and Bountiful Children’s Foundation.”
Leave a Reply