This article is based on a presentation at the Fall 2023 Institute of Business Analytics Conference on Generative AI and Cybersecurity: Navigating the New Era of Threats and Safeguards. by Tori Westerhoff, senior manager of strategy and business planning for mixed reality on the Microsoft AI Red Team
Amid the bustle of innovation in artificial intelligence, thought leaders in both the private and public sectors are more clearly seeing the importance of security measures to make sure AI technology is as powerful as possible without doing harm.
To that end, in late 2023, the Biden Administration issued a landmark Executive Order that established unprecedented standards for AI safety and security. The Executive Order was designed to more carefully regulate the development of trustworthy technology and to protect privacy while promoting innovation and competition in the AI landscape.
Alongside these efforts, companies like Microsoft are finding new ways to stay ahead of the prolific acceleration of the technology with security processes like red teaming.
Tori Westerhoff, senior manager of strategy and business planning for mixed reality on the Microsoft AI Red Team, has found that the awareness of the entire risk landscape for AI is generally low.
“Researchers are running at breakneck speed, but our understanding of how to secure these AI models is actually quite new,” says Westerhoff. “The majority of partners and companies we talk to don’t know how to secure their systems. As a business at the forefront of this technology wave, we’re asking how we can fill gaps in security.”
Microsoft is taking a wide, diverse approach to security, says Westerhoff. The company has developed a six-point plan that emphasizes the ethical standards that businesses in technology are going to be held accountable for—which is what new legislation focuses on and what conversations center around on the international and national stage.
The Importance of Red Teaming in AI
At Microsoft, red teaming is part of security and maintaining a responsible AI product. Red teaming has been a function of national security for quite some time, originating in the national security of physical spaces and when attackers tried to penetrate security. But the term has evolved to more broadly refer to any adversarial attacking mechanism, typically double-blind attacks in which the team that is doing the adversarial attacking is not the same team that owns the product being attacked.
“Responsible AI has informed the evolution of the term red teaming,” says Westerhoff. “When we talk about red teaming with AI, we have a completely different attack surface. Now, it doesn’t just include security and trying to break into the system—it also includes testing the system for harm to users.”
The Microsoft AI Red Team is an independent red team, which means they’re not embedded within one of Microsoft’s individual product departments. Products are brought to the AI Red Team before they launch, and the team’s job is to identify potential harms or vulnerabilities on the responsible or secure side of that AI product.
“Our aim is always to mitigate prior to launch so that users can have the safest experience,” says Westerhoff. “We work in tandem with the product teams of the products we end up testing, but they don’t know how we’re adversarially attacking until after that work is done. We also do adversarial and benign attack methods because benign questions can also create harmful results.”
It’s important that the Microsoft AI Red Team is independent, because as the architecture is evolving so rapidly, there’s a lot of harm that can be done. In the regulatory environment, going in and testing—pretending to be adversarial—is key to finding vulnerabilities. The Microsoft AI Red Team focuses on two sides of red teaming in their testing: responsible AI and security.
Responsible AI: The undesirable responses from an AI system, like bias, are what you see covered in the news, says Westerhoff. When her team performs adversarial testing on these products, they determine if the response will actually happen in the products that Microsoft is putting forth or the models that it is hosting.
“Our goal is to reduce these instances so the shining benefits of the technology are the headline instead,” Westerhoff says.
Security: On the security side, there can be significant harms and vulnerabilities in AI systems. As AI is further integrated into technology services, the types of data that can be accessed increase, including passwords and private information. If you can use similar prompt injections or attacks to get at core intellectual property or data, a big security problem comes up.
The Red Teaming Process
When the Microsoft AI Red Team receives a product before launch, they perform red teaming in three ways.
- Full stack. The team goes through the full architecture of the product with a focus on security. They take an adversarial approach to look at vulnerabilities and how the information goes through the entire tech stack.
- Adversarial ML. They looks at the user interface, identifying vulnerabilities in the technology that can access vulnerabilities.
- Creative prompt. At the user level, they focus on the product’s AI harm. The team uses diverse methods to test the edge cases of how a product absorbs stimuli and returns an output response.
“Red teaming is meant to inform and mitigate vulnerabilities before a product ever launches,” says Westerhoff. “As products evolve, think about how quickly features and functions shift. We consistently red team throughout all of those iterations because, as you add more technology and features into one product stack, you end up exposing particular vulnerabilities that you wouldn’t have had before.”
Microsoft’s Priorities for AI Security
At Microsoft, red teaming is deployed continuously across the product lifecycle as technology changes, even once products are in the market, says Westerhoff. The methods are informed both by the community of people who build the products and the people who use red teaming to mitigate and measure.
“It’s truly an iterative process that leads to a stronger, better understanding on both sides—for the red team, understanding of the types of mitigations and measurements that work, and on the product side, of the types of vulnerabilities they need to harden their products against in the future,” says Westerhoff. “We’re trying to build a really strong community of knowledge sharing every time we test a product.”
In addition to developing more trustworthy products, a broader business priority for the team is building awareness about AI harm and the things that need to be mitigated and integrated into AI strategy for a product.
The team is also committed to diversity, which inherently creates sharper red teaming, Westerhoff says. They try to mimic and harness the diversity of intellect across all of their user communities to make it safer, and work cross-functionally by engaging with their policy team, thought leaders, and in public and private collaboration.
“We also push ourselves to innovate at the same pace as the tech evolution—or faster,” Westerhoff says. “There’s an immense amount of pressure to be a few steps ahead. If we’re mimicking adversaries, we want to be at the forefront.”
Leave a Reply