This article is based on a presentation at the Fall 2023 Accounting & Finance Forum on Data with Dignity by Ankit Virmani, Senior Engineering Manager of Data/ML Operations at CVS.
Even if you’re not leading data and machine learning operations for a global retailer like CVS, you’re likely aware of the risks inherent to the rapid adoption of artificial intelligence in decision-making—-bias and discrimination among the top of those concerns.
And for good reason. A lack of appropriate oversight and AI governance can have catastrophic consequences in the real world.
In 2019, it came to light that Dutch tax authorities employed a self-learning algorithm to identify potential child care benefits fraud. But because there was demographic bias in the data that was training the model, families were wrongly subjected to penalties based on a mere suspicion of fraud using the system’s risk indicators.
Tens of thousands of families, largely from minority or lower income backgrounds, were made to pay substantial debts to the tax agency—some totaling €100,000 or more. Many of those impacted were driven into severe financial hardship and fell below the poverty line. Couples divorced. Some took their own lives. And more than a thousand children were placed in foster care. All of this was a result of bias in a machine learning algorithm.
That’s why Ankit Virmani’s work at CVS (and Deloitte, Amazon, and Google before that) has been increasingly focused on responsible AI, a growing field that aims to operationalize techniques for measuring and mitigating bias across the lifecycle of data engineering and machine learning operations.
“When we are evaluating the efficacy of our engineering teams or our models, the first thing we ask is about AI governance,” says Virmani. “It’s critically important everywhere I’ve worked. Responsible AI will take 5 to 10 years to reach mainstream adoption but will ultimately have a transformational impact on business.”
Gartner’s research around responsible AI echoes Virmani’s emphasis on AI governance and bias mitigation.
“Increased trust, transparency, fairness and auditability of AI technologies continues to be of growing importance to a wide range of stakeholders,” says Svetlana Sicular, research vice president at Gartner. “Responsible AI helps achieve fairness, even though biases are baked into the data, gain trust, although transparency and explainability methods are evolving, and ensure regulatory compliance, while grappling with AI’s probabilistic nature.”
Identifying and Mitigating Bias in AI
The phenomenon of AI bias happens when an AI algorithm produces an inclination for or against certain data in a machine learning model, and the algorithm intentionally or unintentionally becomes prejudiced for or against certain outcomes.
“Bias manifests itself throughout the machine learning operations cycle all the way from data sourcing to model creation,” says Virmani. “It is not always easy to identify that bias has occurred unless there are significant business and technical impacts of the model running in production, but some mitigation strategies can potentially help reduce the impact.”
Breaking Down the Machine Learning Process
During data engineering, which includes data ingestion and data processing, there could be inherent bias in the source data including data that only represents a certain population group, missing or unexpected data or feature values, or a non-randomized split of data between testing and training of the model.
At this stage, Virmani recommends having a very thorough understanding of the data set, bringing in a data subject matter expert to work alongside the machine learning team, doing deep inspections and timeframe monitoring, identifying suitable imputation techniques, and evaluating outlier groups before removing them.
When a project reaches the stages of feature engineering, which is when the features for a given machine learning model are identified based on business requirements and data analysis, the root cause of bias often comes from misalignment between the business context and the type of model used or features selected.
During model training, validation, and evaluation, bias often comes from feedback and training of the model, either from biased outcomes that the model produces or inadvertently biased human annotations or evaluation techniques on model performance.
At each point between feature engineering and model evaluation, Virmani recommends mitigation strategies for bias that include simple random sampling, stratified random sampling, appropriate model selection with cross validation, sensitive feature removal (gender, ethnicity, etc.), establishing performance indicators before model creation, and cross validating along the way.
Ultimately, AI governance should focus on a model’s purpose, its influence on human lives, and its business impact, rather than the model design or complexity, says Virmani.
And most importantly, any final decisions about which model should be used, how it is reviewed, and when it should be discontinued should be made by a human being.
“This ensures that the responsibility resides with a human decision maker and can be an important control for drift in self-learning models,” says Virmani. “Explainability, interpretability, and transparency of models, data, and decision making will be key to even enable an appropriate possibility to manage remaining model risks.”
Leave a Reply