Trust but verify: A sneak peek inside the ‘black box’ of machine learning

AI can be a powerful tool for analyzing vast amounts of data, finding connections and correlations that humans cannot. However, unlike a person who solves an arithmetic problem, many AI models cannot easily explain the steps they took to arrive at their final answers. They are what in computer science are known as black boxes: you can see what goes in and what goes out; What happens between them is a mystery.

Laura Plattner, associate professor of finance at Stanford GSB, explains the black box problem in many machine learning models. “The strength of technology lies in its ability to reflect the complexity of the world,” she says. But the inability to fully understand the complexity generated by the model raises practical, legal, and ethical questions. “If these black boxes are used to make a very risky decision in lending, insurance, health care, or the judicial system, we have to decide whether we feel comfortable not knowing exactly why the decision was made.”

Lending is one of the many areas where black box machine learning models may find new insights into complex data. Currently, credit scores and loan decisions are often based on a few dozen variables. An AI-driven model that looks at more than 600 risk variables may weigh more accurately, benefiting prudent lenders and borrowers who might otherwise be rejected. In theory, AI could make consumer lending not only more accurate, but fairer.

quotes

If these black boxes are used to make a very risky decision in lending, insurance, health care, or the judicial system, we have to decide whether we feel comfortable not knowing exactly why the decision was made.

attribution

Laura Plattner

But US lenders are not in a hurry to adopt these tools: if they can’t explain how they rate loan applicants, they may run counter to fair lending rules. “They wouldn’t be willing to take the risk, especially in a more sensitive lending area like mortgages,” says Plattner. And if federal regulators conclude that these new tools are not trustworthy, “I think that points to a very different future for using AI in consumer lending.”

However, there are ways to take the output of the black box and work backwards to figure out how to create it. Blattner and Associate Professor of Operations, Information and Technology Jann Spiess recently collaborated with him FinRegLaba nonprofit think tank, has been evaluating several tools that attempt to explain credit underwriting models’ predictions about individual applicants as well as minorities and other demographic groups.

“We came out cautiously optimistic,” she says. “If you pick the right tool, it’s good at handling complexity.” They also found that increased transparency can be balanced with performance. “The more complex black box models were more accurate in predicting default, but they were also more equal across demographic groups, which was surprising,” she says.

Even if regulators approve of this type of AI and lenders adopt it, Platner says people need to be part of the equation. “You can’t blindly pick a software tool off the shelf and hope it works,” she says. Users should constantly test and evaluate their models to ensure that they function properly.

And while black boxes can make superhuman calculations, we may still want a loan officer, doctor, or judge to have the last word.

“In all of these cases, the human wants to know why the AI ​​is making a particular recommendation so that they can allow that to influence decision-making one way or another,” Blattner says.

Leave a Comment