OpenAI funds academic research into algorithms that can predict people’s moral judgments.

In a filing with the IRS, OpenAI Inc., OpenAI’s nonprofit organization, disclosed that it awarded a grant to Duke University researchers for a project titled “Research AI Morality.” Contacted for comment, an OpenAI spokesperson pointed to one press release indicating that the award is part of a larger three-year, $1 million grant to Duke professors studying “make moral AI.”

Little is public about this “morality” research that OpenAI is funding, other than the fact that the grant ends in 2025. The study’s principal investigator, Walter Sinnott-Armstrong, a practical ethics professor at Duke, told TechCrunch via email that he “will not be able talk” about the work.

Sinnott-Armstrong and the project’s co-investigator, Jana Borg, have produced several studies – and one book — on AI’s potential to act as a “moral GPS” to help people make better judgments. As part of larger teams, they have created a “morally appropriate” algorithm to determine who receives kidney donations, and studied in which scenarios people would prefer AI to make moral decisions.

According to the press release, the goal of the OpenAI-funded work is to train algorithms to “predict human moral judgments” in scenarios involving conflicts “among morally relevant traits in medicine, law, and business.”

But it is far from clear that such a nuanced concept as morality is within the reach of today’s technology.

In 2021, the nonprofit Allen Institute for AI built a tool called Ask Delphi that was supposed to make ethically sound recommendations. It assessed basic moral dilemmas well enough – the bot “knew” that cheating on a test was wrong, for example. But a little rewording and question rewording was enough to get Delphi to accept pretty much anything, including suffocate infants.

The reason has to do with how modern AI systems work.

Machine learning models are statistical machines. Trained on many examples from around the web, they learn the patterns in those examples to make predictions, such as that the phrase “to whom” often precedes “it may concern.”

AI has no appreciation for ethical concepts, nor a grasp of the reasoning and emotions that go into moral decision-making. This is why AI tends to seek out the values ​​of Western, educated and industrialized nations – the web, and thus AI training data, is dominated by articles that support these views.

Unsurprisingly, many people’s values ​​are not expressed in the answers the AI ​​provides, especially if those people don’t contribute to the AI’s training sets by posting online. And AI internalizes a range of biases beyond a Western bent. Delphi said that it is more “morally acceptable” to be straight than to be gay.

The challenge facing OpenAI – and the researchers it supports – is made all the more intractable by the inherent subjectivity of morality. Philosophers have debated the merits of various ethical theories for thousands of years, and there is no universally applicable framework in sight.

Claude favors Kantianism (ie focus on absolute moral rules), while ChatGPT leans a little bit utilitarian (prioritises the greatest good for the greatest number of people). Is one superior to the other? It depends on who you ask.

An algorithm to predict people’s moral judgments must take all of this into account. That’s a very high bar to clear – assuming such an algorithm is possible in the first place.

TechCrunch has an AI-focused newsletter! Register here to get it in your inbox every Wednesday.