AI and the Limits of Human Judgement

Skeptics Café will be held in the Function Room at The Stolberg Hotel, 197 Plenty Road Preston. The 86 tram route is close, and Bell train station – situated on the Epping line – is a short walk away. It will also be a virtual event accessible via Zoom, as per the details here.
Start Time: 6:00pm for dinner and a chat. Talk and Zoom session starts at 7.30pm.
We aim to wrap up after 9:00pm.

More Moral Than Us?
Why the question matters for alignment, moral progress, and long-term flourishing

TL;DR: On some measurable moral tasks, today's AI already looks competitive with humans, or better, and the alignment community is oddly quiet about it. But moral capability is not the same as moral motivation: a system can compute the right answer without being moved by it. Pulling apart knowledge, reasoning, judgement and motivation is what makes the question tractable, and what makes the honest answer uncomfortable. Caveats attached, in quantity.

Almost nobody working on AI alignment wants to say this out loud, at least not about current systems. So let's say it: AI might already be more moral than us, in some measurable respects.
Two questions fall straight out of that:

Why does it feel like a dangerous thing to claim?
And is human moral reasoning really a standard worth bragging about as an alignment target?

The provocation isn't idle. In a modified Moral Turing Test, people rated a large language model's moral reasoning as better than other humans' on almost all measured dimensions - see paper Attributions toward artificial agents in a modified Moral Turing Test + associated interview and talk with lead author Eyal Aharoni.[1] Findings like that are easy to over-read and easy to wave away, which is most of the problem.

Provocative questions? Yes, and they are backed by empirical research. In a modified Moral Turing Test, people rated GPT-4's moral evaluations as better than other humans' on almost every dimension, though they could still tell which was the machine (A