I don't think that sama was arguing that ChatGPT actually passed a PhD thesis defense. But arguably, it could make for an interesting benchmark.

Please do not get swayed by nor defend the words vomited by a snake oil salesman.

Also what benchmark? How will you you design it?

exactly. this is what whole RL thing is optimizing for, even if that is not the intent.