Reminds me of how an AI agent that tries to minimize the worst case scenario is almost always better than one that tries to maximize the best case scenario.
(admittedly that's a bit anecdotal, maybe someone with more knowledge can give better details on that statement)
Good article from Nassim Taleb [0] relating to rationality/irrationality and survival...
"survival comes first, truth, understanding and science later"
which would seem to relate to a the simpler model for AI centered around preventing disaster being more robust than trying to solve some form of maximization while risking ruin.
In some ways it is arguably better to be paranoid/"irrational" about risk than try to be perfectly sufficiently rational
"I have shown that, unless one has an overblown and a very unrealistic representation of some tail risks, one cannot survive –all it takes is a single event for the irreversible exit from among us. Is selective paranoia “irrational” if those individuals and populations who don’t have it end up dying or extinct, respectively?"
That just depends on how you what your AI to optimize and what results you value. After 3 attempts, would you rather have the best-case twice and the worst-case once or a medium-case three times?
(admittedly that's a bit anecdotal, maybe someone with more knowledge can give better details on that statement)