I always find these "anti-AI" AI believer takes fascinating. If true AGI (which you are describing) comes to pass, there will certainly be massive societal consequences, and I'm not saying there won't be any dangers. But the economics in the resulting post-scarcity regime will be so far removed from our current world that I doubt any of this economic analysis will be even close to the mark.
I think the disconnect is that you are imagining a world where somehow LLMs are able to one-shot web businesses, but robotics and real-world tech is left untouched. Once LLMs can publish in top math/physics journals with little human assistance, it's a small step to dominating NeurIPS and getting us out of our mini-winter in robotics/RL. We're going to have Skynet or Star Trek, not the current weird situation where poor people can't afford healthy food, but can afford a smartphone.
Star Trek only got a good society after discovering FTL and existence of all manner of alien societies. And even after that Star Treks story motivations on why we turned good sound quite implausible given what we know about human nature and history. No effing way it will ever happen even if we discover aliens. Its just a wishful fever dream.
Well they didn't necessarily stop waging war in Star Trek either.. They also spent most of their time trying to not get defeated by parasitic artificial intelligence.
It isn't even just the aliens (although my headcanon is that the human belief that they "evolved beyond their base instincts" is part a trauma response to first contact and World War 3, and part Vulcan propaganda/psyop.) Star Trek's post scarcity society depends on replicators and transporters and free energy all of which defy the laws of physics in our universe (on top of FTL.)
We'll never have Star Trek. We'll also never have SkyNet, because SkyNet was too rational. It seems obvious that any AGI that emerges from LLMs - assuming that's possible - will not behave according to the old "cold and logical machine" template of AI common in sci-fi media. Whatever the future holds will be more stupid and ridiculous than we can imagine, because the present already is.
except it's sort of true and a reasonable assumption to make? Just as when a master painter makes something that looks "sloppy" to the layman, one immediately assumes there is some deep artistry behind it as opposed to poor technique, whereas when a child does it, one does not extend the same charitable attitude.
Sure I think there's some truth the that. You've gotta learn the rules first to know when it's ok break the rules. Somebody with a lot of experience should be able to judge how their message will be received, and what amount of effort is "good enough". Whereas someone with less workspace experience may lack such judgement, and is probably better off erring on the side of "too good" rather than "not good enough".
But it's definitely also very much tied to status, power, and privilege. The same people who have no qualms about firing off a sloppy email to their subordinates often spend a lot more effort on emails to their bosses. But even this discrepancy is justified, I think, given that a manager represents their subordinates to the higher ups. And the potential consequences of a bad impression or misunderstanding are more severe when communicating up the chain of command.
I was surprised at your result for ChatGPT 5.2, so I ran it myself (through the chat interface). On extended thinking, it got it right. On standard thinking, it got it wrong.
I'm not sure what you mean by "high"- are you running it through cursor, codex or directly through API or something? Those are not ideal interfaces through which to ask a question like this.
But also why would you ask whether you should walk or drive if the car is at home? Either way the answer is obvious, and there is no way to interpret it except as a trick question. Of course, the parsimonious assumption is that the car is at home so assuming that the car is at the car wash is a questionable choice to say the least (otherwise there would be 2 cars in the situation, which the question doesn't mention).
But you're ascribing understanding to the LLM, which is not what it's doing. If the LLM understood you, it would realise it's a trick question and, assuming it was British, reply with "You'd drive it because how else would you get it to the car wash you absolute tit."
Even the higher level reasoning, while answering the question correctly, don't grasp the higher context that the question is obviously a trick question. They still answer earnestly. Granted, it is a tool that is doing what you want (answering a question) but let's not ascribe higher understanding than what is clearly observed - and also based on what we know about how LLMs work.
Gemini at least is putting some snark into its response:
“Unless you've mastered the art of carrying a 4,000-pound vehicle over your shoulder, you should definitely drive. While 150 feet is a very short walk, it's a bit difficult to wash a car that isn't actually at the car wash!”
I think a good rule of thumb is to default to assuming a question is asked in good faith (i.e. it's not a trick question). That goes for human beings and chat/AI models.
In fact, it's particularly true for AI models because the question could have been generated by some kind of automated process. e.g. I write my schedule out and then ask the model to plan my day. The "go 50 metres to car wash" bit might just be a step in my day.
> I think a good rule of thumb is to default to assuming a question is asked in good faith (i.e. it's not a trick question).
Sure, as a default this is fine. But when things don't make sense, the first thing you do is toss those default assumptions (and probably we have some internal ranking of which ones to toss first).
The normal human response to this question would not be to take it as a genuine question. For most of us, this quickly trips into "this is a trick question".
Rule of thumb for who, humans or chatbots? For a human, who has their own wants and values, I think it makes perfect sense to wonder what on earth made the interlocutor ask that.
Rule of thumb for everyone (i.e. both). If I ask you a question, start by assuming I want the answer to the question as stated unless there is a good reason for you to think it's not meant literally. If you have a lot more context (e.g. you know I frequently ask you trick or rhetorical questions or this is a chit-chat scenario) then maybe you can do something differently.
I think being curious about the motivations behind a question is fine but it only really matters if it's going to affect your answer.
Certainly when dealing with technical problem solving I often find myself asking extremely simple questions and it often wastes time when people don't answer directly, instead answering some completely different other question or demanding explanations why I'm asking for certain information when I'm just trying to help them.
That's never been how humans work. Going back to the specific example: the question is so nonsensical on its face that the only logical conclusion is that the asker is taking the piss out of you.
> Certainly when dealing with technical problem solving I often find myself asking extremely simple questions and it often wastes time when people don't answer directly
Context and the nature of the questions matters.
> demanding explanations why I'm asking for certain information when I'm just trying to help them.
Interestingly, they're giving you information with this. The person you're asking doesn't understand the link between your question and the help you're trying to offer. This is manifesting as a belief that you're wasting their time and they're reacting as such. Serious point: invest in communication skills to help draw the line between their needs and how your questions will help you meet them.
>Going back to the specific example: the question is so nonsensical on its face that the only logical conclusion is that the asker is taking the piss out of you.
I would dispute that that matters in 99.9% of scenarios.
>The person you're asking doesn't understand the link between your question and the help you're trying to offer.
Sure I, get that and I do always explain why I need to know something but it does add delays to the process (either before or after I ask). When I'm on the receiving end of a support call I answer the questions I'm asked (and provide supplementary information if I think they might need it).
> That's never been how humans work. Going back to the specific example: the question is so nonsensical on its face that the only logical conclusion is that the asker is taking the piss out of you.
Or a typo, or changing one's mind part way through.
If someone asked me, I may well not be paying enough attention and say "walk"; but I may also say "Wa… hang on, did you say walk or drive your car to a car wash?"
Sure, in a context in which you're solving a technical problem for me, it's fair that I shouldn't worry too much about why you're asking - unless, for instance, I'm trying to learn to solve the question myself next time.
Which sounds like a very common, very understandable reason to think about motivations.
So even in that situation, it isn't simple.
This probably sucks for people who aren't good at theory of mind reasoning. But surprisingly maybe, that isn't the case for chatbots. They can be creepily good at it, provided they have the context - they just aren't instruction tuned to ask short clarifying questions in response to a question, which humans do, and which would solve most of these gotchas.
I don't mind people asking why I asked something, I'd just prefer they answer the question as well. In the original scenario, the chatbot could answer the question as written AND enquire if that's what they really meant. It's the StackOverflow syndrome where people answer a different question to the one posed. If someone asks "How can I do this on Windows?" - telling me that Windows sucks and here's how to do it on Linux is only slightly useful. Answer the question and feel free to mention how much easier it is in Linux by all means.
I personally love explaining to people who might want to solve the issue next time so I'm happy to bore them to tears if they want. But don't let us delay solving the problem this time.
Yes but unironically. It may seem obvious now that the LLM is just a word salad generator with no sentience, but look at the astounding evolution of ChatGPT 2 to ChatGPT 5 in a mere 3 years. I don't think it's at all improbable that ChatGPT 8 could be prompted to blend seamlessly in almost any online forum and be essentially undetectable. Is the argument essentially that life must be carbon based? Anything produced from neural network weights inside silicon simply cannot achieve sentience? If that's true, why?
> I think it’s important to highlight at this stage that I am not, in fact, “anti-LLM”. I’m anti-the branding of it as “artificial intelligence”, because it’s not intelligent. It’s a form of machine learning.
It's a bit weird to be against the use of the phrase "artificial intelligence" and not "machine learning". Is it possible to learn without intelligence? Methinks the author is a bit triggered by the term "intelligence" at a base primal level ("machines can't think!").
> “Generative AI” is just a very good Markov chain that people expect far too much from.
The author of this post doesn't know the basics of how LLMs work. The whole reason LLMs work so well is that they are extremely stateful and not memoryless, the key property of Markov processes.
I think the disconnect is that you are imagining a world where somehow LLMs are able to one-shot web businesses, but robotics and real-world tech is left untouched. Once LLMs can publish in top math/physics journals with little human assistance, it's a small step to dominating NeurIPS and getting us out of our mini-winter in robotics/RL. We're going to have Skynet or Star Trek, not the current weird situation where poor people can't afford healthy food, but can afford a smartphone.
reply