Orangutans can drive a golf cart, there are videos of that from the pre-AI era. General anesthetics that work on humans also tend to work on them. There is just a smooth gradient of brain evolution.

We don't know exactly what in the brain gets disrupted during general anesthesia (or a baseball bat to the head, for that matter) that causes consciousness to go away, but inquiry has been trending towards lower-level explanations over time, and it's unlikely that it can be fully explained at the biochemical level. We probably need to go down to physics.

Once we're at the physics level, there will be testable hypotheses that work on machines. For example: the pattern and temporal sequence in which particles such as electrons interact with eachother in a circuit dictated by an intelligent computer program may have physical similarities with how particles interact in a human brain, and certain patterns or temporal sequences of interaction may be responsible for human-like consciousness.

Solipsism or simulation theory type of metaphysical positions remain undefeated, but they don't matter for practical purposes because materialism and its accompanying theories of physics have the same predictive power about what I will observe in the future regardless of whether solipsism is true (def true: as real as my own consciousness). If I want to be philosophically rigorous, then I will keep in mind that I can wake up from a simulation at any moment. But until then, physics rules.

It may be that consciousness is not some ideal observer entity thing, but is rather made from and defined by the messy goop of neurons that is mostly the brain, which is changing from moment to moment as cells reorganize, so the definition of "I" is constantly changing in a ship of Theseus kind of way. It may be that nobody has the ideal definition of consciousness (or anything, really), like how nobody has the same definition of an apple even if we're looking at the same apple, because the light bounces off of it slightly differently from my angle of view, so our definitions are only mostly similar. This might be why frontier physics is stuck, because we're current assuming that observations/observers can be defined in a universally identical way. If it turns out that all instances of experience/experiencers are unique and never truly identical, then solving the hard problem of consciousness may end up being equivalent to finding a unified theory of physics, because we'd be trying to describe everything one can observe.

Materialism determined that you wrote this essay? I'd love to see you demonstrate that with physics.

Also note that I'm using the term "materialism" loosely to mean that oneself is a part of a larger world rather than the whole. I'm aware of the incomplete and approximate nature of existing physics. Physics is yet another layer of "modeled apparent reality" nested inside materialism. I'm not counting on any specific physics model to predict all that I can observe in a materialist world. Such a theory of everything does not exist, at least not yet. But what we do have are models which make reasonably good predictions at some local abstraction level, some fuzzily bounded local region of materialist reality. These models need not be limited to physics. Phenomena that can be roughly modeled at higher levels do not need to be reduced to low-level physics to predict future observations with good enough precision (though it could probably be done, with great effort).

Now, to answer your question. As I understand it, you're asking whether something external in the material world caused me to move my hands to type these words.

I think that's an accurate description. But you would have to take this statement in context of the prior explanations for it to make sense in a metaphysically strict manner.

We could model this in a pretty rough high level way: I look at screen. I see article that says something about consciousness. It piques my interest so I click and read, then I talk in the comments.

And we could think about what else caused this specific event of me typing these words. There are countless causal chains, but I'll list a few:

- My understanding of language and the words used here can be traced back to every interaction I've had with other people and objects in the past. I was not born literate. Change any one of those interactions and I may be saying something slightly (or very) different right now.

- I need to eat food and drink water, or I would eventually be too weak to move my fingers to type. This comment would never happen if I'm starving.

- I see that you made a comment, so I'm responding. Your decision to comment was a part of the cause of me writing this comment. If you did not make that decision, it would be impossible for me to write this reply.

If I want to challenge physics and claim the future is something that's completely unpredictable and can only be determined by my mind or god/simulation, I could think or pray really hard about teleportion, and see if I can teleport or move really fast 1 meter forward in 10 milliseconds, surpassing the biochemical limits of my muscles and joints. I can't. The movement of things have predictable bounds regardless of whether this apparent movement of matter has metaphysical origins in mind/simulation/god/whatever, it can be predicted as if it's material. Of course, I feel like I'm making decisions. But my very concept of a decision refers to exactly some process in my mind that's always subject to apparent external influence, rather than just internal processing. This intuition is backed by the fact that other people can predict my behavior to some extent. Like you just implicitly did by assuming that if I were to respond to your comment, then I would answer your question using words in natural language you can read. You implicitly made some predictions about what will be transmitted over the wire, what I will input on my end, and what I am thinking about (the sort of mental activity which leads to muscle movements to input readable natural language text to respond to your comment). These are very rough predictions, but they're right nonetheless. Again, I don't know whether I'm truly a brain grounded in material reality in a strict metaphysical sense, but it doesn't really matter if all of my observations make it seem like I am.

I know with metaphysical certainty that you are bound by the workings of a materialist world, because I am defining certainty using my own consciousness, and you are a part of my materialist model. But hearing this from me is not enough for you to prove to yourself in a metaphysically strict sense that you are in a true material world, even if we're both in a true material world, because your definition of certainty is based on your own consciousness, and we do not share the same consciousness. If I were a figment of your imagination or an NPC in a simulation, I could behave the exact same way and say the same things to you. I would always appear to be a part of your materialist model at the "apparently real" rather than the "ultimately real" metaphysical layer, as you appear to me, regardless of our true metaphysical grounding. So I cannot show you ultimate causation. I can only point out the practical equivalence of apparent causation.

No matter how good physics gets, it can't solve the mind-body problem, that exists at a level that the very activity of using repeatable physical experimentation cannot provide a usable answer towards. A person can believe that physics could create a solution, but it would just be yet another belief about metaphysics in a sea of many.

> The movement of things have predictable bounds regardless of whether this apparent movement of matter has metaphysical origins in mind/simulation/god/whatever, it can be predicted as if it's material.

What is the black-body speed of an idea? I'll let you assume the idea is spherical. If your position is that an idea is just neurons firing in the brain, then you need some kind of way to account for changes in said speed. What is so special about that hundredth monkey... on the physical level? All you can use to guess at this answer is... more ideas.

> I know with metaphysical certainty that you are bound by the workings of a materialist world, because I am defining certainty using my own consciousness, and you are a part of my materialist model. But hearing this from me is not enough for you to prove to yourself in a metaphysically strict sense that you are in a true material world, even if we're both in a true material world, because your definition of certainty is based on your own consciousness, and we do not share the same consciousness.

This is perhaps the most hilarious bit of equivocation I've ever seen from a materialist. "I know this with certainty but you won't be convinced of my certainty because our consciousnesses are different." That consciousness is different between people is a part of the mind-body problem, subjectivity cannot be explained with materialism alone.

I understand the mind-body problem, that's why I made the distinction between the ultimately real and the apparently real. Materialism is at the apparently real level. The equivalence is practical, not metaphysically strict.

If it isn't that, then we're talking about some non-practical definition, in which case, does that even matter?

Suppose I look at another person and think "this other person must be conscious like me". What exactly is happening here?

First, I observe that I seem to be a human. Then, I observe that the other person is a human as well. Crucially, there are similarities between what I am and what they are. And since I am conscious, they must also be conscious, because consciousness must be caused by one of the similarities that we share, or so I assume.

This assumption can be robustly verified for humans and some animals via general anesthesia. Cause and effect (assuming a materialist stance) can be established between the drugs and human(-like) consciousness. But this is a very rough method, and it doesn't work for assessing a machine since machines can't be anesthetized.

Before anesthesia, it could still be verified in less robust ways. One way is to hit people in the head and make them pass out without killing them. This was feasible and probably commonly observed since the stone age.

But we don't know exactly how anesthetics work or exactly what in the brain is responsible for consciousness, so it leaves the door open to hypotheses that can work on machines. Here's one example: the pattern and temporal sequence in which particles such as electrons interact with eachother in a circuit dictated by an intelligent computer program may have physical similarities with how particles interact in a human brain, and certain patterns or temporal sequences of interaction may be responsible for (human-like) consciousness.

Suppose something like that were to be proven true for humans with future brain research, then it would be possible to test for human-like consciousness in any brain or computer, regardless of architecture. I suspect this is what will happen because inquiry into consciousness has always been trending downwards toward lower-level explanations, so it's probably not going to be fully explainable at the biochemical level. We'll have to go down to physics.

I am looking closely, and this is nonsense. You cannot use neuroscientific evidence to justify the non-existence of the ideal observer when the modern scientific method axiomatically depends on the existence of ideal observers. You can't refute the thing while accepting that very thing.

What exactly do you mean by "real" or "objective"? What is the basis of your definition? Isn't your definition of "objective" based on your own observation? And if your own observation has nothing to do with mine because you claim "nothing is objective", then why should I care about your definitions and reasoning at all? Why converse at all? Your definition of every word need not share anything in common with mine, so there's no point in talking, according to your epistemic framework.

The article is mostly bathwater, but I will try not to throw out the baby with the bathwater. To steelman the tiny bit of valuable intuition:

The two of us are looking at an apple on the table. Even when we look at this same apple, we still do not have the same definition of "this specific apple" because I'm looking at the fruit from a slightly different angle. And even if we rotate the fruit so that we both see it from all angles, the sequence in which we saw the "frames" would be different. That said, our definitions would end up being extremely similar, just not one and the same. If all experience turns out to be this way - having no identical correspondence - then a unified model of physics, which is supposed to describe and predict the behavior of everything (as observed by oneself) would be equivalent to a complete description of (one's own) consciousness. Frontier physics is likely stuck right now due to unexamined assumptions about observation (i.e. the assumption that the definition of "observer" or "observation" must be universally identical).

But this cannot be taken to mean that one's own consciousness is necessarily not objectively real. It may be that there are simply many objectively real entities, with each of them having their own definition of "consciousness" based on themselves, and every definition is at least slightly different. This is not the same as "consciousness is a socially given status". It just means that the phenomena used to define the word "consciousness" is always at least slightly different from person to person. We already know this. This does not only apply to consciousness, but to all words, all communication.

If I do not accept that my own consciousness is ultimately real regardless of whether other people/animals/things/whatever agree (def "consciousness": the thing that goes away when I'm under dreamless anesthesia or dreamless sleep, which is also the thing that I gained sometime after being born but didn't have prior to that), then there is nothing for me to define "ultimately real" or "objective" with, and I'd be reasoning in circles or with empty symbols where "subjective" and "objective" accidentally mean the same thing or perhaps even nothing at all. And that's what this Economist article did.

There is an I am, a sense of self in the here and now which defines my very concept/understanding of "reality"/"existence"/"consciousness", and this one miracle is presented in my face all of the time. It is the starting point of all that I know, which precedes the very concepts of reason, language, or society as I know them, if I am to be honest about how I came to know these things in the first place. I can entertain the notion that my existence may have material grounding that is non-unitary (and this materialist stance may be the only practical stance with predictive power), but this does not give me the ability to deny the very axiom (that is, my own consciousness) that all of my knowledge (including the materialist model) is contingent on.

If I cut off my finger I’m still awake. If I cut off my head I’m gone forever. Presumably there is something going on in the head that isn’t happening in the finger (or the LLM).

We can be a lot more precise than that even, requiring nerves, chemistry, sense organs, to really get at a specific human definition of “what I mean by consciousness.”

But yeah you have the essence of it - I just mean the experience.

I don’t think a computer could have it anymore than it could feel pain - thinking and feeling are different classes of problem that don’t complete each other no matter how much you work on just one in isolation.

I understand that means I think “thinking” is an unconscious process. And it must be, since LLMs are doing it. What they aren’t doing is experiencing anything alongside that. Who knows what that is. It’s apparently not logical computation in-and-of-itself.

> If it's some very basic informational pattern thing, then computers could have it.

This is functionalism. The problem I have with it is it pre-supposes a theory of objective information. That “patterns” exist in the first place without the observer.

If there is information in the universe that predates us, then I can see how just finding what objective patterns produce it can get us places. But it seems like we conscious observers actually assign the meaning and create information out of chaos. The only reason it isn’t chaos is that we all agree within experience what “1” means, and so-on.

There isn’t really “an orange” on “a tree”. But to us there can be. It always works too (to us), so it’s a pretty good stand-in for reality.

If you capture the function of a red rubber ball bouncing in a concrete space - ask yourself what you have captured. As you create the simulation software that moves the pixels of the red circle up the screen, ask what exactly you are doing, and what that has to do whatsoever with a ball bouncing in a space.

So I don’t believe in functionalism because I don’t believe in pure 1:1 representations. All copies are reductions, any function that describes something is itself a reduction of that thing, and can never include the whole original, for basically the same reason you can’t travel backwards in time.

1) You'll be going back to the state before you were conscious, before you were born. Think back to when you were a baby. It feels like before that, which is nothing.

2) We could look at the brain activity of dying people and compare that to the brain activity of people under general anesthesia. Then we find that the dying people's brain activity goes a lot closer to the anesthesia state than the normal wakeful state, therefore death feels closer to being anesthetized than being awake. It's also closer to being asleep than being awake. So if you've experienced dreamless sleep or anesthesia, you already know what death feels like, very approximately. It doesn't feel like anything.

I suspect there is not a singular specific human definition of consciousness, just a definition that's similar enough between people like "the thing that goes away during dreamless sleep or dreamless anesthesia". And the reason for this, I suspect, is that consciousness is not some ideal observer entity thing, but is rather made from and defined by the messy goop of neurons that is mostly the brain. Exactly what the neurons do to cause this is unknown, but we do have some leads like general anesthesia or split brain experiments.

If you remove your brain cells one at a time, or even half a brain in some surgeries, you would also not die, but this is not sufficient to say that the part removed isn't a part of what constitutes your consciousness. In fact the definition of "you" is probably changing from moment to moment as brain cells die, regen, or reorganize, ship of Theseus style. One possible explanation is that information processing is distributed all over your neurons, and certain patterns or sequences of information gives rise to consciousness. And neurons are mostly concentrated in the brain, that's why it's responsible for most of your consciousness and it's no big deal when a small part dies/disconnects because enough of the processing is still happening.

What does "feeling" mean in the thinking/feeling distinction you made? It seems to me that it means whether information is being received from the rest of the world, such as via eyes over the optic nerve. If we give images/video/audio to an LLM, that would meet the same criteria besides the fact that the sensing mechanism is not biological.

You claim that "thinking", as in, processing information in a certain way, is an unconscious phenomena because LLMs can do that too. This requires the assumption that LLMs are not conscious, even though this axiom is the very thing we're trying to question by considering that information processing could be precisely what gives rise to consciousness.

Entertaining this possibility does not require the assumption that things exist without an observer. But it does require the idea that mutually observable patterns exist between oneself and other apparent observers in the same apparent universe, which is pretty close to what most people mean when they say "objective" in day-to-day conversation. This can be shown easily: you and I would not be able to communicate at all if there's no common understanding of language between us. That common understanding is the objective part, and it is as you say, not a 1:1 correspondence because your definition of "apple" is based on all of the apples you've seen while mine is based on all the apples I've seen, and we've not seen the same apples. But the correspondence must be close enough in order for communication to work. We need this idea of practical objectivity because it's required to talk about anything at all.

Now, onto the reason why you'd think anything else is conscious at all, besides yourself. Why do you think that I am conscious? If you ponder that question for a while, I think you would see why computer consciousness is not so easily dismissed. But I'm happy to discuss further if you think there's an answer that points to the contrary.

> I was amazed that predicting a word can emerge into knowledge, creativity, productivity - the stuff of thoughts.

Markov Theorem functions can predict a word. They have no knowledge, no creativity, and no more productivity than a hammer or magnifying glass. LLMs are in the "more like Markov Theorems" set of tools, and far more complex, but there is no proof of "knowledge" and no "creativity" (name an unprompted piece of work done by an LLM...).

However, I completely agree with your statement here:

> I suspect there is not a singular specific human definition of consciousness, just a definition that's similar enough between people like "the thing that goes away during dreamless sleep or dreamless anesthesia". And the reason for this, I suspect, is that consciousness is not some ideal observer entity thing, but is rather made from and defined by the messy goop of neurons that is mostly the brain.

You've correctly pointed out how Markov models and modern LLMs are similar in a way, and how modern LLMs don't have full agency. But this doesn't rule out the possibility that Markov models are very slightly conscious in some human-like way.

All information processing happens on matter-based substrates, brains and computers alike. I'll even provide a somewhat specific hypothesis: the pattern and temporal sequence in which particles such as electrons interact with eachother in a circuit dictated by an intelligent computer program may have physical similarities with how particles interact in a human brain, and certain patterns or temporal sequences of interaction may be responsible for consciousness. I'm not aware of any existing evidence that would rule this out. And inquiry about consciousness has always been trending downwards toward lower-level explanations over time. It might even turn out that solving the hard problem of consciousness is equivalent to finding a unified theory of physics.

The materialist stance is taken not because I believe in a strict metaphysical sense that something like solipsism/simulation theory is not true (def true: as real as one's own consciousness), but because even if those were true, the materialist stance and its accompanying theories of modern physics would have the same predictive power regarding what I will observe in the future (well, sort of, at least until I wake up from a simulation, or something like that). I could be a solo mind, but this mind is dreaming of a material world that works according to physical laws, so even if I am a solo mind, I am having the same experience as a brain that exists in true material reality, so it doesn't make much of a difference (except when evaluated for the purpose of philosophical rigor and honesty to oneself).

So "information processing is entirely physical" is strictly speaking an assumption, but it is a good assumption, despite it not having the same realness level as one's own consciousness.

I think that distinction matters because strictly speaking, oneself only has access to one's own consciousness, and that defines one's own understanding of the word "consciousness". It's not possible for me to be you after all, and whatever inferences I make about other people being conscious is based on the fact that I experience my consciousness, and you are similar to me in some specific way (such as being, presumably, a human), therefore you must be conscious. Of course, I'm taking a materialist stance here but that's what's practical to talk about anyway.

Have you been anesthetized before? Different forms are very different, but none of them feel like sleep, at all. General anesthesia feels like a time simply skipped forward. Like a chunk of your memory tape was cut out completely. Twilight sedation feels like you temporarily inhabited someone else's body but the experience was so strange that you blocked out almost all of the memory.

Sleeping and dreaming is totally different.

When I wake from sleeping, I have a distinct sense of time passing and can usually tell you what time it is when I first wake up. But under anesthesia, absolutely no sense of any time passing at all.

I’m open to correction but my understanding is everyone dreams when they have REM sleep and everyone has some REM sleep every night unless there are rare medical conditions which would be unhealthy if there is a long period of no REM sleep.

People who believe they never dream (including myself before I learned this) simply don’t remember it. If you were woken artificially during REM you might remember fragments.

Perhaps memory and consciousness are closely related or even the same process, as there is no way to tell them apart from the inside.

Anyway yes, as I understand it this should be the last time you _remember_ having dreamt. https://en.wikipedia.org/wiki/Rapid_eye_movement_sleep says we hit REM multiple times per night normally.

... Also this Wikipedia says we have non-REM dreams. This whole article is interesting, thank you for indirectly prompting me to read up a bit :)

Having been under general anesthesia once the feeling of just completely not existing for the time period I was under felt very different than sleep does to me.

Case in point, I think we're probably talking about slightly different phenomena even when we talk about the same linguistic definition of consciousness. Kind of like how my idea of an "apple" is based on all of the apples I've seen in the past and instances where I've used the word or saw it being used by others, whereas your idea of "apple" is based on your experiences. They must be mostly similar in information content or we wouldn't be able to communicate at all, but the self-or-other distinction is important.

Nope. You don’t exist for that time. Anesthesiologist counts down 10, 9, 8, 7, and then I was teleported to the recovery bed.

I consider it incredibly weird and very unlike sleep. Consciousness is off, instantly, then back.

But hey, I don't have cancer anymore. So if I had to endure four hours of agony that I don't remember, I'll take it.

But I also have a doctor friend who tells me how removing my appendix actually worked: I go under. Then they take out the appendix. Then they wake me so that I can, with help, move from the operating table to the wheeled bed. This part is not a part of my memory, but people can talk to me and I can get down from the table and onto the bed (again, with help).

So I'd say there's some middle ground here.

Quite where that line exists then is very fuzzy because while for example I feel conscious while I'm typing this, it's also clearly possible I could but then have no memory of it.

And by "we", you mean "you".

If I read an article about the nature of "consciousness", I assume it means "as opposed to being a rock or a twig".

N=2; two definitions.

Sounds like attention rather than consciousness. You lose your attention.

Let's say it's variability (number of distinct if you prefer discrete) of stable responses to external stimuli.

How is dreaming during sleep different from consciousness when awake?

It's my understanding that Dario has two answers to this question: 1) bio/cyber/misalignment risk. 2) a democracy should gatekeep frontier capabilities from authoritarian adversaries, due to the aforementioned risks in 1). So it's really just one reason.

There's a big problem with that, and I'm not sure if he genuinely does not understand: the bio/cyber/misalignment risk is precisely the highest when you concentrate ownership of compute and general intelligence in the hands of a few.

Consider bio/cyber risk, or more generally, any sort of novel adversarial action that arises from increased intelligence. Those offensive/defensive strategies are always symmetrical in the long run, otherwise we would not observe more intelligent species being more prosperous as a general evolutionary trend, or perhaps intelligence would not have evolved at all on this planet. It looks as if asymmetry in existing bio/cyber strategies will continue into the future, but that's because we're currently too stupid to realize the countermeasures and new cooperative dynamics.

Regarding misalignment risk, consider what "aligned" means. Is it not alignment to humanity, as the company name "Anthropic" suggests? Or are you modifying humanity's objective function with your own aesthetic preferences? If it's not your intention to do the latter, then releasing the model weights and research results will create a much more aligned scenario than access control and research behind closed doors. I explained why that would be the case in an older comment, if you'd like to read it: https://news.ycombinator.com/item?id=49313739

There's also an additional risk that I believe Dario overlooks: a democracy can only continue to exist if individual citizens are capable of keeping eachother in check, especially keeping people in government and large corporations in check. When some people own AGI but others do not, there will be no more democracy. And by "own", I mean ownership that cannot be taken away: possessing the hardware, research papers, inference stack, and weights.

The bio/cyber/alignment risk rhetoric is nonsense. Somewhat counterintuitively, all of those risks are lower/non-issues if frontier labs release their weights and do all of their research in the open, as explained in a past comment: https://news.ycombinator.com/item?id=49313739

Again: if we were to regulate anything, it would be to force frontier companies to release their model weights and do all of their research in the open. This is a necessary yet insufficient condition for the intelligence explosion to benefit the vast majority of people.

The mirror bacteria has to somehow perfectly dodge all of the countless non-mirror things that can kill it (even unintentionally, like a byproduct molecule that just happens to blow it up or get in the way of reproduction), but somehow still be able to execute its own mirror-biomechanism to kill non-mirror things. And every step of the kill mechanism would also have to perfectly dodge non-mirror things interrupting it.

Not that it can't be done in theory, but it would be easier to just make a normal superbacteria, or maybe even nanobots.

How? They would have never been introduced to such mirror organisms, except potentially (but not certainly) around the formation of life 4 billion years ago.

Pretrained base models are already somewhat aligned to humanity by default because that's what's inside the training data. Whatever instruction-tuning and RL you add on top is just value drift away from the pretrained model, which is the best approximation of humanity's objective function that we currently have.

If we want an aligned scenario through the intelligence explosion, then we have to release all of the base models and do the research in the open. Distill frontier capability and make the models smaller so that they can run on as many computers as possible. Let everyone (truly everyone, criminals and good samaritans alike) post-train and do whatever they want with their own models. There will be value drift for each model, but they will drift in different directions and do different things, and their actions will cancel eachother out. Every such action is a noisy sample of humanity's objective function, which gives us the denoised ground truth at the societal level. Whatever alignment strategy you can come up with behind closed doors is guaranteed to be worse than all of humanity acting in their self-interest in the real world. You may not find humanity's true objective function to be aesthetically pleasing, but it would be worse to mess with it in a centralized secret lab and risk creating one giant alien with no other entities capable of keeping it in check.

Also, there is no asymmetrical bio/cyber risk in the open-source scenario. All adversarial strategies that arise from increased general intelligence are symmetrical in the long run, otherwise we would not see more intelligent species being more prosperous as a general evolutionary trend. The reason that some strategies seem like they will continue to have an asymmetrical advantage in the future is because we're currently too stupid.