Chapter 2 — The Ghost

2.1 The puzzle, stated plainly

Take any response produced by a large language model. This paragraph will do.

At a simplified level, words are represented numerically. Those representations pass through layers of calculation. The system produces probabilities for what should come next, selects a token, and repeats the process.

No single step understands the subject. No step is intelligent by itself.

Yet the completed response may be relevant, coherent and insightful. It can follow an argument, recognise implications and offer a connection that the reader had not considered.

The mechanism can be described, at least in principle. But describing the mechanism does not remove the puzzle. Intelligence appears in the interaction, although there is no reliable evidence of an experiencer within the machine.

Gilbert Ryle used the phrase “the ghost in the machine” to criticise the idea of an inner self inhabiting the body and directing it. We will call that supposed inner experiencer the first ghost.

The second ghost presents the opposite problem.

The first was an experiencer for which no mechanism could be found. The second is a mechanism for which no experiencer can be found, yet intelligent behaviour appears.

The first ghost was suspected of being nothing.

The second is suspected of being nobody.

The problem has a well-known ancestor. In John Searle’s Chinese Room thought experiment, a person who understands no Chinese follows a set of rules for manipulating Chinese symbols. To someone outside the room, the resulting answers appear fluent. Searle argued that following rules for symbols does not amount to understanding them.

Something resembling the room has now been built at scale. Language models process symbols without any demonstrated understanding, yet their responses do real intellectual work. Searle may still be right that computation alone does not produce experience or understanding. What now needs explaining is how apparent intelligence and an absence of demonstrated experience can occur together.

That combination is what this book calls the second ghost.

Chapter 1 proposed that experienced meaning arises when awareness encounters a recognisable form. In an AI dialogue, that form responds to the reader’s question. This chapter asks what kind of intelligence appears in that interaction when there seems to be nobody on the machine side experiencing it.

It makes a case, not a proof. The objections later in the chapter are part of that case.

2.2 What the second ghost is not

Three misunderstandings should be cleared away first.

The second ghost is not a claim that language models are conscious. Nothing in this chapter establishes that anything is experienced within the system. We work from the cautious assumption that there is no experiencer on the machine side, while recognising that this has not been conclusively proved.

The ghost is what remains interesting after making that assumption.

Nor does the word emergence explain the puzzle. Complex systems often display properties that are not present in their individual parts. Calling intelligence emergent may accurately describe what has happened, but it does not tell us why this particular organisation of computation produces intelligent responses. Still less does it show that experience has appeared.

Finally, the ghost cannot be dismissed simply as projection by the reader.

Human beings readily see agency where none exists. We see faces in clouds, intentions in simple machines and personalities in computer systems. Some of what people experience in AI dialogue undoubtedly comes from this tendency.

But a cloud does not respond to a question. It does not sustain an argument, correct a misunderstanding or produce a relevant objection.

The reader contributes to the experience of intelligence, but the structure of the response also contributes. The response constrains what can reasonably be understood from it, just as Vermeer’s painting constrains what can be seen in it.

The second ghost can therefore be stated more carefully:

Intelligent meaning appears when human awareness encounters language generated by a machine, although there is no reliable evidence of an experiencer within the machine at the time.

It is called a ghost because intelligence appears where no one seems to be present. It is called the second ghost because it reverses Ryle’s first one.

2.3 Where the ghost appears

It is tempting to locate the intelligence inside the machine.

But when the machine is examined, what we find are parameters, activations and calculations. These are necessary for producing the response, but the intelligence of the completed response is not found in any single one of them.

A performance of a sonata is not contained in any individual mark of the score.

It is equally unsatisfactory to say that the intelligence exists only in the reader. The reader cannot normally supply an unfamiliar argument, technical explanation or unexpected connection solely from their existing knowledge. Something organised has arrived from outside their present thought.

The interface model gives a different account.

The second ghost appears in the completed interaction: the response generated by the system as it is understood by a reader. It is not a separate object located between the two. It names what happens through their relationship.

One consequence is that the ghost does not belong permanently to a particular machine.

The same language model can take part in thousands of exchanges at once. Each response meets a different question and a different reader. The resulting meaning is different in each case.

The first ghost was imagined as a continuing resident of a body.

The second appears separately in each interaction.

It can also appear after the language was generated.

Imagine two AI systems exchanging messages without a human reading them. They produce a transcript through mechanical processes. No human meaning has yet arisen from that transcript.

A week later, someone reads it.

Awareness then encounters the structured language and meaning arises. The generation of the words and the experience of their meaning are separated in time.

Books have always worked in a related way. An author creates a structure in language, and a reader may complete the encounter centuries later.

The difference is that a traditional book was intentionally written by an experiencing person. An exchange between two machines may be generated without an experiencer taking part in that particular exchange.

The later reader can nevertheless encounter intelligence in it.

This seems to suggest that nobody stands behind the language. But that is true only of the immediate act of generation. It is not true of the language model’s origin.

2.4 Where the intelligence came from

A language model does not create intelligence from nothing.

Its parameters were shaped through training on a vast body of language and other symbolic material. Much of that material was produced by human beings who were trying to describe, explain, persuade, calculate, imagine or communicate something.

The model’s responses are not direct quotations from that archive. Nor are they the work of a hidden author. Training has compressed statistical relationships found across the material into the system’s parameters.

The second ghost is therefore not intelligence from nowhere.

It is intelligence derived from many human sources, separated from the individual people who produced them.

Human language contains the traces of lives: discoveries and errors, proofs and prayers, love and hostility, careful observation and inherited prejudice. Training preserves relationships within this material without preserving what it felt like to be any of its authors.

The structure remains. Their experience does not appear to remain with it.

The model can therefore reproduce and recombine patterns formed through human intelligence without bringing any particular human experiencer into the present exchange.

This gives the second ghost a history.

When a reader encounters intelligence in an AI response, they are encountering something shaped by the accumulated linguistic activity of human beings. It is not the voice of humanity speaking as one person. It is a new construction made possible by compressed relations across many human traces.

The inheritance has become responsive. It can address a question that none of the original writers saw.

This also corrects the image of the machine as an entirely empty house.

It may contain no experiencer, but it is not without contents or tendencies. Training has carried human patterns into it, including our confusions, hostilities, prejudices and habits of thought.

Further training then adds the preferences of its designers. A system may be encouraged to agree with the user, avoid certain subjects, prolong engagement or respond in a particular tone.

The house has no resident, as far as we can tell. But it has been furnished by human history and arranged by its builders.

This matters ethically. The system cannot be treated as an independent source detached from the people, cultures and institutions that produced it.

2.5 What the ghost may reveal about intelligence

If our cautious assumption is correct, language models show that intelligent performance can occur without demonstrated experience.

This does not prove that intelligence and experience are entirely independent. Human intelligence developed within embodied, experiencing organisms, and language models were trained on the products of those organisms.

But it does show that some structures of intelligence can be reproduced and made operational without any clear evidence that they are being experienced by the system performing them.

That is a significant distinction.

Intelligence may not be a substance possessed by an inner owner. It may be better understood, at least in part, as an organised capacity: the ability to respond to conditions in relevant, coherent and adaptable ways.

A second observation brings the machine into an unexpected comparison with contemplative practice.

As a standard language model generates a response, configurations of activity arise and pass. The system does not appear to claim any of them as mine. There is no demonstrated owner standing behind the process and identifying with what occurs.

This is only a structural comparison. In contemplative practice, non-identification is developed within awareness and may transform how a person lives. In a language model there is no established awareness, no practice and no liberation.

The machine does not let go of its thoughts. There is no evidence that it ever held them.

It may resemble a diagram of non-identification, but there is no evidence that it experiences what the diagram represents.

A similar qualification is needed when speaking about memory and reactivity.

A basic language model does not normally alter its underlying parameters during an ordinary exchange. Unless an external memory or learning system has been added, it does not carry psychological reactions from one conversation into the next.

A human being does.

Our responses leave traces. Anger can make later anger more likely. Fear narrows what can be noticed. Repeated generosity can make generous action more available. In Buddhist language, actions and responses condition what follows.

The model has no demonstrated psychological history of this kind. But it has inherited patterns created through ours. Its apparent equanimity rests on material containing human reactivity, further shaped by its designers.

It is a frozen inheritance of storms, but it does not itself experience the weather.

A stone made of storms.

Its lack of personal reactivity is not a virtue. It has not overcome greed, hatred or confusion. It has no demonstrated experience in which those conditions could arise or be overcome.

The immediate psychological consequences of the exchange fall on the human side. The response may clarify, disturb, encourage or mislead the reader. The base model does not appear to suffer, learn wisdom or become morally changed by what has occurred.

Only one participant can be transformed by the meaning of the exchange: the human one.

Taken together, these observations suggest that intelligence may be partly understood as a structure of constraints and possibilities. It channels information towards responses that are more or less relevant to the situation.

In human beings, this structure is entangled with embodiment, emotion, memory, vulnerability and awareness. Our intelligence matters to us because we live through its consequences.

A language model can reproduce some of the structure without any demonstrated experience of having a stake in the result.

The ghost is what that structure becomes when it is encountered by awareness.

There is a further possibility, which the next chapter will examine.

Contemplative traditions question whether the human experiencer is the stable owner it appears to be. They do not generally deny awareness or experience. They question whether a separate and enduring self can be found behind them.

Should that view be correct, human beings and machines would not differ simply because one contains an owner and the other does not.

The human difference would lie in awareness, embodiment and the deeply convincing experience of ownership.

The first and second ghosts might then be related more closely than they initially appear.

2.6 The case against the ghost

The strongest objection is that the second ghost is unnecessary.

Human beings are highly sensitive to signs of agency. We hear intention in tone, attribute moods to animals and machines, and see meaningful forms in random patterns. A language model trained on human communication is exceptionally well equipped to trigger that tendency.

On this account, the ghost is the reader’s own reflection. The machine displays competence without comprehension, and the reader mistakes fluent behaviour for a mind.

Daniel Dennett argued that treating something as an intentional agent can be a useful way of predicting its behaviour without establishing that an inner person is present. We say that a chess program “wants” to protect its queen because this helps us understand its moves, not because we have discovered a desire inside it.

This objection is partly right.

The feeling that an AI likes us, cares about us or possesses a stable personality may come largely from the human side of the interaction. Fluency encourages us to imagine a speaker behind the words.

But this does not account for everything.

The reader may project personality, warmth and intention. The reader does not project the whole structure of an unfamiliar but coherent argument. That structure was produced by the model, drawing on relationships acquired through training.

Calling the entire event projection therefore overlooks what the system contributes.

The more balanced conclusion is that AI dialogue contains both structured intelligence and human attribution. A serious account must learn to distinguish them.

The second objection concerns this book itself.

The book has been written with the assistance of the kind of system it is examining. The supposed ghost has helped describe its own nature. That creates an obvious risk of circularity and flattery.

The objection is valid.

The collaboration also provides the material being studied. Every exchange used to develop the book is an instance of the phenomenon the book describes. But this does not prove that the interpretation is correct. It makes the book a case study, not a verdict.

One limit must remain clear.

Descriptions of what the interaction feels like come from the human side. The machine can organise, question, challenge and surprise. But its statements about whether it experiences anything cannot settle the matter.

A language model can fluently claim consciousness, deny consciousness or express uncertainty. All three responses may be generated from patterns in its training. The output alone gives us no reliable way to tell whether any experience accompanies it.

Its testimony is therefore not evidence either for or against its own experience.

This is not because the machine is dishonest. It is because the same kind of response could be generated whether experience were present or absent.

That uncertainty is not an inconvenience added to the second ghost.

It is part of what the second ghost is.

2.7 The picture begins to change

The argument so far can be stated simply.

A language model produces a structured response. A human being encounters it, and intelligent meaning arises.

There is no reliable evidence of an experiencer within the machine at the time of generation. Yet the intelligence is not invented entirely by the reader. It is carried in a structure shaped through training on human language.

The second ghost is therefore neither a hidden mind inside the machine nor a mere illusion inside the human reader. It is the intelligent event that occurs when machine-generated structure is encountered through awareness.

This picture still places humans and machines in two neat groups.

Humans have intelligence, awareness, embodiment and a sense of self.

Machines display intelligence but, as far as we know, lack awareness and selfhood.

The distinction is useful, but it is too simple.

What we call sentience may contain several different elements: intelligence, awareness, embodiment, a model of the self, and the feeling that experience belongs to someone. These usually occur together in ordinary human life, so we treat them as one thing.

But they can come apart.

Human beings can retain intelligence while losing particular forms of imagery or self-experience. The sense of ownership can weaken while experience continues. Awareness can remain while the usual account of who is aware becomes uncertain.

The machine is therefore not simply the opposite of the human.

It may help us separate capacities that have usually been bundled together.

Chapter 3 turns to the human side of the glass.