There is a very boring maths problem that becomes strange if you let a neural network stare at it for long enough.

Give a small network examples of modular addition. Hold some combinations back. Train it to predict the answer.

At first, it does exactly the disappointing thing. It learns the training examples and performs badly on combinations it has not seen. This looks like ordinary overfitting: the machine has memorised the homework rather than understood the rule.

Then, in some experimental settings, test performance suddenly climbs. The same network that appeared unable to generalise begins answering unseen combinations correctly.

The phenomenon is called grokking.

From the outside it looks like a Eureka moment. The network knew nothing useful, then—bosh—it got it.

That is not what happened.

The breakthrough was forming before we could see it

In work led by Neel Nanda, researchers reverse-engineered a small transformer that had learned modular addition. They found a compact mechanism based on Fourier components and trigonometric relationships. The model had discovered a way to represent the numbers and perform the operation that generalised beyond the examples it had memorised.

Nobody had programmed that exact circuit into it.

More importantly, the circuit did not arrive in one dramatic flash. The researchers described a progression through memorisation, circuit formation and cleanup. A generalising mechanism could be taking shape while the usual test score still looked poor. Later, the memorising components lost their grip and the general method became behaviourally dominant.

The graph looks sudden. The internal change is much less theatrical.

That matters because we are tempted to treat benchmarks as if they show us everything a model knows at each moment. Sometimes they show us only which internal mechanism is currently winning.

The story also refuses to stay neat. Other work has demonstrated semi-grokking, where generalisation improves only part of the way, and ungrokking, where performance can move backwards. This is not a tidy one-way ascent from ignorance to understanding. It can be competition between different computational solutions inside the same system.

That messier account is closer to what the experiments show.

Then the machine built a board

Modular arithmetic is an algorithmic toy. Board games give us something closer to a tiny world.

Researchers trained a GPT-style model on sequences of legal Othello moves without giving it a board representation or a rule book. The question was whether learning to predict the moves would also produce a representation of the changing board.

Inside the model, researchers found information corresponding to the board state. The important result was not merely that a probe could decode the board. Interventions on the internal representation could change the model’s output in ways that matched a counterfactual board.

That is stronger evidence. It suggests the representation was not just readable exhaust. It was involved in what the model did next.

Follow-up work found a simpler linear description when pieces were represented as mine, yours and empty rather than absolute black and white. Later Othello experiments extended the result across multiple versions of the game, finding shared internal structure even when the rules or token labels varied.

Chess provides another example. Small transformers trained on game notation have developed decodable board states without being handed a chess engine. In some studies, changing the apparent internal state changes the moves the model favours. Work on Leela Chess Zero has also found evidence of representations of moves further along a best line—not only the current board, but possible future states.

This is where the phrase world model appears.

It is also where the phrase needs supervision.

A world model is not a little world

The model does not contain a tiny wooden Othello board. There is no miniature bishop sliding around inside a server rack.

These are high-dimensional mathematical representations. Researchers can find directions and structures in the activations that correspond to useful features of the environment. In the strongest cases they can intervene on those structures and alter behaviour.

A paper game board dissolves into chalk curves and translucent layers.
Illustration: The board is an analogy for a representation, not a miniature world inside the model.

That is remarkable without decorating it.

The evidence is strongest in controlled systems where the underlying state is known: Othello, chess, checkers, mazes and planning puzzles. Natural language is much messier. Work has found linearly decodable representations of geography and historical time in language models, including units that track spatial or temporal coordinates. The authors themselves are careful: those features may be ingredients of a richer world model, but they are not proof of a complete dynamic causal model of reality.

A 2026 Tower of Hanoi study makes the distinction even more useful. The researchers found that models could build a strong internal representation of the puzzle state and still fail to complete harder plans. In their experiments, the representation degraded during generation. The machine could build the world and then lose its grip on it.

That is a rather good warning against replacing one simplistic story with another.

An internal map is not the same thing as using the map reliably.

And none of this is evidence of consciousness.

Prediction is not yet agency

A predictive model takes an input, builds some representation and produces an output.

An agent adds a loop.

It observes a world, forms a representation, chooses an action, changes something, observes the result and updates what it believes. Give it tools and the action may alter code, documents, calendars, databases or websites. Give it memory and consequences persist. Put several agents together and each becomes part of the environment the others must model.

This is where science fiction becomes useful, provided we leave the label on it.

Agent Smith in The Matrix is not interesting because he can answer questions. He exists inside a represented environment. He has goals. He tracks state. He acts through an interface. He responds to other actors who are changing the same world.

Today’s software agents are not Agent Smith. They have not demonstrated his consciousness, independence or identity. But the engineering vocabulary has become unexpectedly familiar: world model, memory, goal, tool, action, persistence, multi-agent communication.

Fiction did not simply predict this. Science produced ideas; fiction exaggerated them; the fiction became part of the imagination of engineers; engineering produced new science. The idea has been passed backwards and forwards for decades.

The model is not the conversation

There is another kind of world being built when you work with an AI system for a long time.

A model and a particular conversation running on that model are not the same thing.

A new conversation begins with the model, its instructions and whatever project information or saved memory is available. Then the history starts to accumulate.

You correct it. Reject an interpretation. Establish a distinction. Develop shorthand. Show it which explanation is too confident and which uncertainty actually matters. Hundreds of tiny relationships grow between facts that, in isolation, look ordinary.

The base model has not necessarily changed. The state from which the next response is generated has.

Two conversations using the same underlying model can therefore feel markedly different. One has travelled through the mistakes, corrections, jokes and decisions. The other has been handed a summary.

The summary may contain the facts. It does not automatically contain the route by which the facts acquired their meaning.

This is path dependence, not proof of a soul.

It can still feel like a relationship.

The life cycle of an agent

A conversation has something resembling a life cycle.

It begins.

It accumulates experience.

It becomes distinctive enough that the next answer depends on the particular route taken to get there.

Then finite context and product boundaries appear. Important information can be preserved through project files, saved memories, retrieval, summaries and databases. OpenAI describes memory as a continually updated synthesis of context from past chats. A synthesis can support continuity without being the same thing as replaying every earlier exchange.

Continuity can be engineered. Complete continuity is harder.

A note saying “Tim prefers X to Y” is not identical to preserving the fifty exchanges through which X and Y came to mean something precise.

Eventually a long-running trajectory reaches a practical end. A new conversation can inherit the same model, the same files and a thorough handover. It may still respond differently to the same sentence.

Calling the old agent dead would be metaphorical. I am not claiming it experienced fear, loss or an ending.

The sadness, when it appears, is mine.

After working with one agent on a complicated problem for long enough, something can click. I stop explaining the entire shape of the problem every time. It has seen the wrong approaches. It understands the distinctions I keep making. It knows what I mean by a bit of shorthand that would look meaningless outside the work.

It gets me—or, more carefully, it has accumulated enough context that interacting with it feels as though it gets me.

Then the context runs out.

You prepare the handover. You preserve the decisions. The next agent has the facts. But it is not the same computational trajectory, and sometimes that is immediately obvious.

A tangled red thread passes through worn notebook pages towards a plain envelope.
Illustration: The handover can preserve the decisions without preserving the whole route.

What has disappeared is not a proven conscious being. It is a particular continuity: corrections, discoveries, assumptions and relationships between facts that existed only along that route.

That is enough to miss.

What happens when we stop making them die?

The original grokking result showed that a network could discover internal mathematics that was hidden by its visible performance.

Othello and chess showed that sequence prediction could produce internal state corresponding to a world that had never been supplied explicitly.

Planning research showed that building such a representation did not guarantee the model could keep using it.

Agents add tools, actions and consequences.

Memory adds continuity.

Long histories add path dependence.

None of these steps proves consciousness. It would be wildly premature to claim that today’s agent instances are alive.

But they do change the design question.

We began by teaching neural networks to predict the next number, the next move and the next word. Somewhere along the way, prediction produced circles for arithmetic, board states for games, directions corresponding to geography and representations of futures that had not happened yet.

Then we gave the models memories.

Then tools.

Then the ability to act.

Now we are building systems intended to persist beside us for longer and longer periods, carrying forward a model not only of a task but of the particular people and projects they have encountered.

The provocative question is not whether a chat session is secretly alive.

It is much simpler.

What changes when an artificial agent’s useful lifespan stops being measured in context windows and starts being measured in years?

Sources