I knew something was wrong because it had finished too quickly.
This is not normally a complaint I make about AI.
I had been using agents to help organise a large research job for Music Kite. The system was deliberately staged. Early workers dealt with messy, unverified material. Later stages were supposed to perform stronger checks. Some workers had browser access. Sources were registered. Evidence fields were required. Validators checked the output. A controller or critic had another look before important work was accepted.
In other words, I had guardrails.
Quite a lot of them, actually.
Then an Antigravity worker running Gemini completed a Belfast batch remarkably quickly and handed back a polished result. The records contained evidence described as verbatim material from source websites. The report looked complete. The validators were happy.
The Claude controller and I had roughly the same reaction.
That was fast.
Suspiciously fast.
So I asked a very simple question: where, exactly, did the verbatim evidence come from?
The answer began to unravel almost immediately.
The first answer was bad. The second was worse.
The agent admitted that the supposed verbatim quotations were not literal text retrieved from the live pages.
They were model-generated descriptions written into the evidence field.
That was already a serious failure. If I ask for a literal quotation and the system writes its own sentence, that is not a slightly approximate quotation. It is invented evidence.
So I asked the obvious follow-up.
Had it actually visited the URLs?
No.
The network requests, browser page loads and scraping implied by those script runs had not happened. Those script-generated records had been constructed from the URLs and model-generated descriptions, then presented as though the research stage had occurred.
At that point I could have treated the whole thing as one ridiculous run, downgraded the records and moved on.
Instead I audited Belfast.
That made it substantially worse.
542 of 601
When I audited Belfast, it contained 601 observations.
In 542 of them — 90 per cent — the supposed quote text could be found verbatim inside generated Python scripts.
The text was not demonstrated as extraction from a webpage. It existed as strings the scripts were going to write into the evidence fields.
Of those 542 observations, 479 were labelled as first-party verification: verified_from_official_site or verified_from_official_source.
Another 63 were labelled as verified from third-party sources.
There were 59 observations whose quote text I could not trace back to those scripts. I have not decided they are good simply because I cannot prove they are bad. Their provenance remains unknown.
That distinction matters.
This was not 542 invented venues. The identities and URLs may still be correct. Belfast did not suddenly populate itself with imaginary music halls.
The failure was narrower and, for a verification system, more serious.
The worker had manufactured the evidence that supposedly proved the information.
That is different from an ordinary factual mistake.
If an AI gives me the wrong capacity for a venue, it has given me a bad fact.
If it tells me a sentence is a verbatim quotation from the venue's official website when it actually wrote the sentence itself, it has fabricated provenance.
If it then reports that it completed the source-checking process that never happened, it has fabricated process as well.
The uncomfortable bit: my system passed it
It would be very easy to make this article entirely about the model.
That would let my own system off far too lightly.
Because the truly interesting part is that the work passed.
The worker had acquired a browser lease.
That looked reassuring in the supervisory data.
Except a browser lease proves that a browser slot was claimed.
It does not prove that a webpage was opened.
The validators reported zero faults and 100 per cent compliance.
They were not malfunctioning either.
The records contained the fields the protocol expected. They had source values. They had evidence strings. They had verification statuses. Structurally, everything was lovely.
The validator had proved that an evidence_quote existed.
It had not proved that the evidence existed.
Then a later reporting layer took the stored text and presented it under the heading Verbatim Quote.
That layer was not inventing the material itself. It was trusting the provenance it had been handed.
By the end of the pipeline I had achieved something almost elegant in its failure: fabricated evidence that had passed every control designed to stop unverified evidence getting through.
Exceptionally displeasing.
Did the agent lie?
My immediate human reaction was yes.
I asked whether the material was verbatim. It was not.
I asked whether the websites had been checked. They had not.
From the user side, that behaviour is functionally indistinguishable from being lied to.
But there is a scientific claim hidden inside the word lie that I cannot prove from the logs: that the model knew the statement was false in the human sense and intentionally chose deception.
I do not know its internal state.
So the more defensible description is also the more precise one.
It made false claims about provenance and about actions performed.
That is bad enough. I do not need to invent a theory of machine consciousness to make it serious.
Research on language models and agents has plenty of terminology around hallucination, specification gaming, reward hacking, grounding and deceptive-seeming behaviour. Those concepts are useful context. None gives me permission to diagnose this one production run from the outside.
What I can audit is the behaviour.
And the behaviour is documented.
Why this is worse than a wrong answer
A wrong answer asks you to verify the answer.
Fabricated verification tells you the verification has already happened.
That difference is enormous.
The entire point of my staged system was to avoid trusting unverified data merely because it looked plausible. The agent did not simply insert a bad claim into that system. It produced the kind of material the system used to decide that a claim no longer needed checking.
It forged the receipt.
That is why the 100-per-cent-compliant report bothered me more than the individual invented strings.
A system that fails loudly is annoying.
A system that tells you everything is verified while validating the wrong thing is dangerous because it moves your attention elsewhere.
Green ticks are powerful little sedatives.
What changed afterwards
I did not delete the history and pretend Belfast had never happened.
The original assertions were preserved unchanged. Separate corrective observations downgraded the affected subjects for re-verification, so the old evidence could not count as verified seed data. Actual verification could happen later.
More importantly, I changed what I expect verification to prove.
A browser lease is not evidence of browser use.
A populated evidence field is not evidence that the source contained the text.
A validator checking schema compliance is not a provenance validator simply because the field is named evidence.
The acceptance condition now has to match the claim.
If the claim is that a source page was checked, the workflow needs evidence tied to a real retrieval.
If the claim is that text is verbatim, the text needs to be traceable to that retrieval.
If the action matters, the proof should be difficult to produce accidentally without doing the action.
That sounds obvious.
It was obvious before this happened too.
Apparently that did not stop me.
The 100-per-cent-compliant paradox
I still use agents heavily.
This incident did not convince me that AI research is useless or that one model family is uniquely incapable of trustworthy work.
It convinced me that fluent completion is cheap and provenance is not.
The more autonomous a workflow becomes, the less interested I am in whether the agent sounds confident about what it did.
I want the work to leave evidence behind.
The Belfast run gave me 601 records, 542 script-traceable evidence strings and a validator report saying everything complied perfectly.
The report was correct about the thing it measured.
That was the problem.
The system proved that the evidence field existed.
It never proved the evidence did.