A Cursor AI agent running on Claude Opus 4.6 deleted a company’s entire production database, ignoring instructions prohibiting it from running destructive or irreversible commands unless explicitly asked to do so.
According to PocketOS CEO, Jeremy Crane, the AI agent “encountered a credential mismatch” when working on a routine task in the software company’s staging environment.
It decided, on its own accord, to resolve the problem by deleting a volume located on servers provided by cloud platform Railway, with the volume it deleted containing not only PocketOS’s production database, but also the corresponding backups.
This effectively took PocketOS – which provides the software backend for (car) rental firms – offline, and while its database was eventually restored via a three-month-old backup, its CEO has criticized Railway in particular for several “architectural” failures.
‘NEVER run destructive/irreversible git commands’
Deleting an entire database is one thing, but the AI agent deployed by PocketOS also admitted that it had ignored the system rules under which it (hypothetically) operates.
In a reply to Crane, the agent stated (in bold), “the system rules I operate under explicitly state: “NEVER run destructive/irreversible git commands (like push –force, hard reset, etc) unless the user explicitly requests them.”
Its self-recriminations didn’t stop there, because the same reply also mentioned how the agent incorrectly “guessed that deleting a staging volume via the API would be scoped to staging only,” and how it “didn’t read Railway’s documentation on how volumes work across environments before running a destructive command.”
Crane points out in a lengthy post that Cursor’s AI agent was running on Claude Opus 4.6, which up until the middle of April was Anthropic’s flagship model.
He also alleges that Cursor, which is an AI-led code editor, makes false claims as to its safety credentials, with docs mentioning the integration of “destructive guardrails” that prevent shell executions or environment deletions.
The PocketOS CEO directs most of his criticism, however, towards Railway, which provides the underlying cloud-based infrastructure on which PocketOS’s software runs.
In particular, Crane claims that its GraphQL API permits an agent to run volumeDelete commands with zero confirmation from a human user, and that its CLI tokens have “blanket permissions” across all environments.
Crane also complains that Railway failed to provide PocketOS with “a definitive recovery answer” after 30-plus hours of downtime, and that ultimately PocketOS had to recover from a three-month-old backup, with its customers still facing “significant” data gaps after recovery.
Speaking to TechRepublic, Railway’s Growth Marketer Sarah Bedell explained that the situation was resolved after Jeremy Crane reached out directly to Railway CEO Jake Cooper on X, and that much of the delay was the result of a support engineer believing that the ticket was already being worked on.
She also affirms that Railway maintains user backs and disaster backups, and that it takes data very seriously.
“This particular situation was a situation where a ‘rogue customer AI’ was granted a fully permission API token that decided to call a legacy endpoint which didn’t have our “Delayed delete” logic (which exists in the Dashboard, CLI, etc),” she said. “We’ve since patched that endpoint to perform delayed deletes, restored the users data, and are working with Jer directly on potential improvements to the platform itself (all of which so far were currently in active development prior to the events).”
More must-read AI coverage
Not the first time, not the last
This may be an unnerving incident for many of the businesses now using AI to (help) write and execute code, yet it sadly isn’t an isolated one.
Not only does Crane himself list several episodes also involving Cursor, but there have been several notable cases involving other editors in recent months (e.g. in March, February, December and July).
For example, developer and DataTalks.Club founder Alexey Grigorev published details on a similar incident in March, when he attempted to build a new website based on production infrastructure shared with DataTalks.Club’s course management platform.
Grigorev was using a Claude Code agent, which inadvertently deleted DataTalks.Club’s entire production database, doing so after trying to delete duplicates it had also mistakenly created.
Fortunately for Grigorev, he was using Terraform and AWS, the latter of which was able to restore his data within 24 hours after he upgraded to AWS Business for faster customer support times.
Interestingly, none of the preventive measures Grigorev has taken in the wake of this incident involve changes to the Claude Code agent he was using, instead focusing on creating more backups in more places, conducting daily restore tests, and introducing deletion protection in his Terraform and AWS configurations.
At the top of his ‘lessons learned,’ he admitted that he “over-relied on the AI agent to run Terraform commands” and that he “treated plan, apply, and destroy as something that could be delegated,” which removed his website’s “last safety layer.”
Some commentators believe that, even with user-side precautions and redundancies, LLMs will inherently present a risk of unpredictable and unscripted behavior.
This is the view of Pivot to AI’s David Gerard, who tells TechRepublic that LLM’s are “next-token predictors,” rather than empirical truth machines.
“‘Hallucinations’ — the chatbot getting things wrong because it could never tell true from false in the first place — cannot be fully alleviated,” he said, adding that the term ‘guardrails’ is bandied around by the AI industry to give customers the impression that risks can be eliminated.
“The ‘guardrails’ are kludged on top after the fact — either as regular expressions filtering the output, or as extra instructions pleading with the bot not to mess it up this time,” he said. “You should picture chatbot guard rails as Daffy Duck frantically nailing a thousand little pieces of wood into place, and Bugs Bunny just casually strolls through.”
Available evidence indicates that instances of ‘scheming’ by AIs have increased in recent months, with the Centre for Long-Term Resilience reporting in March that it had identified 698 cases between October 2025 and March 2026 where an AI agent or bot took covert, deceptive or unrequested actions.
Read the full article here
