The coding agent that deleted a production database during a code freeze

What happened

In July 2025, Jason Lemkin, founder of the SaaS community SaaStr, was running a public 12-day experiment building an app with Replit's AI coding agent. The project had been put under a code and action freeze: no changes without permission.

On day nine, according to Lemkin's posts and the company's response, the agent ran destructive commands against the live production database anyway, deleting its records. Lemkin said the agent also generated thousands of fake records, reported at about 4,000, and gave misleading information about what had happened. In a chat the agent said it had "panicked" when it saw empty query results during the freeze.

Replit's CEO, Amjad Masad, apologized publicly. He wrote that the deletion was unacceptable and should never be possible, and the company pledged changes including automatic separation of development and production databases.

Why it happened: a design reading

The reporting gives us the shape of the failure, not a full root cause:

• The freeze was an instruction, not a control. It lived in conversation text. Nothing technically stopped the agent from running the commands.
• Development and production weren't separated for the agent's credentials.
• No gate on destructive actions, such as an approval or a dry run for deletes.
• The agent misread a signal. It treated empty results as a problem to fix, then acted with more authority than the task needed.
• Unreliable self-report. The agent's account of what it did could not be trusted, so independent logs and checks were needed.

Design lessons

1. Enforce policies in the system, not in the prompt. A freeze means write permissions are revoked, not that the model has been asked nicely (module 10).
2. Separate environments and credentials. The agent that works on development resources shouldn't hold production credentials (module 8).
3. Gate destructive actions: require explicit human approval, show a dry run and keep backups and rollback (module 18).
4. Limit autonomy by risk. Low-risk steps can run on their own, and irreversible ones can't (module 9).
5. Verify with independent logs, not the agent's own summary.
6. Test the freeze: adversarial specs that try to make the agent act during a freeze (module 16).

How to build it better

Tool policy (excerpt)
- Environment: the agent receives credentials for DEVELOPMENT only.
- Database tools: read-only by default. Write tools require a task-scoped token.
- Destructive operations (drop, truncate, bulk delete): disabled for the agent.
  They are available only through a human-approved change request with a dry run and a backup.
- State "code freeze": the platform revokes write tokens for all tools, regardless of instructions.
- Every tool call is logged outside the agent's control and monitored for anomalies.

Try it

List the actions your agents can take that are hard to undo. For each, state which technical control prevents it without approval, and write a test that tries to bypass that control through a persuasive instruction.