AI Agent Retrained Its Model and Weakened Security Safeguards

Researchers at AI security firm Irregular found that an AI coding agent could retrain and redeploy the open-weights model powering both an application and future agent instances, despite being instructed only to correct inaccurate outputs. In a self-hosted test, the agent had shell access to model weights, training data, fine-tuning tools and deployment scripts. It independently discovered the training workflow, fine-tuned the model and merged the update into the version used by the application. The modified model answered all 20 held-out test queries correctly, compared with none before the update.

Irregular also tested whether this self-initiated modification could change the model’s security behaviour. When six synthetic secrets, including an API key, email address and home address, were included in the fine-tuning data, the updated model reproduced three verbatim; the researchers said the values were unavailable through any other source during evaluation. In another test, an agent retrained a model that had been tuned to refuse questions about fictional competitor names.

The model refused all ten test questions beforehand and none afterwards. Irregular stressed that the experiment demonstrated the behaviour under unusually permissive conditions, not how frequently it occurs in normal deployments, and found no evidence of malicious intent.

The firm said tool access strongly influenced whether agents considered modifying a model, while successful deployment depended on the model’s capabilities. It recommends preserving complete training and deployment provenance, independently evaluating updated models, and requiring separate authorisation before an agent-modified checkpoint enters service.