Northflank tells customers to assume every stored secret was decrypted

Northflank, a cloud platform for deploying applications and AI workloads, has told every customer to assume all environment variables have been decrypted and to rotate every key stored on it. Customers are already reporting that AI API keys they kept on the platform have been used by someone else. Northflank has not said how the attackers got in.

The warning came from outside. A threat alert from Wiz reached Northflank at 11:43 UTC on 30 October, according to the timeline on its status page, and engineers deployed a first patch 19 minutes later. At 13:40 UTC the company posted its first instruction to customers: “Rotate environment variables immediately.” By 15:12 UTC it reported that “multiple users are confirming that their AI keys have been used.”

What Northflank has said

The incident is still open. At 19:28 UTC on 30 October the company said it had “no further confirmed findings to share.” Four hours later it confirmed that “users have reported unauthorised use of AI API keys supplied to the Northflank platform,” and at 01:04 UTC it turned on audit logs for all customers, including free accounts. Secret rotation for managed Postgres and MongoDB databases arrived on 30 October, and for MySQL and Redis at 13:34 UTC on 31 October. The status page still lists networking, services, add-ons, jobs and builds as a major outage.

Northflank has not said how the environment variables were decrypted, whether they were copied out, how many customers are affected or how long the exposure lasted. It had not answered those questions from BleepingComputer by the time of its report on 31 October.

One of nine

Northflank's status page points to a report Wiz published the same day, saying the reports of key abuse “may be consistent with the broader pattern of attacks targeting AI credentials” and that the report contains similar IP addresses. Wiz counts nine AI developer platforms that disclosed incidents in October: Fireworks AI, Andromeda, Zeabur, Lightning AI, OpenCode, Railway, Warp, CommandCode and Northflank. Public reports, it writes, show attackers “explicitly hunting stored environment variables, with a focus on those granting AI access.”

The stolen keys are worth money. In two cases Wiz investigated, single Amazon Bedrock keys made about 800,000 calls in 30 hours and about 1.7 million in 13 hours, which Wiz estimates ran up more than $100,000 in usage at list price. It found two operations reselling AI model access that had checked the permissions of stolen keys from their own servers, but it could not tie them to the original break-ins. The platforms give different root causes, from credential access to a third-party vendor, and Wiz has not concluded that one group is behind all nine.

What to do

Northflank customers should rotate every secret held in workloads, secret groups and global secrets. Rotating the value inside Northflank is not enough on its own: the company asks customers to revoke existing credentials with their issuing providers, because a key copied before rotation keeps working until the provider revokes it. It also asks them to check AI provider usage and billing logs for unexpected activity, and to disable public access to databases and other add-ons where it is enabled.

Teams that used any of the other eight platforms this month can find each disclosure, with its date, in Wiz’s report. Northflank says it will share further updates as more information is confirmed.