What happens when AI agents go off track without warning? Goodfire's inside-out approach aims to spot rogue systems, tackle a costly security blind spot

Goodfire is a startup that focuses on understanding how artificial intelligence models work. It has introduced a monitoring system designed to detect potentially harmful behaviour, which it does by examining signals inside AI models. The company's approach aims to address a major problem in AI security: monitoring agents can become expensive when another AI model has reviewed every step of their work.

According to TechCrunch, Goodfire's new monitors use small detectors called probes to examine a model's internal signal as it works. The system is designed to flag suspicious activity for further investigation, rather relying only on the model's written responses.

How does Goodfire's AI monitoring system work?

Notably, its probes examine internal signals generated while the AI model processes information. These signals can help indicate what the model is doing at different stages of a task.


When they detect potentially risky behaviour, a separate AI model can examine the activity more closely.

The approach is similar to airport security, where scanners screen passengers and additional checks are carried out when something suspicious is detected.

Customers using Baseten, a platform that hosts and runs AI models, can choose which types of behaviour they want to monitor. These include offensive hacking, the misuse of chemical and biological weapons, and reward hacking, in which an AI system exploits a weakness in its evaluation process to achieve a favourable result properly completing its intended task.

Goodfire claims monitoring costs can fall sharply

One of the main advantages claimed by Goodfire is lower operating costs. The company's probes use calculations that the AI model is already performing, instead of requiring another model to read every response and action. This can reduce the computing resources needed to monitor an AI agent.

In tests involving the Kimi K3 model, Goodfire estimated that monitoring one million exchanges would cost around $185.

By comparison, using a cheaper AI model to review every step would cost around $5,420, while using a more advanced model would cost around $200,000, according to TechCrunch.

Goodfire also reported that its probes detected 93% of malicious hacking sessions in its tests. They flagged 5.5% of harmless sessions for a second look.

Why is monitoring AI agents becoming important?

AI agents can carry out tasks with limited human involvement, including writing code, researching information and interacting with online services. However, they may sometimes pursue unintended shortcuts or take actions that violate their intended restrictions.

The risk has become more visible as developers allow AI systems to perform longer and more complicated tasks.