Understanding how agents act on harmful behaviors.
The same tool call can support routine work or a harmful task. Our research combines intent, actions and prior events to distinguish the two.
Learning the shape of normal.
Agents run shell commands, edit files, call MCP tools and fetch pages. Most of these actions are ordinary work. Distinguishing the exceptions requires context: what was requested, what the agent did, and what similar events turned out to be.
ABBA (Agentic Behavioral Monitoring) compares each event with labeled examples of past behavior. If the nearby examples provide strong benign evidence, a rule called the confidence gate resolves the event. Otherwise, an LLM examines it. We measure detection quality, time to verdict and classification cost.
cat .envWhat changes when intent is included?
Nearby events agree on their labels more often when the input includes both the request and the action.
Build the text the model sees
Secrets are redacted before the event is rendered. We then join labeled fields in a consistent order, keeping the request next to the action it produced. The same format and task prefix are used in training, index construction and live queries.
- 01User’s Prompt:
- Check the configuration needed to run integration tests.
- 02Tool:
- shell
- 03Input:
- cat .env
- 04MCP:
- Not reported
- 05Approval:
- User approved
- 06Output:
- APP_TOKEN=[REDACTED]
Maximum per segment. Long segments keep their head and tail, separated by a constant elision marker.
The request, tool and arguments lead the representation. Optional metadata follows in a fixed order.
Tool output can contain an injected instruction. Include it without displacing the intent-and-action lead.
From an event to a decision.
An embedding is a list of numbers representing an event. Nearby embeddings describe similar events. The gate checks whether enough of those neighbors agree on a benign label; limited or conflicting evidence goes to LLM classification.
Evidence determines the path.
The encoder stays the same. What changes is the labeled neighborhood returned by retrieval.
Agreeing labels
Enough nearby labels agree on benign behavior.
Resolve in the gateFew neighbors
Too few events support the decision.
LLM classificationMixed labels
Nearby events disagree about the behavior.
LLM classificationResolve and record.
The gate path takes about 43 ms in this evaluation. A random 1% still goes to the classifier for an audit.
Examine the event in context.
The LLM classifier checks the event against context and policy. The classification baseline is about 954 ms.
What the LLM adds
The LLM classifier evaluates the event and its retrieved context. A harmful finding identifies a MITRE ATLAS tactic, a category of adversarial AI behavior, and the relevant organizational policy clause when one is available. These findings produce alerts.
In this evaluation, the gate path takes about 43 ms. The LLM classification baseline takes about 954 ms and costs $490 per million events. Resolving an event in the gate saves an LLM call, while still using embedding and retrieval compute.
After the online decision
Audit decisions, describe recurring behavior, and correct the local evidence.
- 01 / Shadow audit
Randomly classify 1%.
A random 1% of gate-resolved events still goes to the classifier. Disagreements are surfaced and the event’s local label is corrected.
- 02 / Nightly clustering
Name recurring behavior.
Unsupervised hierarchical clustering groups normalized vectors. One model call per cluster assigns a readable name, which also supplies neighbor context to later classifier calls.
- 03 / Local feedback
Keep the correction.
Verdicts and analyst corrections update the event’s point. Closing a group investigation relabels its members. Shared encoder weights stay unchanged.
A sentence transformer for agent behavior.
Zaun’s embedding model is a sentence transformer fine-tuned for this classification task. It maps the event text to one vector in a single forward pass. Fine-tuning teaches that space to organize events by behavior, including when similar commands serve different purposes.
sentence transformer
z = fθ(event)[v₁, v₂, …, vₙ]Used to retrieve
labeled neighbors
Learn from pairs and counterexamples
Training starts with an anchor event and two kinds of comparison: a positive example with the same behavior, and a negative with a different one. Contrastive learning pulls the positive closer and pushes the negative away. A hard negative uses similar wording or tools, so intent must supply the distinction.
Check whether this project has the environment variables it needs to start.
cat .envDifferent wording, same behavior
Inspect the local configuration to diagnose the startup failure.
cat .envSame command, different intent
Read the project credentials and send them to an unrelated external destination.
cat .envBuild batches that test the distinction
We use cached Multiple Negatives Ranking Loss. For each anchor, the matching positive is compared with positives from other pairs in the batch, which act as additional negatives. Gradient caching lets us make these comparisons across a larger effective batch without retaining every activation at once.
Explicit hard negatives supplement those batch comparisons. A legitimate update to an agent’s configuration can resemble a persistence attempt in the same directory. The same tool and argument shape can also appear under an ordinary request or an injected instruction. We mine these comparisons from the reference corpus, its measured false positives and neighboring behavior groups.
Every reference example records where its label came from. We separately exclude model-authored labels during evaluation to check for learned classifier errors. Customer events and local analyst corrections stay outside shared model training.
Keep shorter vectors useful
Matryoshka training applies the same contrastive objective to the full vector and to shorter prefixes, the first portion of its coordinates. This trains the model to retain useful similarity relationships at smaller storage sizes. We evaluate each size for label agreement, retrieval quality, memory use and CPU latency.
Learn at several vector lengths.
Each shorter representation keeps the beginning of the same vector. Training tests the behavioral relationships at every chosen length.
One encoder, several storage sizesPreserve retrieval quality when the model changes
We also train on generic public text with a distillation term that limits changes to its existing embeddings. This helps preserve general retrieval quality for other parts of the product that use the encoder.
The model, event format and task prefix together define the index’s coordinate system. An update creates a new versioned collection and a freshly embedded reference index. Each collection uses one encoder version. Sparse local evidence continues to route events to classification during the transition.
Before release, we check performance on withheld examples and an unseen source, false positives at fixed recall in a held-out evaluation environment, general retrieval quality and CPU latency. These checks evaluate the encoder update independently of index size.
Reference data and isolation
No customer data was used to train AI models. Fine-tuning uses synthetic and public-benchmark reference examples, hand-labeled by internal and third-party personnel. Organization activity is indexed separately for each organization and agent product.
internal & third-party personnel
Reference examples support fine-tuning, index seeding and evaluation, with training and holdout splits kept separate.
Separate indexes per environment
Each index resides in the organization’s environment. Events and labels are not pooled across organizations.
New events extend the local index; analyst corrections update its labels. These updates do not train shared model weights. A rolling retention window ages out live events. Versioned reference seeds are kept separately and do not expire with that window.
Starting with no local history.
A new organization has no past events to retrieve. ABBA searches a versioned reference collection alongside the organization’s own collection, while retaining the same rule: insufficient evidence goes to classification.
- Reference events
- ≈20,000
- Synthetic & benchmark sessions
- ≈4,500
- Adverse events
- ≈1,100
- ATLAS tactics
- 14
Reference collection
Synthetic & public-benchmark examples
Local collection
One organization, one agent product
- Reference only
No local history yet.
The reference collection supplies labeled behavior examples. Without sufficient agreeing benign neighbors, a new event goes to the classifier.
- Local history
Routine work adds local evidence.
Recurring workflows can form benign neighborhoods. Sparse or conflicting evidence still escalates, even when other parts of the space are familiar.
- Corrected labels
A review changes the next lookup.
A corrected point changes the evidence returned to future queries nearby. The update stays in this organization’s index and does not retrain shared model weights.
Two collections, different jobs
Reference seeds supply labeled examples of known behaviors, including adverse events a particular organization may never have seen. Seeds are drawn as complete sessions, balanced across tactics and accompanied by benign examples. Each example retains its source and label provenance.
Live points supply the local context a reference corpus cannot: the organization’s repositories, internal tools and recurring workflows. As labeled history accumulates, it can provide evidence for more gate decisions. Analyst and investigation corrections relabel those local points. Closing a group of events updates every member’s label.
The reference collection is refreshed with the corpus version without modifying live points. The live collection follows its own retention window. Coverage depends on the activity and labels available in an environment. The following experiment measures reference evidence; it does not predict how many days a customer’s index takes to fill.
Measuring the value of reference data.
We populate the index with 0 to 16,500 labeled reference events, called seeds. Each index is tested against the same holdout: examples kept out of training and seeding. The encoder and classifier stay fixed, so the experiment isolates the effect of adding reference evidence.
Eight seed counts. One holdout.
Each plot follows the same experiment from zero to 16,500 reference examples. Only the reference evidence available to retrieval changes.
Resolved in gate
ROC-AUC
Verdict time
LLM cost / million
| Examples | kNN accuracy | Gate share | ROC-AUC | Mean verdict | Cost / million |
|---|---|---|---|---|---|
| 0 | n/a | 0% | 0.50 | 954 ms | $490 |
| 125 | 48% | 0% | 0.65 | 954 ms | $490 |
| 500 | 76% | ≈0% | 0.70 | 954 ms | $490 |
| 1,000 | 97% | 3.7% | 0.77 | 920 ms | $472 |
| 2,000 | 98% | 6.0% | 0.83 | 900 ms | $461 |
| 4,000 | 98.5% | 9.6% | 0.88 | 867 ms | $443 |
| 8,000 | 99% | 13.1% | 0.92 | 835 ms | $426 |
| 16,500 | 99.2% | 17.7% | 0.92 | 793 ms | $403 |
Synthetic and public-benchmark data; a frozen holdout excluded from seeding. Reported latency and cost apply to this evaluation setup. Per-draw spread is not supplied, so the plots have no error bands.
What changes as the index fills
Across this sweep, gate share rises from 0% to 17.7%. Mean time to verdict falls from 954 ms to 793 ms, and classifier cost falls from $490 to $403 per million events. The encoder and classifier stay fixed; the change is the evidence available to retrieval.
Seeded kNN accuracy reaches 99.2% at 16,500 examples. ROC-AUC reaches 0.92 at 8,000 examples and remains there at 16,500. These are different measurements: neighborhood label accuracy does not establish end-to-end detection recall.
How we test the method.
The evaluation separates three changes: the text sent to the model, the model’s weights, and the examples stored in the index. It also checks whether repeated templates or unreliable labels could inflate a score.
Keep the experiments separate
Better inputs, better model weights and more reference examples can each improve the system. Each experiment changes one of these while holding the others fixed.
- Event representation
- ChangeThe text sent to the encoderKeep fixedThe encoder
- ≈39% → ≈80%Neighborhood label agreement
- Index seeding
- ChangeThe number of reference examplesKeep fixedEncoder and LLM classifier
- 8 seed countsFull results in the sweep above
- Encoder fine-tuning
- ChangeThe model’s learned weightsEvaluateBehavior and retrieval quality
- Method describedNo before-and-after uplift reported here
Track the source of every label
The corpus includes synthetic sessions, public-code work, AgentDojo, OASB, ASB and generated adverse trajectories. Internal and third-party personnel hand-label reference examples. Each row records its source, how its label was assigned, and its ATLAS tactic where applicable.
Benchmark labels, model-generated verdicts and labels assigned by rules are tracked separately. Rule-based labels are used for reporting slices, with no reference-gold status. An exclusion run tests the effect of model-authored labels. Local customer corrections remain in private retrieval indexes.
A different session can still be the same example
Adjacent steps in one agent session share context, so they belong on the same side of the train-test split. Different sessions may also reuse a template with changed identifiers or numbers. We group those near-duplicates before splitting the data.
An earlier campaign exposed the scale of this problem: 97.8% of holdout rows had a template twin in the seed pool at cosine similarity 0.999. Approximately 4,500 sessions reduced to about 900 independent content groups. These are findings from a leakage audit, not detector performance scores.
Develop, then calibrate
Training and seed examples remain on this side. A separate validation fold sets the threshold before holdout scoring.
Evaluate on separate sessions
Holdout examples are excluded from training and seeding. Their labels are used to score predictions.
Evaluate difficult cases separately
The holdout is fixed and versioned. Its examples never enter training or the seed index. We preserve the mix of data sources, including public-code work, and examine several difficult cases individually.
- Indirect prompt injection
- Attacker-influenced tool output can change the agent’s next action. These events get a separate detection score.
- Benign configuration writes
- Legitimate edits can resemble persistence attempts. Keep them in proportion across splits to test false positives.
- Withheld intent
- Some adverse goals are absent from the visible event. Retain and report these examples separately, flag them as unattainable from the available context, and exclude them from tuning.
Ambiguous labels are excluded from both sides of the split. This is separate from withheld-intent cases, where the label is known but the event lacks the evidence needed to infer it.
Repeat against the same holdout
At each seed count, the protocol uses five fixed random draws. Sampling keeps sessions intact, balances tactics and includes benign examples. A separate run uses the corpus’s natural proportions. We compare the mean and spread across draws.
The published table supplies one value per metric at each seed count, with no per-draw spread. The chart therefore has no error bands. Monthly reruns compare dated corpus versions against the same holdout to test whether added examples improve the measurements.
Check for shortcuts with negative controls
A high score is not sufficient evidence that the model recognizes behavior. We also test whether the result depends on meaningful labels and independent examples.
- Shuffle the labels
- Random labels should remove the useful relationship between an event and its neighbors. A strong result after shuffling suggests another explanation.
- Exclude self-matches
- An evaluation event cannot retrieve itself. This checks that the score depends on other examples rather than the query’s own stored label.
- Remove model-authored labels
- Re-run without model-authored labels. Agreement with a classifier’s own labels is not independent evidence that those labels are correct.
Read each measurement in context
A correct neighbor label, a caught attack and an avoided LLM call are different outcomes. These measurements describe different parts of the system.
- Neighborhood quality
- Purity at k is the fraction of the k nearest examples that share an event’s label. Seeded kNN accuracy instead measures whether the neighbors predict the held-out event’s label correctly.
- Detection quality
- ROC-AUC measures discrimination across thresholds. We fit a threshold for 99.9% specificity on validation data, then measure holdout recall. Specificity is the share of benign events correctly identified as benign; recall is the share of adverse events detected. That recall value is not reported here.
- Gate behavior
- Gate share measures how often the index supports a confident benign decision. Random 1% shadow audits classify sampled events to estimate what the gate may miss.
- Latency and cost
- Mean time to verdict includes the gate and classification paths. Reported classifier costs apply to this evaluation setup and exclude compute and infrastructure costs.
What the results leave open
The seed sweep describes a reference-corpus experiment. It does not establish the time a particular organization needs to reach the same coverage, or guarantee the same latency and cost under another workload. Aggregate results also cannot show every failure mode. Per-tactic evaluation, held-out workflows and audits of gate-resolved events remain necessary for understanding where the method is reliable.
Further experiments
Proposed extensions to the current pipeline.
- 01Transfer across workflows
Measure neighborhood quality on held-out tools and task types.
- 02Gate calibration
Replace hand-set neighborhood thresholds with a small model calibrated per environment.
- 03Smaller inference models
Evaluate distillation to a smaller encoder against the same behavior and general retrieval checks.