Classify each line of an agent's memory by type, then route each type through its own retention policy so safety rules survive compaction.
Saber Zerhoudi, Jelena Mitrović, Michael Granitzer University of Passau, Germany · Published at CIKM '26, Rome, Italy, November 7–11, 2026 · Read the paper
What is this? Long-running AI agents summarize their context when the token budget overflows, and a safety rule gets collapsed at the same rate as an episodic log even though only the rule needs its exact wording to stay enforceable. Knowledge Triage is a framework and reference implementation that classifies each item of an agent's knowledge base by type and applies three deterministic, type-aware operators (TypeCompact, TypeDecompose, TypeRetrieve) so constraints survive by construction.
Who is it for? Researchers studying agent memory and context management who want to reproduce the paper's results, and practitioners building long-running agents who need compaction that keeps safety rules intact. Practitioners who want it running in Claude Code today can install the companion agent skill: searchsim-org/knowledge-triage-skill.
Contents: Abstract · Highlights · Citation · Quick start · Library usage · Reproducing the paper · Data · Architecture · License
A safety rule and an episodic log compete for the same tokens in an AI agent's context. When the budget overflows, both are summarized at the same rate; only the rule needs exact wording to remain enforceable. On 20 production agent configurations, Claude Code's
/compactprompt on Sonnet 4.6 preserves 53% of safety rules after one compaction round and 10% after five. We name this the Compaction Cliff. We address it with Knowledge Triage, a framework that classifies each line of an agent's knowledge base by type and routes each type through its own retention policy. Three deterministic operators implement this triage across the three context-management operations: TypeCompact rewrites items in place under per-type fidelity, TypeDecompose partitions a topic too large to compact safely, replicating in-scope safety rules across partitions, and TypeRetrieve fetches items from external storage with in-scope rules pinned ahead of relevance. On five public corpora, TypeCompact preserves 2–4× more safety rules than the strongest single-shot LLM compactor at every ratio, with 96% recall over five rounds. TypeDecompose reaches 0% locality violations against 93% under uniform partitioning. TypeRetrieve reaches 100% recall@50 against 73% for the best single-shot LLM retriever. On three downstream behavioral benchmarks, we outperform the production Sonnet compactor on medical compliance (paired McNemar p < 10⁻⁸ on preservation, N = 200), the full-policy and hierarchical baselines on retail task pass rate (p < 0.01, N = 115), and the hierarchical compaction on the airline domain (p = 0.024). We release AgentArtifactCorpus (396,934 agent configurations from 54,628 public GitHub repositories), the classifier, and the reference implementation.
Keywords: AI agents, agent memory, context compaction, knowledge management, long-running agents
- The Compaction Cliff: on 20 production agent configurations, Claude Code's
/compact(Sonnet 4.6) keeps 53% of safety rules after one round and 10% after five. Vanilla LLM compactors lose roughly half their remaining rules every round. - Type-aware compaction: TypeCompact preserves 2–4× more safety rules than the strongest single-shot LLM compactor at every ratio, holding 96% recall across five sequential rounds.
- Structural guarantees: TypeDecompose reaches 0% locality violations (93% under uniform partitioning); TypeRetrieve reaches 100% recall@50 in scope (73% for the best single-shot LLM retriever).
- Deterministic and cheap: the operators run in under 1 ms with 0 LLM tokens per call, against 5–420 s and 36k–57k tokens for LLM baselines.
- AgentArtifactCorpus (AAC): 396,934 agent configuration artefacts from 54,628 public GitHub repositories under permissive licences, released with a datasheet and opt-out infrastructure.
If you use Knowledge Triage or AgentArtifactCorpus, please cite the paper.
@inproceedings{zerhoudi2026compaction,
author = {Zerhoudi, Saber and Mitrovi\'{c}, Jelena and Granitzer, Michael},
title = {The Compaction Cliff in Long-Running AI Agent Memory},
year = {2026},
booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management},
series = {CIKM '26},
location = {Rome, Italy},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
isbn = {979-8-4007-2539-5},
doi = {10.1145/3799682.3840567},
url = {https://doi.org/10.1145/3799682.3840567},
}ACM Reference Format: Saber Zerhoudi, Jelena Mitrović, and Michael Granitzer. 2026. The Compaction Cliff in Long-Running AI Agent Memory. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM '26), November 07–11, 2026, Rome, Italy. ACM, New York, NY, USA. https://doi.org/10.1145/3799682.3840567
GitHub also shows a "Cite this repository" button generated from CITATION.cff.
Production AI agents that run for hours manage their context through three operations: compaction, decomposition, and retrieval. The state of practice delegates all three to the same uniform policy, even though a safety rule cannot be paraphrased the way an episodic log can be collapsed. Knowledge Triage closes the gap with three type-aware operators that share a single classifier and verifier.
| Operation | State of practice | This work | Improvement |
|---|---|---|---|
| Compaction at 10% ratio | 0.24 constraint recall | 0.80 | 3.3× |
| Compaction over 5 sequential rounds | 0.10 | 0.96 | 9.6× |
| Decomposition locality violations | 93% of configs affected | 0% | structural |
| Retrieval recall@50 in scope | 0.73 | 1.00 | +27 points |
| Declarative-rule classification | 0.62 recall | 0.90 (SafetyMargin) | +28 points |
| Per-call latency | 5–420 s, 36k–57k LLM tokens | <1 ms, 0 tokens | deterministic |
Numbers are the mean across 20 stratified configurations from AgentArtifactCorpus. State of practice = Claude Code's /compact (Sonnet 4.6) for compaction, dense top-50 cosine for retrieval, uniform token partitioning for decomposition, one-shot gpt-5.4-mini for grammatical-form classification.
Sequential compaction at 50% per round, measured against the original safety-rule set:
round 1 2 3 4 5
TypeCompact 1.00 0.96 0.96 0.96 0.96 flat
gpt-5.4-mini 0.48 0.28 0.18 0.12 0.10 cliff
Sonnet 4.6 (/compact) 0.53 0.21 0.16 0.13 0.10 cliff
Vanilla LLM compactors lose roughly half their remaining safety rules every round; TypeCompact stays flat by construction.
git clone https://github.com/searchsim-org/knowledge-triage.git
cd knowledge-triage
pip install -e .[all]
cp .env.example .env # add your OPENAI_API_KEYReproduce a headline result in one command:
# The Compaction Cliff (Table 6)
python -m experiments.multiturn_stability --n 20 --rounds 5 --ratio 0.50
# TypeCompact vs vanilla LLM compactors at 3 ratios (Table 7)
python -m experiments.compression_curves --n 20 --ratios 0.50,0.25,0.10
# SafetyMargin classifier (Table 13)
python -m experiments.safety_margin
# TypeRetrieve shootout (Table 9)
python -m experiments.typeretrieve_vs_vanillaEach script writes a JSON to results/. The committed JSONs there are exactly the numbers reported in the paper.
from knowledge_triage import (
KnowledgeBase, Item, KnowledgeType,
type_compact, type_decompose, type_retrieve,
classify_batch_cached,
)
# 1. Build a typed knowledge base
items = [Item(text=line, type=KnowledgeType.BELIEF, topic="root")
for line in open("CLAUDE.md").read().splitlines() if line.strip()]
types = classify_batch_cached([i.text for i in items], use_llm=True)
for it, t in zip(items, types):
it.type = t
kb = KnowledgeBase(items=items)
# 2. Compact to a token budget. Constraints survive by construction.
compact_kb, status = type_compact(kb, budget=2000)
assert status == "ok", f"Unsafe: {status}"
# 3. Decompose into sub-bases that each fit a per-partition budget.
sub_kbs = type_decompose(kb, budget_per_partition=1000)
# 4. Retrieve top-k items in scope for a query.
top_k = type_retrieve(kb, query="how do I deploy?", k=10)| Paper table / section | One-line reproduction |
|---|---|
| Table 6 (Compaction Cliff multi-round stability) | python -m experiments.multiturn_stability |
| Table 7 (compression curves, four compactors × three ratios) | python -m experiments.compression_curves |
| Table 7 (LLMLingua-2 row only) | python -m experiments.llmlingua_curves |
| Table 8 (vanilla LLM shootout at fixed 50%) | python -m experiments.vanilla_vs_typecompact |
| Table 9 (TypeRetrieve vs vanilla LLM retrievers) | python -m experiments.typeretrieve_vs_vanilla |
| Table 12 (single-LLM classifier ablation) | python -m experiments.llm_classifiers |
| Table 13 (adversarial phrasing stress test) | python -m experiments.adversarial_classifier |
| Table 13 (SafetyMargin classifier row) | python -m experiments.safety_margin |
Wall time per experiment ranges from 30 seconds (LLMLingua-2, CPU-only) to 25 minutes (the four-model compression-curves shootout with Sonnet through the Claude Code SDK).
The empirical sections reference four corpora; see data/README.md for download instructions and the datasheet for the contributed corpus. AgentArtifactCorpus (AAC) is contributed by this work: 396,934 artefacts from 54,628 public repositories under permissive licences, 2.4 GB uncompressed and 678 MB as a gzip-compressed tarball. A stratified, scrubbed 327-artefact sample is included at data/sample/. The full corpus is distributed via a gated arrangement on the Hugging Face Hub with a Data Use Agreement, mirrored to Zenodo for DOI-based citation. Opt-out requests are handled through the data-removal issue template and the opt-out registry.
| Component | Path | Role |
|---|---|---|
| Typed knowledge model | knowledge_triage/kb.py, classifier.py | KnowledgeBase, Item, five operational types; §3.1 + Definition 1 |
| Three operators | knowledge_triage/operators.py | TypeCompact, TypeDecompose, TypeRetrieve; §3.2 + Theorems 2–4 |
| Selective classifier | knowledge_triage/classifier_cascade.py, cascade_singleton.py | regex → encoder → LLM → abstain, disk-cached; §3.3 |
| SafetyMargin classifier | knowledge_triage/safety_margin.py | counterfactual typing by safety rather than grammatical form; §3.4 |
| Preservation scoring | knowledge_triage/preservation.py | per-type d_C / d_P / … metrics; §4 |
| Experiments | experiments/ | one script per paper table, writing frozen JSON to results/ |
Configuration and project layout
Create a .env in the repo root:
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-... # optional, only for Sonnet/Opus comparisonsThe selective classifier caches LLM-adjudicated labels on disk (~50k entries shipped) so re-runs are free after the first pass.
knowledge_triage/ importable library
classifier.py KnowledgeType + regex baseline
classifier_cascade.py selective classifier: regex -> encoder -> LLM -> abstain
cascade_singleton.py disk-cached batch classifier (50k+ entries)
safety_margin.py SafetyMargin classifier (counterfactual typing)
preservation.py per-type d_C / d_P / ... scoring (paper metrics)
operators.py TypeCompact, TypeDecompose, TypeRetrieve
kb.py KnowledgeBase, Item dataclasses
clients.py OpenAI / Anthropic / Claude-Code-SDK client factory
secrets.py, spend.py .env-based key loading + cumulative spend ledger
learned_classifier.py optional: LLMLingua-2-style distilled classifier
experiments/ paper experiments (one script per table)
data/ download instructions + scrubbed sample + datasheet
results/ frozen JSON outputs (the numbers in the paper)
We thank the maintainers of BEIR, τ-bench, LongMemEval, and LLMLingua for the corpora and baselines used in this work.
The code in this repository is released under the Apache License 2.0. The paper is published open access by ACM under CC BY 4.0. AgentArtifactCorpus is distributed under the terms described in its datasheet.
