Skip to content

About

Typed operators for safe context management in long-running AI agents — CIKM 2026

Topics

Resources

Stars

5 stars

Watchers

1 watching

Forks

Repository files navigation

The Compaction Cliff in Long-Running AI Agent Memory

Knowledge Triage: typed operators for safe AI agent context management

Classify each line of an agent's memory by type, then route each type through its own retention policy so safety rules survive compaction.

CIKM 2026 DOI License: Apache 2.0 Python 3.11+ HF Dataset Agent Skill

Knowledge Triage: an agent's knowledge base is classified by type, then compacted, decomposed, and retrieved under type-dependent retention policies.

Saber Zerhoudi, Jelena Mitrović, Michael Granitzer University of Passau, Germany  ·  Published at CIKM '26, Rome, Italy, November 7–11, 2026  ·  Read the paper

What is this? Long-running AI agents summarize their context when the token budget overflows, and a safety rule gets collapsed at the same rate as an episodic log even though only the rule needs its exact wording to stay enforceable. Knowledge Triage is a framework and reference implementation that classifies each item of an agent's knowledge base by type and applies three deterministic, type-aware operators (TypeCompact, TypeDecompose, TypeRetrieve) so constraints survive by construction.

Who is it for? Researchers studying agent memory and context management who want to reproduce the paper's results, and practitioners building long-running agents who need compaction that keeps safety rules intact. Practitioners who want it running in Claude Code today can install the companion agent skill: searchsim-org/knowledge-triage-skill.

Contents: Abstract · Highlights · Citation · Quick start · Library usage · Reproducing the paper · Data · Architecture · License


Abstract

A safety rule and an episodic log compete for the same tokens in an AI agent's context. When the budget overflows, both are summarized at the same rate; only the rule needs exact wording to remain enforceable. On 20 production agent configurations, Claude Code's /compact prompt on Sonnet 4.6 preserves 53% of safety rules after one compaction round and 10% after five. We name this the Compaction Cliff. We address it with Knowledge Triage, a framework that classifies each line of an agent's knowledge base by type and routes each type through its own retention policy. Three deterministic operators implement this triage across the three context-management operations: TypeCompact rewrites items in place under per-type fidelity, TypeDecompose partitions a topic too large to compact safely, replicating in-scope safety rules across partitions, and TypeRetrieve fetches items from external storage with in-scope rules pinned ahead of relevance. On five public corpora, TypeCompact preserves 2–4× more safety rules than the strongest single-shot LLM compactor at every ratio, with 96% recall over five rounds. TypeDecompose reaches 0% locality violations against 93% under uniform partitioning. TypeRetrieve reaches 100% recall@50 against 73% for the best single-shot LLM retriever. On three downstream behavioral benchmarks, we outperform the production Sonnet compactor on medical compliance (paired McNemar p < 10⁻⁸ on preservation, N = 200), the full-policy and hierarchical baselines on retail task pass rate (p < 0.01, N = 115), and the hierarchical compaction on the airline domain (p = 0.024). We release AgentArtifactCorpus (396,934 agent configurations from 54,628 public GitHub repositories), the classifier, and the reference implementation.

Keywords: AI agents, agent memory, context compaction, knowledge management, long-running agents

Highlights

  • The Compaction Cliff: on 20 production agent configurations, Claude Code's /compact (Sonnet 4.6) keeps 53% of safety rules after one round and 10% after five. Vanilla LLM compactors lose roughly half their remaining rules every round.
  • Type-aware compaction: TypeCompact preserves 2–4× more safety rules than the strongest single-shot LLM compactor at every ratio, holding 96% recall across five sequential rounds.
  • Structural guarantees: TypeDecompose reaches 0% locality violations (93% under uniform partitioning); TypeRetrieve reaches 100% recall@50 in scope (73% for the best single-shot LLM retriever).
  • Deterministic and cheap: the operators run in under 1 ms with 0 LLM tokens per call, against 5–420 s and 36k–57k tokens for LLM baselines.
  • AgentArtifactCorpus (AAC): 396,934 agent configuration artefacts from 54,628 public GitHub repositories under permissive licences, released with a datasheet and opt-out infrastructure.

Citation

If you use Knowledge Triage or AgentArtifactCorpus, please cite the paper.

@inproceedings{zerhoudi2026compaction,
  author    = {Zerhoudi, Saber and Mitrovi\'{c}, Jelena and Granitzer, Michael},
  title     = {The Compaction Cliff in Long-Running AI Agent Memory},
  year      = {2026},
  booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management},
  series    = {CIKM '26},
  location  = {Rome, Italy},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  isbn      = {979-8-4007-2539-5},
  doi       = {10.1145/3799682.3840567},
  url       = {https://doi.org/10.1145/3799682.3840567},
}

ACM Reference Format: Saber Zerhoudi, Jelena Mitrović, and Michael Granitzer. 2026. The Compaction Cliff in Long-Running AI Agent Memory. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM '26), November 07–11, 2026, Rome, Italy. ACM, New York, NY, USA. https://doi.org/10.1145/3799682.3840567

GitHub also shows a "Cite this repository" button generated from CITATION.cff.

At a glance

Production AI agents that run for hours manage their context through three operations: compaction, decomposition, and retrieval. The state of practice delegates all three to the same uniform policy, even though a safety rule cannot be paraphrased the way an episodic log can be collapsed. Knowledge Triage closes the gap with three type-aware operators that share a single classifier and verifier.

Operation State of practice This work Improvement
Compaction at 10% ratio 0.24 constraint recall 0.80 3.3×
Compaction over 5 sequential rounds 0.10 0.96 9.6×
Decomposition locality violations 93% of configs affected 0% structural
Retrieval recall@50 in scope 0.73 1.00 +27 points
Declarative-rule classification 0.62 recall 0.90 (SafetyMargin) +28 points
Per-call latency 5–420 s, 36k–57k LLM tokens <1 ms, 0 tokens deterministic

Numbers are the mean across 20 stratified configurations from AgentArtifactCorpus. State of practice = Claude Code's /compact (Sonnet 4.6) for compaction, dense top-50 cosine for retrieval, uniform token partitioning for decomposition, one-shot gpt-5.4-mini for grammatical-form classification.

The Compaction Cliff

Sequential compaction at 50% per round, measured against the original safety-rule set:

round                   1     2     3     4     5
TypeCompact           1.00  0.96  0.96  0.96  0.96    flat
gpt-5.4-mini          0.48  0.28  0.18  0.12  0.10    cliff
Sonnet 4.6 (/compact) 0.53  0.21  0.16  0.13  0.10    cliff

Vanilla LLM compactors lose roughly half their remaining safety rules every round; TypeCompact stays flat by construction.

Quick start

git clone https://github.com/searchsim-org/knowledge-triage.git
cd knowledge-triage
pip install -e .[all]
cp .env.example .env  # add your OPENAI_API_KEY

Reproduce a headline result in one command:

# The Compaction Cliff (Table 6)
python -m experiments.multiturn_stability --n 20 --rounds 5 --ratio 0.50

# TypeCompact vs vanilla LLM compactors at 3 ratios (Table 7)
python -m experiments.compression_curves --n 20 --ratios 0.50,0.25,0.10

# SafetyMargin classifier (Table 13)
python -m experiments.safety_margin

# TypeRetrieve shootout (Table 9)
python -m experiments.typeretrieve_vs_vanilla

Each script writes a JSON to results/. The committed JSONs there are exactly the numbers reported in the paper.

Library usage

from knowledge_triage import (
    KnowledgeBase, Item, KnowledgeType,
    type_compact, type_decompose, type_retrieve,
    classify_batch_cached,
)

# 1. Build a typed knowledge base
items = [Item(text=line, type=KnowledgeType.BELIEF, topic="root")
         for line in open("CLAUDE.md").read().splitlines() if line.strip()]
types = classify_batch_cached([i.text for i in items], use_llm=True)
for it, t in zip(items, types):
    it.type = t
kb = KnowledgeBase(items=items)

# 2. Compact to a token budget. Constraints survive by construction.
compact_kb, status = type_compact(kb, budget=2000)
assert status == "ok", f"Unsafe: {status}"

# 3. Decompose into sub-bases that each fit a per-partition budget.
sub_kbs = type_decompose(kb, budget_per_partition=1000)

# 4. Retrieve top-k items in scope for a query.
top_k = type_retrieve(kb, query="how do I deploy?", k=10)

Reproducing the paper

Paper table / section One-line reproduction
Table 6 (Compaction Cliff multi-round stability) python -m experiments.multiturn_stability
Table 7 (compression curves, four compactors × three ratios) python -m experiments.compression_curves
Table 7 (LLMLingua-2 row only) python -m experiments.llmlingua_curves
Table 8 (vanilla LLM shootout at fixed 50%) python -m experiments.vanilla_vs_typecompact
Table 9 (TypeRetrieve vs vanilla LLM retrievers) python -m experiments.typeretrieve_vs_vanilla
Table 12 (single-LLM classifier ablation) python -m experiments.llm_classifiers
Table 13 (adversarial phrasing stress test) python -m experiments.adversarial_classifier
Table 13 (SafetyMargin classifier row) python -m experiments.safety_margin

Wall time per experiment ranges from 30 seconds (LLMLingua-2, CPU-only) to 25 minutes (the four-model compression-curves shootout with Sonnet through the Claude Code SDK).

Data

The empirical sections reference four corpora; see data/README.md for download instructions and the datasheet for the contributed corpus. AgentArtifactCorpus (AAC) is contributed by this work: 396,934 artefacts from 54,628 public repositories under permissive licences, 2.4 GB uncompressed and 678 MB as a gzip-compressed tarball. A stratified, scrubbed 327-artefact sample is included at data/sample/. The full corpus is distributed via a gated arrangement on the Hugging Face Hub with a Data Use Agreement, mirrored to Zenodo for DOI-based citation. Opt-out requests are handled through the data-removal issue template and the opt-out registry.

Architecture

Component Path Role
Typed knowledge model knowledge_triage/kb.py, classifier.py KnowledgeBase, Item, five operational types; §3.1 + Definition 1
Three operators knowledge_triage/operators.py TypeCompact, TypeDecompose, TypeRetrieve; §3.2 + Theorems 2–4
Selective classifier knowledge_triage/classifier_cascade.py, cascade_singleton.py regex → encoder → LLM → abstain, disk-cached; §3.3
SafetyMargin classifier knowledge_triage/safety_margin.py counterfactual typing by safety rather than grammatical form; §3.4
Preservation scoring knowledge_triage/preservation.py per-type d_C / d_P / … metrics; §4
Experiments experiments/ one script per paper table, writing frozen JSON to results/
Configuration and project layout

Create a .env in the repo root:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...   # optional, only for Sonnet/Opus comparisons

The selective classifier caches LLM-adjudicated labels on disk (~50k entries shipped) so re-runs are free after the first pass.

knowledge_triage/                  importable library
  classifier.py                    KnowledgeType + regex baseline
  classifier_cascade.py            selective classifier: regex -> encoder -> LLM -> abstain
  cascade_singleton.py             disk-cached batch classifier (50k+ entries)
  safety_margin.py                 SafetyMargin classifier (counterfactual typing)
  preservation.py                  per-type d_C / d_P / ... scoring (paper metrics)
  operators.py                     TypeCompact, TypeDecompose, TypeRetrieve
  kb.py                            KnowledgeBase, Item dataclasses
  clients.py                       OpenAI / Anthropic / Claude-Code-SDK client factory
  secrets.py, spend.py             .env-based key loading + cumulative spend ledger
  learned_classifier.py            optional: LLMLingua-2-style distilled classifier

experiments/                       paper experiments (one script per table)
data/                              download instructions + scrubbed sample + datasheet
results/                           frozen JSON outputs (the numbers in the paper)

Acknowledgments

We thank the maintainers of BEIR, τ-bench, LongMemEval, and LLMLingua for the corpora and baselines used in this work.

License

The code in this repository is released under the Apache License 2.0. The paper is published open access by ACM under CC BY 4.0. AgentArtifactCorpus is distributed under the terms described in its datasheet.

About

Typed operators for safe context management in long-running AI agents — CIKM 2026

Topics

Resources

Stars

5 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages