Declarative control of LLM pipelines via beliefs and policies.
See details in our VLDB 2026 demonstration paper. If you use our work, please cite:
Credo: Declarative Control of LLM Pipelines via Beliefs and Policies.
Duo Lu, Andrew Crotty, and Uğur Çetintemel.
PVLDB 19 (VLDB 2026 Demonstration Track)
Requires Python >= 3.10.
git clone https://github.com/brown-db/credo.git
cd credo
pip install -e . # framework only
pip install -e ".[experiment]" # + the FinanceBench experimentModel credentials are read from the environment or from a .env file at the
repository root:
OPENAI_API_KEY=... # OpenAI models and the LLM judge
TOGETHERAI_API_KEY=... # Together AI models
OLLAMA_BASE=... # local Ollama endpointModel keys (gpt-4.1, together-70b, gemma3-27b, …) are defined in
credo/config.py, which also holds the price table used to
compute the cost metric. Self-hosted entries point at local vLLM endpoints —
adjust the ports there to match your deployment.
A complete, runnable pipeline with stand-in operators — no API keys needed:
from credo import CredoEngine, Pipeline
def do_retrieve(params, context):
k = params['top_k']
context['evidence'] = 'thin evidence' if k < 10 else 'detailed evidence'
return {'output': context['evidence'], 'cost': 0.0001}
def do_generate(params, context):
context['model_answer'] = f'answer from {params["model"]}'
return {'output': context['model_answer'], 'cost': 0.01}
def do_verify(params, context):
context['verified'] = 'pass'
return {'output': 'pass', 'cost': 0.001}
# belief extractors return (value, cost, confidence)
def rate_complexity(input_data, context):
return (0.4 if 'margin' in input_data['question'] else 0.1), 0.0001, 1.0
def grade_relevance(input_data, context):
return ('incorrect' if 'thin' in context.get('evidence', '') else 'correct'), 0.002, 0.9
engine = CredoEngine(
extractors={'complexity': rate_complexity, 'relevance': grade_relevance},
executors={'retrieve': do_retrieve, 'generate': do_generate, 'verify': do_verify},
)
# beliefs: `relevance` is runtime-only — it cannot exist until `evidence` does
engine.register_belief('complexity', tier=1, extractor='rate_complexity',
est_cost=0.0001)
engine.register_belief('relevance', tier=2, extractor='grade_relevance',
est_cost=0.002, runtime_only=True, depends_on=['evidence'])
# policies: lower priority number wins; condition 'True' is the default rule
engine.register_policy(operator='retrieve', priority=10,
condition='complexity >= 0.25',
params={'top_k': 12}, required_beliefs=['complexity'])
engine.register_policy(operator='retrieve', priority=100, condition='True',
params={'top_k': 5}, required_beliefs=[])
engine.register_policy(operator='generate', priority=10,
condition='complexity >= 0.25',
params={'model': 'strong'}, required_beliefs=['complexity'])
engine.register_policy(operator='generate', priority=100, condition='True',
params={'model': 'weak'}, required_beliefs=[])
engine.register_policy(operator='verify', priority=100, condition='True',
params={'method': 'entailment'}, required_beliefs=[])
# corrective: a non-empty `invalidates` makes this a global rewrite
engine.register_policy(operator='retrieve', priority=5,
condition='relevance == "incorrect"',
params={'top_k': 15}, required_beliefs=['relevance'],
invalidates=['evidence', 'model_answer', 'verified'],
description='poor relevance -> re-retrieve')
pipeline = Pipeline().retrieve(source='docs').generate(task='answer').verify()
result = engine.run(pipeline, query='What was the total revenue?', verbose=True)
print(result['operators_executed']) # ['retrieve', 'retrieve', 'generate', 'verify']
print(result['corrections']) # 1The simple question scores low complexity, so the compiler picks top_k=5 and
the weak model. That yields thin evidence, relevance evaluates to
"incorrect" at runtime, and the corrective policy fires — invalidating the
evidence and re-retrieving before generation proceeds. Ask about a margin
instead and complexity clears the threshold: the plan compiles to top_k=12
and the strong model, retrieval succeeds, and no correction is needed.
Use engine.explain(pipeline, goal_id) to print a compiled plan without
executing it.
Conditions are evaluated by a restricted AST interpreter
(credo/core/conditions.py) — comparisons, boolean
operators, not, and literals only. No function calls, attribute access, or
side effects. Unknown belief names raise ConditionError, and a policy whose
condition raises is skipped rather than crashing the run.
complexity >= 0.25
relevance == "incorrect"
groundedness <= 2 and has_hallucinations == True
True # always-matches default rule
credo/
├── credo/ # the framework
│ ├── config.py # model registry + price table
│ ├── model.py # LLM client wrappers, token accounting
│ ├── core/
│ │ ├── pipeline.py # Pipeline builder (logical operators)
│ │ ├── operators.py # Retrieve / Generate / Verify
│ │ ├── conditions.py # restricted AST condition evaluator
│ │ ├── compiler.py # policy resolution → ExecutionPlan
│ │ ├── engine.py # reactive loop, corrective rewrites
│ │ ├── executor.py # step execution + logging
│ │ ├── context.py # DB-backed execution context
│ │ └── db.py # SQLite: beliefs, policies, plans, decisions
│ └── operators/ # physical implementations
│ ├── retrieval.py # dense / BM25 / TF-IDF / MMR / ensemble
│ ├── generation.py # DSPy strategies, model routing
│ └── verify.py # entailment / LLM judge / symbolic
├── experiment/ # FinanceBench experiment
│ ├── download_data.py # fetch FinanceBench from upstream
│ ├── beliefs.py # belief extractors
│ ├── policies.py # policy rules
│ ├── pipeline.py # engine wiring
│ ├── preprocess.py # PDF extraction + retriever
│ └── run_financebench.py # experiment CLI
└── data/ # datasets (downloaded, not committed)
Everything the engine does is persisted to SQLite — beliefs and their costs, compiled plans, the policy that decided each operator, and a per-step execution log. A run DB can be queried directly:
sqlite3 logs/credo/run.db \
"SELECT operator, policy_id, params, beliefs_used FROM decisions"MIT — see LICENSE.
Datasets are not covered by this license and are not redistributed here. Obtain them from their upstream providers and comply with their respective terms.