KVM-isolated execution environments for LLM agents, at container-class single-node density.
Terrarium is a scheduling and control layer for agent execution environments. It pairs hardware-level VM isolation (Cloud Hypervisor) with a composable layered filesystem (EROFS + OverlayFS + virtiofs) and a warm pool, so untrusted agent code runs in real VMs (cold boot ~40 ms, warm-pool claim ~9 ms; the guest agent comes online shortly after the VM) that share read-only environment layers across the host.
Tenant-first model: VMs are tenant isolation boundaries. A Sandbox is a session inside a VM, not a VM itself. Multiple sandboxes in the same tenant share one VM with isolated workdirs. This gives you both strong security isolation (between tenants) and high density (within a tenant).
Status. Self-hosted, single-node research/engineering prototype. No multi-host orchestration, no managed service, no third-party verification yet — the numbers in this README are our own measurements on one host. It targets teams that need real per-tenant VM isolation with environment reuse (RL training, isolated CI, sandboxed agent execution). See docs/strategy.md for the scope and docs/performance.md for what is measured.
| Container | MicroVM | Terrarium | |
|---|---|---|---|
| Isolation | Shared kernel | KVM | KVM + sandbox |
| Density | High | Low | High (shared page-cache layers) |
| Provisioning | Fast | ~1s | Warm pool (pre-booted VMs) |
| Environments | OCI image | Disk image | Composable named layers |
| Backends | — | — | CH microVMs + in-guest confine (default) / sandlock (alternative) |
Real measurements (same host, 2026-08-15; full methodology and raw data in docs/performance.md):
| Agent-relevant metric | Terrarium | docker exec | gVisor runsc do |
|---|---|---|---|
| One tool call (p50) | ~3 ms | ~89 ms | ~293 ms |
| fileio workload (1500 files) | 15.5 ms | 136.3 ms | 619.9 ms |
| subproc workload (150 forks) | 122.2 ms | 172.2 ms | 1094.7 ms |
| mixed workload (write→build→read) | 37.5 ms | 117.1 ms | 564.7 ms |
Why it matters for agents: Terrarium VMs are long-lived and the agent
reuses a single vsock channel, so every tool call is one ~3 ms round trip.
docker exec pays a ~90 ms CLI/runc round trip per call, and gVisor's
one-shot runsc do pays ~300 ms of sandbox startup per call — over
thousands of calls per episode, that is the difference between training
throughput and timeouts.
Governance (the L2 in-guest sandbox, terra-confine) is zero-cost on
real workloads (0.87–1.04× across file/process/cpu/mixed loads): Landlock
is a static kernel verdict, seccomp only filters low-frequency network
syscalls, and cgroup is passive accounting. VM isolation is workload
dependent — ~1.3–2× on pure CPU, at parity on I/O and process-heavy agent
work. The host-side data planes (Cloud Hypervisor, virtiofsd) run under a
dedicated unprivileged user with no measurable cost on the exec path.
Density: 100 long-lived instances on one host — Terrarium restores snapshots at 59.2 instances/s (docker 3.2/s, gVisor empty shells 35.3/s), costs 61.6 MB host memory per real VM (vs gVisor 85.8 MB), and sustains 1787 execs/s aggregate (vs docker 165/s). Cold boot of a full VM is ~9/s — the snapshot restore path is the RL/density fast create; it beats gVisor's empty shells while provisioning a complete environment (layers + guest agent + networking). Shared read-only layer page cache keeps per-VM cost flat as tenants grow (~52 MB Pss at 12 VMs; a 99 MB layer adds only ~2 MB/VM). Warm-pool claim ~9 ms; net restores at 539 VMs/s. Full detail in docs/performance.md and docs/benchmarks.md.
┌─ Host ──────────────────────────────────────────────────────┐
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Terrarium Engine │ │
│ │ daemon · CLI · Python SDK · MCP Server │ │
│ └──────────────────────┬───────────────────────────────┘ │
│ ┌──────────────┴──────────────┐ │
│ ▼ ▼ │
│ ┌───────────────┐ ┌────────────────┐ │
│ │ CH Adapter │ │ SandboxAdapter │ │
│ │ (VmAdapter) │ │ Sandlock │ │
│ └───────┬───────┘ └───────┬────────┘ │
│ ▼ ▼ │
│ ┌─────────────────────────────────────────────┐ │
│ │ Cloud Hypervisor VM (per tenant) │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ │ Sandbox │ │ Sandbox │ │ Sandbox │ │ │
│ │ │ /workdir │ │ /workdir │ │ /workdir │ │ │
│ │ │ session │ │ session │ │ session │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ │ │
│ │ guest-proxy (vsock relay, per VM) │ │
│ │ virtiofs rootfs (shared layers + per-sb wd)│ │
│ └─────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
The engine is decoupled from backends via two trait families:
VmAdapter (Cloud Hypervisor) and SandboxAdapter (default:
terra-confine; sandlock kept as an alternative backend).
terrarium/
├── README.md / README_zh.md
├── LICENSE / NOTICE / THIRD-PARTY
├── crates/
│ ├── engine/ # Engine daemon: PyO3-backed lib with 7 command submodules
│ ├── adapter/
│ │ ├── traits/ # VmAdapter / SandboxAdapter traits, VmSpec, FsSpec, error types
│ │ ├── cloud-hypervisor/ # CH adapter (FS/VM decoupled), virtiofs, hotplug, network, landlock
│ │ ├── confine/ # Default SandboxAdapter (terra-confine backend)
│ │ └── sandlock/ # Alternative SandboxAdapter (sandlock backend)
│ ├── guest/
│ │ ├── proxy/ # In-guest command relay (vsock), backend argv translation
│ │ └── confine/ # In-guest limiter (terra-confine: Landlock/seccomp/cgroup)
│ ├── fs/ # Independent filesystem crate: EROFS, cpio, layer build/list/remove (PyO3 bindings)
│ ├── protocol/ # Shared Command / Response types (single source of truth)
│ ├── network/ # Tap / NAT / dnsmasq DHCP, tc QoS
│ └── mcp/ # MCP server (stdio JSON-RPC, 28 user-facing tools)
├── sdk/python/ # Python SDK (terra package: Sandbox, Pool, Template, client, daemon, assets, images)
├── images/ # Guest kernel / rootfs / initramfs build scripts and examples
└── docs/ # Index (docs/README.md), SDK/MCP/protocol, design,
# security, performance+benchmarks, paper, CI
Install from the repository root:
pip install -e .Sandbox API — the recommended high-level entry point (auto-starts daemon, context-manager cleanup):
from terra.sandbox import Sandbox
# Sandbox = session inside a tenant VM (tenant-first model)
with Sandbox(tenant="my-org", template="alpine", network=True) as sb:
result = sb.exec("echo hello")
print(result.stdout) # "hello\n"
sb.files.write("/workdir/hello.txt", "Hello, Terrarium!")
print(sb.files.read("/workdir/hello.txt"))
print(sb.id) # "sb-a3f2b1c4" (engine-allocated id)
print(sb.vm) # "tenant-my-org"
# Multiple sandboxes share one tenant VM
sb1 = Sandbox(tenant="research-team", template="alpine")
sb2 = Sandbox(tenant="research-team") # reuses same VM, new workdir
sb3 = Sandbox(tenant="research-team")
sb1.kill() # kills this session (running execs + workdir) — VM survives for siblings
Sandbox.destroy_tenant("research-team") # destroys the VM and all sessionsWarm pool — pre-booted shared-tenant VMs, layers hot-plugged on claim:
from terra.pool import Pool
pool = Pool(template="alpine", size=3) # 3 pre-booted VMs
sb1 = pool.acquire() # Sandbox in a shared pool VM
sb2 = pool.acquire() # same VM, different workdir
print(sb1.exec(["uname", "-r"]).stdout)
pool.release(sb1) # back to idle
pool.release(sb2)CLI — three steps to a running sandbox:
terra setup alpine # one-time: kernel + rootfs + initramfs + base layer (with confine) + template
terra daemon start # engine daemon (self-elevates via sudo; --no-root for rootless)
terra sandbox create --template alpine --net # high-level sandbox (VM = tenant)
terra sandbox exec sb-xxxxxxxx -- echo hi # sandboxed (confine) by default
terra sandbox kill sb-xxxxxxxx # kill the session (VM survives)
terra pool create -n mypool --size 3 # warm pool
terra pool claim --template alpine # claim a ready sandbox
terra daemon status # engine state at a glanceterra setup ubuntu does the same for ubuntu. Tool layers (python3 and
friends) are built on a distro template:
terra setup also creates the dedicated terra-vmm system user: with the
daemon running as root, Cloud Hypervisor and virtiofsd drop to that
unprivileged user (tap fds are passed by the daemon; guest-visible root
ownership is preserved via virtiofsd uid translation), so a guest escape
into the host-side data planes no longer immediately means host root.
Skip it (legacy root mode) if the machine has no sudo during setup.
terra tool create -n python312 --template alpine --script images/examples/python312.sh
terra tool ls
terra tool remove -n python312Tool layers are also the RL environment-baseline mechanism: bake the
environment's READY state into a layer (images/examples/rl-env.sh), then
Batch.reset_in_place() clears only the episode upper — the ready state
survives every reset. The minimal training loop is
sdk/python/examples/rl_episode_loop.py (inject input → run the layer
task → collect → reset_in_place); regression check in
sdk/python/tests/manual_envlayer.sh.
Agent execution — the same layer toolchain serves the agent use case:
repo + toolchain + tests baked into a layer, N agent environments restored
from one snapshot, each agent edits its workspace, and the in-place reset
returns every environment to the layer baseline. Proven end-to-end on a
real SWE-bench instance (pallets__flask-4160): bug reproduces at
baseline in 4 parallel envs, the fix turns the suite green (33 passed),
reset restores the broken baseline (sdk/python/tests/manual_swebench.sh).
The batch run extends this to 5 flask instances across versions
2.0/2.1/2.3 — all pass (sdk/python/tests/manual_swebench_batch.py,
raw results in docs/benchmark-results-2026-08-05-swebench-batch.json).
Agent CI — isolated verification of agent-written code, dogfooded on
this repo: sdk/python/examples/ci_verify.py restores a pristine
snapshot, copies the repo into the per-VM workspace, applies the agent's
patch and runs the test suite. The ci-terra layer bakes repo + test
toolchain (baseline suite verified at build time). Demo: good patch
passes (42 passed), broken patch rejected (1 failed), ~1.5s per check.
MCP — point your agent at the stdio server:
{"mcpServers": {"terrarium": {"command": "terra-mcp", "env": {"TERRA_SOCKET": "/tmp/terra.sock"}}}}- High-level Sandbox API —
terra.Sandbox/terra.AsyncSandboxwith tenant-first model: VMs are tenant isolation boundaries, sandboxes are sessions within a VM. Sandboxes are engine-level entities — the engine keeps the registry (tenant → VM, sandbox → workdir), so every client shares one view. Multiple sandboxes in the same tenant share one VM with isolated workdirs. Automatic daemon start, context-manager cleanup, file operations (read/write/upload/download/list), metrics, and online resize. - Two-layer isolation — between tenants, KVM microVMs; within a tenant VM, every
Sandbox.execis confined by the defaultterra-confinebackend (Landlock fs + seccomp network supervision + cgroup; sandlock remains an alternative), baked into the layers. Default policy: read-only system dirs, read-write only the session workdir and/tmp, sibling sessions' workdirs unreachable; egress denied by default. The policy is user-controllable via a capability-basedSandboxPolicy(default deny, explicit grant): file grants (FileRead/ReadWritecapabilities), an egress allowlist (NetworkOutboundcapabilities), and memory/process/fd/cpu limits — set at sandbox create or overridden per exec (Sandbox(policy={...})). A given policy is self-contained (replaces the engine default; omit it to get the injected default, so sandboxed exec always carries a complete policy). Opt out per call withsandboxed=False/--no-sandbox. - Layered filesystem — read-only EROFS layers star-composed on the host (arbitrary combinations, shared page cache), exposed via virtiofs. Distro base layers come from a config-driven pipeline (alpine and ubuntu ship today; more are a 3-line config). Tool layers are built by configuring a real VM and packing the delta, so environments are runnable by construction.
- Warm pool — pre-booted idle VMs as shared tenant containers; acquiring returns a sandbox session within a pool VM. Two slot kinds: warm (agent initramfs, layers hot-plugged on claim) and ready (restored from a snapshot — filesystem mounted, agent running, ~9 ms claim). Manual
grow()/scale()adjust size; auto-scaling is deliberately not planned (fixed, operator-configured resource). - Named templates —
terra.template.Templatepersists kernel + base distro + tool layer compositions, written byterra setupor the SDK. - In-guest exec — blocking and background execution inside VMs through the guest agent, per-command timeouts, and structured
ExecResult. Background sessions are tracked at the protocol level (session_status,session_kill,session_list) and exposed by all three clients: the Python SDK (Sandbox.exec(background=True)returns an engine-trackedSessionhandle), the CLI (sandbox exec --detach,sandbox session status|kill|ls), and the MCP server (terra_exec_background,terra_session_status,terra_session_kill). - Networking — one-flag NAT networking (
--net) with DHCP; lifecycle managed viaterra net. - Dynamic resize — CPU and memory online adjustment without reboot.
- Zero-config Python SDK — managed directories, automatic binary and image resolution, daemon auto-start, programmable host configuration (
HostConfig).
- docs/README.md — full documentation index (usage, design, security, performance, paper, ops)
- docs/tutorial-real-agent.md — real-agent tutorial (MCP, LangGraph/Claude SDK)
- docs/protocol.md — engine wire protocol (commands, transports, semantics)
- docs/sdk.md — Python SDK and CLI reference
- docs/mcp.md — MCP server tools
- docs/performance.md — measured performance vs docker / gVisor
- docs/security-threat-model.md — L1/L2 threat model
- docs/design/decisions.md — design ADRs and decisions
- ✅ CH base, engine daemon, adapter layer, Python SDK
- ✅ virtiofs layered filesystem, warm pool, NAT networking, layer build-by-doing
- ✅ High-level Sandbox / Pool / Template API, exception hierarchy, async support
- ✅ Warm-pool-backed tenant sandboxes (claim on create, release on destroy); MCP session-scoped exec
- ✅ Audit observability (per-policy tracing events; structured deny signal)
- ✅ Performance & density measurement — exec path ~3 ms/call (vs docker
~89 ms, gVisor ~293 ms), governance ≈ 0 overhead, 100-instance density
sweep, snapshot-restore create at 59.2/s, shared-layer memory curve
(see
docs/performance.md,docs/benchmarks.md) - ✅ P1: fast sandbox reset (snapshot/restore, verified ~200 ms on real KVM,
deterministic rollback) + batch lifecycle orchestration
(
terra.batch.Batch— parallel restore/recycle/collect + density report; VM lifecycle runs lock-free: 8 concurrent restores in 232 ms) - ✅ P2: security verification loop — real-KVM escape/deny suite (31
adversarial tests, 4 real defects fixed), default-deny networking, L2
tenant isolation (ebtables), audit persistence, resource limits; see
docs/security-verification.mdanddocs/security-adversarial.md - 🔲 P3: multi-host orchestration; policy/quota management (tenant quotas, task policy templates)
Strategy and scenario focus: docs/strategy.md.
Deliberately not planned: pool auto-scaling. The warm pool is a fixed,
operator-configured resource; manual grow() / scale() already cover
capacity changes, and automatic scaling would add threshold policy and
metrics-driven complexity without a proven demand.
Apache-2.0. Built on Cloud Hypervisor and Linux kernel features. See THIRD-PARTY for acknowledgments.



