Skip to content

feat(evals): add a Harbor example for evaluating versioned skills - #4164

Open
ngoncharenko wants to merge 18 commits into
skills-evaluation-hackathonfrom
harbor-evals-skill-example/ngoncharenko
Open

ngoncharenko wants to merge 18 commits into
skills-evaluation-hackathonfrom
harbor-evals-skill-example/ngoncharenko

Conversation

@ngoncharenko

@ngoncharenko ngoncharenko commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add a Harbor Taskset example that measures how the published NeMo Evals skill changes Codex behavior on a real offline Helix evaluation. The pair shares a pinned task and effective images, with managed skill injection from #4143 as the treatment. The second example runs a live inner agent Taskset and checks a skill-only alias suffix on the persisted job.

Changes

What

  • Add a two-row exact-match task and Oracle solution using a digest-pinned Helix task image, configurable baked-in skill removal, and a separate verifier.
  • Make the shared Codex launcher accept pinned Taskset and skill refs, resume submissions with saved idempotency keys, compare the effective images, and retain both trial bundles.
  • Read Harbor scores from the run result record's scores field and trial counts from its summary, matching the response verified on dev-blue.
  • Add a scorer for the stored Harbor replay metrics and views, skill archive metrics, and native Codex discovery, activation, and token evidence.
  • Add a single-task Supercool agent-evaluation example with a deterministic loopback agent, separate verifier, named Harbor rewards, and a verified skill-use view.
  • Recognize a verified leading skill read in a Codex shell command list by matching its output to the pinned document.
  • Allow helpers in the Supercool verifier and make its launcher fail when either trial is unscored.

Why

  • Exercise the shipped Evals skill against a real local Helix service and distinguish a verified task outcome from evidence of skill discovery or activation.

UX Before/After

  • The Supercool example demonstrated skill injection with a synthetic naming rule. The new example walks through publishing a reusable Taskset, running the Oracle and Codex pair, then replaying the saved trials with custom metrics.

Review Focus

  • Check the collector and separate verifier's handling of invalid submissions and infrastructure errors, the launcher's pinning and recovery, and the scorer's treatment of absent Codex evidence.
  • Review team @NVIDIA-NeMo/helix-evals-team.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with documentation updates
  • Documentation only
  • Contributor tooling or automation
  • CI, build, or test infrastructure

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Documentation updated for user-visible behavior
  • Documentation not applicable — justification:

Verification

  • Pull request title follows the repository's Conventional Commit format
  • Every commit includes an appropriate Signed-off-by: trailer
  • uv run pre-commit run -a passes, or any blocked checks are identified below
  • Targeted tests pass, or tests are marked not applicable above
  • No secrets, API keys, or credentials are included

Targeted validation:

  • uv run --frozen pre-commit run -a with the existing environment and Flox host access.
  • uv run --frozen --no-sync pytest -q on the four example test files, the Harbor scoring and CLI tests, the sandbox and trial credential tests, and the SDK skill metrics. 446 passed and 13 skipped.
  • harbor run --path plugins/nemo-evals/examples/harbor_evals_skill/dataset --agent oracle --env docker produced Harbor reward 1.0 and inner mean 0.5. A forced startup failure retained startup.log, and a local inference fixture exercised the pinned Codex binary and rollout decoder.
  • The broader evaluator unit run reported three auth-service readiness timeouts and one metric-selection documentation mismatch in unchanged sources.
  • The launcher, runs CLI, and Taskset CLI suites passed 255 tests after the result parser fix. All four example test files passed 56 tests after the activation fix. Repository pre-commit checks passed on the final patch.
  • Dev-blue Helm revision 10 in ngoncharenko-dev uses published runtime images from the PR base 42b158350. The checkout supplied the PR's Taskset, launcher, and scorer. The API kept its previous UID and GID of 0 for compatibility with the existing Files volume.
  • Builder produced distinct digest-pinned agent and verifier images. Upgraded OpenSandbox Oracle run-64ce52eb29322fb2 scored 1.0, verified an inner mean of 0.5 across two rows, and collected evidence in the separate verifier. The inner service was ready in 11 seconds.
  • Codex baseline run-182d9ccab047d5f4 and treatment run-da5d4e1c51c18430 both scored 1.0, using the same Taskset and effective image digests. Both logs confirm preinstalled Codex and setup egress closure. Catalog discovery and a successful document read confirmed activation in the treatment. Resubmitting the saved baseline request returned the original run ID.
  • Scoring jobs nemo-evals.agent-eva-plxrtpm4 and nemo-evals.agent-eva-hzbrvjzg each completed all four stored metric evaluations with zero errors.
  • Dev-blue retained its existing auth-off configuration. The workspace-editor inference key and scoped-key isolation smoke still require an auth-enabled environment. Both scored bundles recorded the probe disabled.

Dev-blue custom metrics

One saved trial per arm. Both arms earned the same Harbor reward and passed all three judge assertions.

Metric Baseline With skill
Harbor reward 1.0 1.0
Skill activation (activated, as_expected) Unmeasured (skill absent) 1.0 each
Judge assertions 3/3 met 3/3 met
Judge views (expected_behavior, all_assertions_met) 1.0 each 1.0 each
Uncached prompt plus completion tokens 28,023 28,416
Token efficiency score 0.8771 0.8756
Total tokens including cached input 271,586 236,394

The baseline supplied skills: [], and Codex's launch catalog excluded the pinned Evals skill. The scorer left baseline activation blank (0 of 1 trials scored). A score of 0.0 would represent a confirmed observation of non-activation. In the treatment, Codex discovered and read the pinned SKILL.md (1 of 1 trials scored).

The treatment used 393 more uncached prompt plus completion tokens (+1.4%) and 35,192 fewer total tokens (−13.0%).

The pinned skill archive had zero structural defects, zero broken references across 20 checked, and an estimated 2,851 tokens across 247 lines, within the context budget.

Dev-blue Supercool Taskset example

  • The 335 focused example, Harbor scoring, publication, and runs CLI tests passed. Repository pre-commit passed. The Docker Oracle earned reward=1, inner_eval_passed=1, and suffix_applied=1.
  • ngoncharenko-mvp on dev-blue runs Helm revision 18 with the published 42b158350 runtime. The release now points to the existing workspace Builder credentials and the OpenSandbox server was restarted to clear a stale sandbox status. Oracle run-c161818e855ef62f passed on the two pinned images.
  • GPT-6 Luna baseline run-c80bba8417618dda and treatment run-d2cf7d1bc90180fb completed with the same pinned Taskset, agent image, and separate verifier image. Each inner Helix Taskset job called the loopback agent and scored exact match 1.0. The server persisted agent-smoke in the baseline and agent-smoke-supercool-7cf95d1e203ab864 in the treatment.
  • Scoring jobs nemo-evals.agent-eva-bybgt422 and nemo-evals.agent-eva-hhtsek1w replayed all three metrics with zero errors. The treatment catalog and a successful read of the digest-verified skill confirmed activation.
  • The namespace uses auth-off for this demonstration. Archive read isolation remains unverified.
Score Baseline With skill
harbor_reward.inner_eval_passed 1 1
harbor_reward.suffix_applied 0 1
harbor_reward.reward 0 1
skill_activation.activated Unmeasured 1
view.verified_skill_use Unmeasured 1

@ngoncharenko ngoncharenko self-assigned this Nov 19, 2026
@github-actions github-actions Bot added the feat label Nov 19, 2026
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
@a2bondar
a2bondar force-pushed the harbor-evals-skill-example/ngoncharenko branch from b36107d to b69d293 Compare November 19, 2026 17:23
Signed-off-by: abondarenko <abondarenko@nvidia.com>
…/abondarenko

feat(evals): score Harbor skill evidence in the example
@ngoncharenko
ngoncharenko marked this pull request as ready for review November 19, 2026 18:01
@ngoncharenko
ngoncharenko requested review from a team as code owners November 19, 2026 18:01
Signed-off-by: abondarenko <abondarenko@nvidia.com>
ngoncharenko and others added 5 commits November 19, 2026 11:35
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
feat(evals): persist explicit baseline links for taskset runs
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

a2bondar and others added 7 commits November 19, 2026 11:48
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
feat(studio): visualize saved skill evaluation comparisons
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants