Repository navigation
feat(evals): add a Harbor example for evaluating versioned skills - #4164
Open
ngoncharenko wants to merge 18 commits into
Open
ngoncharenko wants to merge 18 commits into
ngoncharenko wants to merge 18 commits into
Conversation
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
a2bondar
force-pushed
the
harbor-evals-skill-example/ngoncharenko
branch
from
November 19, 2026 17:23
b36107d to
b69d293
Compare
Signed-off-by: abondarenko <abondarenko@nvidia.com>
…/abondarenko feat(evals): score Harbor skill evidence in the example
ngoncharenko
marked this pull request as ready for review
November 19, 2026 18:01
Signed-off-by: abondarenko <abondarenko@nvidia.com>
This was referenced Nov 19, 2026
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
feat(evals): persist explicit baseline links for taskset runs
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Contributor
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
feat(studio): visualize saved skill evaluation comparisons
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a Harbor Taskset example that measures how the published NeMo Evals skill changes Codex behavior on a real offline Helix evaluation. The pair shares a pinned task and effective images, with managed skill injection from #4143 as the treatment. The second example runs a live inner agent Taskset and checks a skill-only alias suffix on the persisted job.
Changes
What
Why
UX Before/After
Review Focus
Type of Change
Quality Gates
Verification
Signed-off-by:traileruv run pre-commit run -apasses, or any blocked checks are identified belowTargeted validation:
uv run --frozen pre-commit run -awith the existing environment and Flox host access.uv run --frozen --no-sync pytest -qon the four example test files, the Harbor scoring and CLI tests, the sandbox and trial credential tests, and the SDK skill metrics. 446 passed and 13 skipped.harbor run --path plugins/nemo-evals/examples/harbor_evals_skill/dataset --agent oracle --env dockerproduced Harbor reward1.0and inner mean0.5. A forced startup failure retainedstartup.log, and a local inference fixture exercised the pinned Codex binary and rollout decoder.ngoncharenko-devuses published runtime images from the PR base42b158350. The checkout supplied the PR's Taskset, launcher, and scorer. The API kept its previous UID and GID of 0 for compatibility with the existing Files volume.run-64ce52eb29322fb2scored1.0, verified an inner mean of0.5across two rows, and collected evidence in the separate verifier. The inner service was ready in 11 seconds.run-182d9ccab047d5f4and treatmentrun-da5d4e1c51c18430both scored1.0, using the same Taskset and effective image digests. Both logs confirm preinstalled Codex and setup egress closure. Catalog discovery and a successful document read confirmed activation in the treatment. Resubmitting the saved baseline request returned the original run ID.nemo-evals.agent-eva-plxrtpm4andnemo-evals.agent-eva-hzbrvjzgeach completed all four stored metric evaluations with zero errors.Dev-blue custom metrics
One saved trial per arm. Both arms earned the same Harbor reward and passed all three judge assertions.
activated,as_expected)expected_behavior,all_assertions_met)The baseline supplied
skills: [], and Codex's launch catalog excluded the pinned Evals skill. The scorer left baseline activation blank (0 of 1 trials scored). A score of0.0would represent a confirmed observation of non-activation. In the treatment, Codex discovered and read the pinnedSKILL.md(1 of 1 trials scored).The treatment used 393 more uncached prompt plus completion tokens (+1.4%) and 35,192 fewer total tokens (−13.0%).
The pinned skill archive had zero structural defects, zero broken references across 20 checked, and an estimated 2,851 tokens across 247 lines, within the context budget.
Dev-blue Supercool Taskset example
reward=1,inner_eval_passed=1, andsuffix_applied=1.ngoncharenko-mvpon dev-blue runs Helm revision 18 with the published42b158350runtime. The release now points to the existing workspace Builder credentials and the OpenSandbox server was restarted to clear a stale sandbox status. Oraclerun-c161818e855ef62fpassed on the two pinned images.run-c80bba8417618ddaand treatmentrun-d2cf7d1bc90180fbcompleted with the same pinned Taskset, agent image, and separate verifier image. Each inner Helix Taskset job called the loopback agent and scored exact match1.0. The server persistedagent-smokein the baseline andagent-smoke-supercool-7cf95d1e203ab864in the treatment.nemo-evals.agent-eva-bybgt422andnemo-evals.agent-eva-hhtsek1wreplayed all three metrics with zero errors. The treatment catalog and a successful read of the digest-verified skill confirmed activation.harbor_reward.inner_eval_passedharbor_reward.suffix_appliedharbor_reward.rewardskill_activation.activatedview.verified_skill_use