Skip to content

feat(studio): visualize saved skill evaluation comparisons - #4180

Merged
a2bondar merged 6 commits into
harbor-evals-skill-example/ngoncharenkofrom
skill-comparison-view/a2bondar
Nov 19, 2026
Merged

a2bondar merged 6 commits into
harbor-evals-skill-example/ngoncharenkofrom
skill-comparison-view/a2bondar

Conversation

@a2bondar

@a2bondar a2bondar commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Studio users can load saved Control and Variant executions or import skill-eval-summary.json from the Harbor Evals skill example and compare the baseline and with-skill results without starting another evaluation. The new Evaluations → Skill Comparisons page adds paired score bars, a detailed metric table, and a task/trial heatmap with evidence drill-down.

Based on harbor-evals-skill-example/ngoncharenko with the merged baseline linkage.

Changes

  • Preserve zero versus unmeasured results, categorical judge labels, metric coverage, incomplete scoring, and input-drift warnings. Withhold metric differences when the observations are not a complete controlled comparison.
  • Display live Harbor rewards, replay scores, and package checks separately. Retain pinned resources, image digests, scoring-job links, and local artifact paths for review.
  • Add token accounting that includes cached input once. Extend the example scorer's report with its existing trial measurements; older reports still load with unavailable counts left unmeasured.
  • Adapt SkillEvaluator's paired-result and evidence patterns and the supplied grouped-bar/table/heatmap references to Studio's design system and Helix report contract.
  • Load selected saved executions and preselect a linked baseline, including older runs outside the recent list. Older executions without a link can still be paired manually. Match the latest completed scoring replay by exact trial ID and load its retained per-trial bundle in memory. The first loader supports one task/trial per arm and one skill, validates pinned tasksets/images, bounds archive sizes, and leaves runtime discovery unconfirmed until rollout verification is available.
  • Show endpoint values and observed ranges on Variant mean bars. Keep the thin neutral Control bars and the green matrix gradient. Display scores and counts with up to two decimal places in Every Metric. Keep categorical labels and token counts in tables/evidence instead of matrix score columns.
  • Render the outputs of all 29 skill metric types when present, including scan findings and trial diagnostics. Keep count outputs out of the normalized score chart.
  • Keep task and arm labels visible while scrolling across the matrix. Package checks open on demand and can be searched by name, output, or finding.
  • Keep report import in the browser tab. No persistence migration, evaluation launch, external result publication, or dev-namespace deployment is included.

Type of Change

  • Code change with documentation updates

Quality Gates

  • Tests added or updated for changed behavior
  • Documentation updated for user-visible behavior

Verification

  • Pull request title follows the repository's Conventional Commit format
  • Every commit includes an appropriate Signed-off-by: trailer
  • uv run pre-commit run -a passes, or any blocked checks are identified below
  • Targeted tests pass, or tests are marked not applicable above
  • No secrets, API keys, or credentials are included

Targeted validation (from web/, using the repository-pinned Node 22.23.2 and pnpm 10.34.5):

  • pnpm --filter nemo-studio-ui test src/routes/evaluation/SkillComparisonRoute src/routes/WorkspaceLayout/WorkspaceSideNav.test.tsx: 43 passed.
  • pnpm --filter @nemo/common test src/components/EntityEmptyState/EntityEmptyState.test.tsx: 8 passed.
  • pnpm --filter nemo-studio-ui typecheck: passed.
  • ESLint and Prettier checks on changed frontend files: passed.
  • pnpm --filter nemo-studio-ui build:fastapi: passed.
  • Desktop browser validation of the production build: import and saved-run loading, bars, tables, heatmap, trial evidence modal, and scoring-job links; zero browser errors. Used the existing Evals skill example's retained baseline/treatment and scoring bundles. No new evaluation runs or deployment changes.
  • Full repository uv run --frozen pre-commit run -a is blocked: the checkout requires uv >=0.13.4, while the available executable is 0.10.10. The full Python suite was not run for the additive report-measurements export.

Desktop checks rendered all 29 skill metric types in a multi-trial report scored by the SDK against synthetic evidence. They also covered older imports, the last matrix column, pinned task labels, and package search and expansion. The linked-baseline picker is covered by a test using the generated API client contract.

@github-actions github-actions Bot added the feat label Nov 19, 2026
@a2bondar a2bondar self-assigned this Nov 19, 2026
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
@a2bondar
a2bondar force-pushed the skill-comparison-view/a2bondar branch from edac863 to c658cfd Compare November 19, 2026 19:42
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
Signed-off-by: abondarenko <abondarenko@nvidia.com>
@a2bondar
a2bondar marked this pull request as ready for review November 19, 2026 21:02
@a2bondar
a2bondar requested review from a team as code owners November 19, 2026 21:02
@a2bondar
a2bondar merged commit 0cfef85 into harbor-evals-skill-example/ngoncharenko Nov 19, 2026
6 checks passed
@a2bondar
a2bondar deleted the skill-comparison-view/a2bondar branch November 19, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant