Skip to content

feat(evals): score Harbor skill evidence in the example - #4171

Merged
ngoncharenko merged 1 commit into
harbor-evals-skill-example/ngoncharenkofrom
codex/harbor-skill-evidence/abondarenko
Nov 19, 2026
Merged

ngoncharenko merged 1 commit into
harbor-evals-skill-example/ngoncharenkofrom
codex/harbor-skill-evidence/abondarenko

Conversation

@a2bondar

Copy link
Copy Markdown
Contributor

The Harbor example now scores package readiness from the verified skill snapshot and command outcomes from saved Codex trials. Replay carries outcome flags for matched root exec_command calls and marks the trace partial. Unknown outcomes are counted separately from successes and failures.

Run score_evidence_example.py to see synthetic trace, scan, and bundled-test evidence and their scores. The fixture runs offline and is separate from live scoring.

Stacked on #4164.

Validation includes 573 targeted SDK, Harbor runtime, and plugin tests, plus the offline fixture. The full local hook run stopped on unavailable documentation tooling. Live Harbor execution was not part of this check.

Signed-off-by: abondarenko <abondarenko@nvidia.com>
@github-actions github-actions Bot added the feat label Nov 19, 2026
@a2bondar a2bondar self-assigned this Nov 19, 2026
@a2bondar
a2bondar marked this pull request as ready for review November 19, 2026 17:32
@a2bondar
a2bondar requested review from a team as code owners November 19, 2026 17:32
@ngoncharenko
ngoncharenko merged commit 05b0cad into harbor-evals-skill-example/ngoncharenko Nov 19, 2026
4 checks passed
@ngoncharenko
ngoncharenko deleted the codex/harbor-skill-evidence/abondarenko branch November 19, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants