Skip to content

P3-B: PPO + AlphaZero trainer-step smoke tests - #11

Merged
NorbertRop merged 2 commits into
mainfrom
tests/p3-trainer-step
May 25, 2026
Merged

NorbertRop merged 2 commits into
mainfrom
tests/p3-trainer-step

Conversation

@NorbertRop

Copy link
Copy Markdown
Contributor

Summary

Adds three trainer-step smoke tests (`tests/test_trainer_step.py`):

Test What it pins
`test_ppo_train_step_changes_model_params` One PPO update moves params by >1e-6 — catches "gradients dead", "optimizer state stale".
`test_az_train_step_changes_model_params` Same bar for AlphaZeroTrainer, with an `mcts_policy` aux-target wired into the buffer.
`test_ppo_train_step_reduces_loss_on_repeated_apply` 5 consecutive SGD steps on the same batch produce a lower final loss than the first — catches sign-flipped gradients and wrong-direction optimizer steps.

Scope

These are smoke / direction tests, not convergence tests. They catch dead-trainer wiring without needing a tiny-problem RL setup. Actual reward-converges-to-X tests belong in P3-D (integration), which spins up a real env and runs a real training loop.

Test plan

  • `uv run pytest tests/` — 177/177 (was 174 before; this PR adds 3 new tests).

Roadmap

P3-B complete. Next: P3-C (rollout buffer + league + checkpoints + evaluator tests).

🤖 Generated with Claude Code

Three new tests:

- test_ppo_train_step_changes_model_params: one train_epochs call must
  move params by > 1e-6. Catches "gradients dead", "optimizer state
  stale", "params not written back".

- test_az_train_step_changes_model_params: same bar for the AlphaZero
  trainer, with an mcts_policy aux target wired into the buffer
  (normally the AlphaZeroTrainer's rollout transform injects this).

- test_ppo_train_step_reduces_loss_on_repeated_apply: 5 consecutive
  SGD steps on the same batch must produce a lower final loss than
  the first step. Catches sign-flipped gradients and optimizer
  direction bugs that aren't visible in a single-step delta check.

These are not convergence tests — the bar is just "the trainer
plumbing actually wires up". Real convergence belongs in the P3-D
integration tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The module docstring referenced "the previous PR's static_argname
recompile fix" and a sibling P3-D wave. Describe the current
invariant the tests guard instead — a step that doesn't move params
silently wastes compute — so the docstring stays meaningful after
the surrounding PR/wave context is gone.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@NorbertRop
NorbertRop merged commit f16b77c into main May 25, 2026
1 check passed
@NorbertRop
NorbertRop deleted the tests/p3-trainer-step branch May 26, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant