Repository navigation
P3-B: PPO + AlphaZero trainer-step smoke tests - #11
Merged
Merged
Conversation
Three new tests: - test_ppo_train_step_changes_model_params: one train_epochs call must move params by > 1e-6. Catches "gradients dead", "optimizer state stale", "params not written back". - test_az_train_step_changes_model_params: same bar for the AlphaZero trainer, with an mcts_policy aux target wired into the buffer (normally the AlphaZeroTrainer's rollout transform injects this). - test_ppo_train_step_reduces_loss_on_repeated_apply: 5 consecutive SGD steps on the same batch must produce a lower final loss than the first step. Catches sign-flipped gradients and optimizer direction bugs that aren't visible in a single-step delta check. These are not convergence tests — the bar is just "the trainer plumbing actually wires up". Real convergence belongs in the P3-D integration tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks done
The module docstring referenced "the previous PR's static_argname recompile fix" and a sibling P3-D wave. Describe the current invariant the tests guard instead — a step that doesn't move params silently wastes compute — so the docstring stays meaningful after the surrounding PR/wave context is gone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three trainer-step smoke tests (`tests/test_trainer_step.py`):
Scope
These are smoke / direction tests, not convergence tests. They catch dead-trainer wiring without needing a tiny-problem RL setup. Actual reward-converges-to-X tests belong in P3-D (integration), which spins up a real env and runs a real training loop.
Test plan
Roadmap
P3-B complete. Next: P3-C (rollout buffer + league + checkpoints + evaluator tests).
🤖 Generated with Claude Code