Repository navigation
Compute LayerNorm statistics in at least float32 - #924
Open
DiadkoShmek wants to merge 1 commit into
Open
DiadkoShmek wants to merge 1 commit into
DiadkoShmek wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LayerNorm can return zero activations for ordinary finite float16 inputs because its variance overflows the input dtype. For example,
[[1000, -1000]]normalizes to[[0, 0]]instead of approximately[[1, -1]]. Both the default variance path anduse_fast_variance=Trueare affected; InstanceNorm inherits the same implementation.Compute low-precision statistics and normalization intermediates in float32. Keep parameter creation in the input dtype and return the dtype implied by the input and affine arguments. Float32/float64 inputs keep their existing path. Explicit low-precision handling preserves strict dtype-promotion behavior and avoids narrowing externally supplied higher-precision affine parameters.
The new regression matrix checks eager/JIT execution, f16/bf16/f32/f64, learned affine parameters and a separate float32-affine control. Outputs and input/parameter gradients are checked against an independent float64 formula. Exact base: 12 failures, 24 passing controls; fixed: all 36 pass. The selected LayerNorm/InstanceNorm and mixed-precision suite passes 99 tests, including those regressions.
Independent review additionally covers axes, nonunit affine values, strict promotion, InstanceNorm formats and higher-precision external affine parameters. Its 64 f32/f64 controls have bitwise-identical outputs and gradients on exact base and candidate. CPU only: no GPU/TPU or performance result is claimed. Ruff error/whitespace checks pass; configured pylint emits legacy configuration diagnostics, so no project pylint pass is claimed.
Related #911 addresses RMSNorm separately; this change only repairs LayerNorm and its InstanceNorm subclass.