Skip to content

Compute LayerNorm statistics in at least float32 - #924

Open
DiadkoShmek wants to merge 1 commit into
google-deepmind:mainfrom
DiadkoShmek:lezo3-layernorm-stats-precision
Open

DiadkoShmek wants to merge 1 commit into
google-deepmind:mainfrom
DiadkoShmek:lezo3-layernorm-stats-precision

Conversation

@DiadkoShmek

Copy link
Copy Markdown

LayerNorm can return zero activations for ordinary finite float16 inputs because its variance overflows the input dtype. For example, [[1000, -1000]] normalizes to [[0, 0]] instead of approximately [[1, -1]]. Both the default variance path and use_fast_variance=True are affected; InstanceNorm inherits the same implementation.

Compute low-precision statistics and normalization intermediates in float32. Keep parameter creation in the input dtype and return the dtype implied by the input and affine arguments. Float32/float64 inputs keep their existing path. Explicit low-precision handling preserves strict dtype-promotion behavior and avoids narrowing externally supplied higher-precision affine parameters.

The new regression matrix checks eager/JIT execution, f16/bf16/f32/f64, learned affine parameters and a separate float32-affine control. Outputs and input/parameter gradients are checked against an independent float64 formula. Exact base: 12 failures, 24 passing controls; fixed: all 36 pass. The selected LayerNorm/InstanceNorm and mixed-precision suite passes 99 tests, including those regressions.

Independent review additionally covers axes, nonunit affine values, strict promotion, InstanceNorm formats and higher-precision external affine parameters. Its 64 f32/f64 controls have bitwise-identical outputs and gradients on exact base and candidate. CPU only: no GPU/TPU or performance result is claimed. Ruff error/whitespace checks pass; configured pylint emits legacy configuration diagnostics, so no project pylint pass is claimed.

Related #911 addresses RMSNorm separately; this change only repairs LayerNorm and its InstanceNorm subclass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant