Component-wise logit attribution

Which write creates the max answer margin?

The final [ANS] logits decompose cleanly into the initial residual stream plus the four attention head writes. The main metric here attributes the model's final winning margin to those additive pieces.

The decomposition reconstructs the model logits

For each input, the report computes residual + head 0 + head 1 + head 2 + head 3 at the answer position and compares it with the model's actual logits.

Validation

Max reconstruction error: 3.41061e-13

Sampled accuracy: 1.0000

Mean final margin: 15.79

Average component contributions
Average contribution to the final winning margin on the stratified sample.

Contribution by max digit

The attribution uses the model's final strongest wrong digit for each input, so every component's margin contribution adds exactly to the model's final margin.

Margin contribution by true max digit

Diagnostic examples

Diagnostic stacked margin attribution
Bars show the additive components; black dots show the reconstructed total margin.
Input Component Correct logit Own top Own margin Final-margin contribution
[3, 7, 2, 5, 1]residual2.3170.460.66
[3, 7, 2, 5, 1]head 031.373-23.439.36
[3, 7, 2, 5, 1]head 115.173-0.723.63
[3, 7, 2, 5, 1]head 23.789-11.236.10
[3, 7, 2, 5, 1]head 335.316-23.38-2.89
[3, 7, 2, 5, 1]total87.94716.8616.86
[9, 1, 2, 3, 4]residual0.987-1.33-0.87
[9, 1, 2, 3, 4]head 020.80912.7319.02
[9, 1, 2, 3, 4]head 1-4.293-20.18-10.55
[9, 1, 2, 3, 4]head 235.21912.9712.97
[9, 1, 2, 3, 4]head 3100.646-11.30-2.74
[9, 1, 2, 3, 4]total153.35917.8317.83
[5, 4, 5, 1, 0]residual1.657-0.660.26
[5, 4, 5, 1, 0]head 022.583-33.6537.88
[5, 4, 5, 1, 0]head 111.543-4.349.66
[5, 4, 5, 1, 0]head 29.622-59.5925.07
[5, 4, 5, 1, 0]head 3107.786-57.78-57.78
[5, 4, 5, 1, 0]total153.16515.0915.09
[0, 0, 0, 0, 0]residual-2.157-4.46-1.16
[0, 0, 0, 0, 0]head 017.123-39.11-27.87
[0, 0, 0, 0, 0]head 1-3.983-19.87-10.44
[0, 0, 0, 0, 0]head 251.802-17.46-16.50
[0, 0, 0, 0, 0]head 3300.11068.3868.38
[0, 0, 0, 0, 0]total362.89012.4012.40
[0, 1, 0, 0, 0]residual-0.997-3.301.16
[0, 1, 0, 0, 0]head 044.993-11.2327.87
[0, 1, 0, 0, 0]head 16.463-9.4310.44
[0, 1, 0, 0, 0]head 268.302-0.9616.50
[0, 1, 0, 0, 0]head 3154.300-45.53-45.53
[0, 1, 0, 0, 0]total273.06110.4610.46