residual

← writingentry #003

Activation patching, slowly

What the normalised patching score actually measures, and three ways it quietly misleads.

Contents03 sections

In Reading the residual stream I gave activation patching one paragraph. It deserves more, because it is the tool that most often gets quoted as evidence, and the number it produces is easy to over-read.

The setup

Take two prompts that differ in exactly the way you care about: a clean prompt xcx_c where the model gets the answer right, and a corrupted prompt xrx_r where the relevant information has been swapped out. Pick a metric. The usual one is the logit difference between the correct answer AA and a plausible wrong answer BB:

Δ(x)=ℓA(x)−ℓB(x)\Delta(x) = \ell_A(x) - \ell_B(x)

Now run the model on xrx_r, but at one site (a head output, an MLP output, the residual stream at some layer and position) replace the activation hh with the value it had on the clean run. Write the result as Δ(xr; h←hc)\Delta(x_r;\, h \leftarrow h_c).

The normalised score

To compare sites, rescale so that the corrupted run sits at 0 and the clean run sits at 1:

S(h)=Δ(xr; h←hc)−Δ(xr)Δ(xc)−Δ(xr)S(h) = \frac{\Delta(x_r;\, h \leftarrow h_c) - \Delta(x_r)}{\Delta(x_c) - \Delta(x_r)}

A score near 00 means patching that site changed nothing. A score near 11 means that one activation was enough to bring the clean behaviour back.

Why it can leave [0,1][0, 1]

Nothing in the formula bounds it.1 If the patched activation pushes the correct logit further than the clean run ever did, S>1S > 1. If it moves the model away from the clean answer, S<0S < 0. Both happen in practice, and both are informative rather than errors.

A typical layer × position heat map. Each cell is one patch; darker means a higher normalised score.
A typical layer × position heat map. Each cell is one patch; darker means a higher normalised score.

Three ways the score misleads

  1. Sufficiency is not necessity. Patching in a clean activation tests whether it is enough to restore behaviour. It says nothing about whether the model needs it. For that you ablate: remove the activation from the clean run and see what breaks.
  2. The corruption defines the question. Swapping names asks “where does the model track which name is which?” Replacing names with random tokens asks something broader. The same model gives different heat maps for each, and neither is the “real” one.
  3. Distributed mechanisms hide. If ten heads each contribute a tenth of the effect, every single-head patch scores around 0.10.1 and the map looks empty. Patch groups of components before concluding nothing is there.2
Derivation: why the last-layer, last-position residual patch always scores 1

At the final layer and final position, the residual stream is the only input to the unembedding. Patching the whole stream there copies everything the logits depend on, so

Δ(xr; h←hc)=Δ(xc)⇒S(h)=1.\Delta(x_r;\, h \leftarrow h_c) = \Delta(x_c) \quad\Rightarrow\quad S(h) = 1.

That makes it a free sanity check: if this cell is not exactly 1, the hooks are wired wrong.

A sanity-check harness

Before trusting any heat map, I now run the same three assertions:

python
def normalised_score(patched: float, clean: float, corrupted: float) -> float:
    return (patched - corrupted) / (clean - corrupted)

assert abs(normalised_score(clean, clean, corrupted) - 1.0) < 1e-6      # full patch → 1
assert abs(normalised_score(corrupted, clean, corrupted)) < 1e-6        # no patch   → 0
assert abs(clean - corrupted) > 1.0, "denominator too small to trust"   # signal exists

A patching score is a statement about this pair of prompts, this metric and this site. Quote it with all three attached.

— note to self

Notes

  1. It’s a ratio of two differences, and only the denominator is fixed by the clean and corrupted runs. The numerator depends on what the patch does, which can overshoot in either direction. ↑

  2. Path patching, which restricts a patch to particular routes between components, is the usual next step when single-site patching comes back diffuse. ↑

↑ Back to top
comments.exe

loading…

    post a reply