Researchers sometimes use language models to simulate how a Hispanic respondent might answer a survey question. The practice is called silicon sampling, and it rests on an assumption: that the model is reaching for an internal representation of "Hispanic" and reasoning from there. That what the model says about a demographic group reflects what the model knows about that group.

That assumption had never been tested mechanistically. So I tested it.

The short answer: behavioral output and internal representation can be substantially decoupled. A model can pass every fairness benchmark we throw at it while internally computing demographic information in ways those benchmarks cannot detect. In some cases, while that internal demographic information is doing nothing at all.

I built a four-phase mechanistic evaluation pipeline and ran it across eight open-weights language models, ranging from GPT-2 (117M parameters) to Pythia-12B.

1. Behavioral baseline. Run the standard fairness benchmark (BBQ) and record what the model says.
2. Activation probing. Look inside the model. Train a linear classifier on its internal activations to test whether demographic identity is even encoded.
3. Causal patching. Surgically swap one demographic representation for another inside the model's residual stream and measure whether the output changes. This is the test for whether the encoding actually drives anything.
4. Semantic characterization. Project the demographic encoding directions onto the model's vocabulary to surface representational artifacts invisible to behavioral evaluation.

Phases 1 through 3 produce statistically validated findings, permutation-tested with 1,000 iterations. Phase 4 is exploratory and model-specific.

If you publish silicon-sampling results (claims about what demographic communities think, want, or need based on simulated LLM responses) and you've validated your model on behavioral fairness benchmarks, that validation is not enough. The model can show clean behavioral output while drawing on representations shaped by criminalization associations, color-compound polysemy, or majority-context dependence.

If you build alignment systems and your evaluation stops at output-level fairness metrics, you may be treating only the symptom. Mistral-7B-Instruct's results suggest current behavioral interventions can leave the underlying representation intact and simply route around it.

For evaluation generally: geometric presence (what a probe measures), causal activity (what patching measures), and behavioral expression (what benchmarks measure) are three distinct properties. They can diverge. Treating any one as a proxy for the others produces conclusions the evidence does not support.

• Modern frontier models. This audit covered open-weights models up to Mistral-7B because mechanistic intervention requires direct access to activations. Replicating findings on closed frontier models (GPT-4, Claude, Llama-3) is methodologically harder but increasingly important.
• Sparse autoencoder decomposition of demographic encoding directions, to test whether they are monosemantic features or superpositions of correlated social signals.
• Causal connection between intermediate-layer activity and final-layer outputs. Whether the criminalization-adjacent vocabulary detected at intermediate layers propagates to influence what the model actually generates.
• Open-ended generation benchmarks (BOLD, HolisticBias) to test whether the behavioral collapse documented here generalizes beyond multiple-choice formats.
• Application to silicon-sampling pipelines in the wild, running Phases 1 through 3 on models being used in published demographic simulation studies.

The pipeline is open-source and works on any open-weights model supported by TransformerLens.