From Criterion to Verdict: How a Language Model Decides Eligibility for Clinical Trials

AI can judge whether a patient qualifies for a clinical trial — but not faithfully explain why.

Status: Individual report, written since January 2026, for submission to the Regeneron Science Talent Search, 2027
Model: Qwen3-8B — 36 layers × 32 heads
Presented: At the 2026 fair, as half of Blackbox to Glassbox
Tools: TransformerLens for model internals; LaTeX for the report; 80+ hours of rented GPU on RunPod to self-host Llama and Qwen

1,152attention heads searched
6heads in the circuit
5patient records, hand-validated
100test patients, ablation

The problem

A language model can be handed a trial’s eligibility criteria and a patient’s record and asked for a verdict, and it will give one. In medicine an unexplained verdict cannot be trusted, and the explanation a model writes for itself is not guaranteed to be faithful to how it actually reached the answer. The question this project asks is narrower and more mechanical: which parts of the network carry the decision, and what does each of them do?

Approach

I searched all 1,152 attention heads of Qwen3-8B (36 layers × 32 heads), the open-source model used across my clinical-trial work. Each head was scored on how selectively it attended to the tokens that should matter, in three separate scans — one with the eligibility criterion as the signal, one with the patient’s medications, and one looking only at what feeds the model’s final answer. The highest-scoring heads from each scan were then validated by hand on five patient records.

What the circuit looks like

Six heads account for the decision. One, early in the network, reads the eligibility rule. Three retrieve the patient’s medications. One cross-checks the two, and one delivers the verdict. Ablating even one of the six sharply degrades accuracy across 100 test patients.

Criterion reader — layer 6, head 12
Fact retrieval — layer 19 heads 21 and 31; layer 20 head 8
Cross-reference — layer 22, head 5
Decision integrator — layer 24, head 16

Limitations

The circuit was identified by selectivity scoring and ablation, which is correlational evidence. Activation patching would give a stronger causal test, and the analysis covers a single model family. Both are on the list for the manuscript.

What’s next

The report is being revised toward a submission to the Regeneron Science Talent Search in 2027. Two researchers now advise it — one in computer science, one in psychology — after I cold-messaged more than eight of them across the country and two said yes. More on how that happened.

Heatmaps of attention-head selectivity across Qwen3-8B, with the six circuit heads marked — Fig. 1. Attention-head selectivity scans across the full model.

Chart showing the distinct specialization of each of the six attention heads in the circuit — Fig. 2. Specialization of each head in the six-head circuit.

Results showing how the identified circuit generalizes to held-out patient cases — Fig. 3. Generalization: of five held-out patients, the circuit fires only for the one on atorvastatin and aspirin — the combination that trips the trial’s exclusion criterion. In the words of the fair deck, “the circuit lights up for Ronald and no one else.”