AI-Based Dementia Diagnosis and Monitoring for the Elderly

Open-source models perform poorly when prompted — but their internal representations carry far richer information.

Venue: Alameda County Science & Engineering Fair, Software — HS-SOFT-341
Status: Abstract No. 1368, submitted October 2025 to the 2026 Annual Meeting of the American Academy of Neurology
With: Co-authored with a school project partner
Data: DementiaBank Pitt Corpus — Cookie Theft picture descriptions, 231 transcripts. Access won through a teacher-sponsored, 10-page written proposal
Models: GPT-4o, Claude Sonnet 3.5 and 3.7, o1-preview, o1-mini, Llama 3-8B, Llama 3.1-8B

0.88F1 — Llama 3.1-8B, probed
0.55F1 — the same model, prompted
0.83F1 — best prior ML baseline, BERT
231transcripts, 71 held out for testing

The problem

By one estimate cited in the project, 59% of people with dementia are undiagnosed or unaware of their condition. Many older adults live alone, face real barriers getting to a clinic, and avoid testing because of stigma. The question I set out to answer: can a high-performance, secure, low-cost, and easily accessible screening tool be built from large language models?

Data

The Pitt Corpus is restricted. I wrote a ten-page proposal and won teacher-sponsored access to it. DementiaBank’s Pitt Corpus — recordings of the standard “Cookie Theft” picture-description task, transcribed to text. 105 transcripts from cognitively normal speakers and 126 from patients with dementia, split into 160 for training and 71 held out for testing.

Two approaches

Prompting. Six models — GPT-4o, Claude Sonnet 3.5, o1-preview, Claude Sonnet 3.7, Llama 3-8B, and Llama 3.1-8B — were instructed, as an expert in linguistics and neuropsychology, to weigh each transcript’s syntax, vocabulary, structure, narration, grammar, semantic discourse, stylistics, and pragmatics, and classify it.

Probing. The same transcripts were run through the two open-source Llama models, and the final-token representation was read out at two points in every layer (4,096 features per probe). A linear classifier — either distance-from-mean or logistic regression — was trained on each probe, the best probe was chosen on a validation set, and only then evaluated on the test set.

Results

The cleanest comparison is one model against itself. Prompted, Llama 3.1-8B managed an F1 of 0.55 and an accuracy of 0.61. Probed — the same weights, read from the inside — it reached 0.88 and 0.87.

Across the whole field: prompting topped out at an F1 of 0.73 (o1-preview); GPT-4o scored 0.66 and Claude Sonnet 3.5 scored 0.72. Prompted, the open-source Llama models managed only 0.42 and 0.55. Probed, those same two models reached 0.87 and 0.88 with logistic regression — above the SVM, logistic-regression, and BERT baselines reported in prior work (0.78–0.83). More reasoning did not help: on Claude Sonnet 3.7, raising the thinking budget from 3K to 10K tokens lowered F1 from 0.61 to 0.55, and o1-mini (0.60) trailed o1-preview (0.73).

What it means

Prompt-based classification falls short; text alone may miss subtle markers of cognitive decline.
A transformer’s internal state carries a signal its own answer does not, and a linear classifier on top of it beat the traditional pipeline.
Quality data is the bottleneck. Privacy makes dementia data scarce, and synthetic patient data is one way forward.

Limitations

One dataset and one task. The test set is small, at 71 transcripts. The SVM, logistic-regression, and BERT baselines are taken from prior studies rather than re-run on the same split.

Where it led

The same move — read the model’s internal state instead of trusting its output — became the clinical-trial probes a year later. The same instinct, that a system’s output is not evidence that it works, is what I now spend my weekends on at Neural Care.

Table of F1, accuracy, precision and recall for prior machine-learning baselines, six prompted models, and probes on two Llama models — Fig. 1. F1, accuracy, precision, and recall across prior ML baselines, six prompted models, and probes on two Llama models.

Diagram of a transformer layer showing the two probe points, before the attention block and before the MLP — Fig. 2. Where the two probes sit in each transformer layer.

Table showing F1 falling from 0.61 to 0.55 as Claude 3.7's thinking budget rises from 3K to 10K tokens — Fig. 3. Claude Sonnet 3.7: more thinking tokens, lower F1.

Table comparing o1-mini (F1 0.60) with o1-preview (F1 0.73) — Fig. 4. Model size: o1-mini vs. o1-preview.

Preview of the ACSEF 2025 poster, AI Based Dementia Diagnosis and Monitoring for the Elderly — Fig. 5. The ACSEF 2025 poster (HS-SOFT-341). Click to open at full size.