Technical Report · A4A-TR-2026-003
No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus
AI for Altruism; Independent Researcher
Download PDF (707 KB)
Abstract
Supplying context at inference time to a large multimodal model is an inexpensive lever for adapting speech transcription to a domain, and earlier results on smaller models reported large gains. This work tested that mechanism where it ships, in the prompt-conditioning layer of a production oral-history transcription tool, on a sample from its own production corpus. Full prompt-level context did not detectably change side-level word error rate (WER), and none of the four preregistered hypotheses was supported. The design was a within-item paired ablation, preregistered with the analysis code frozen by hash before the confirmatory batch was scored; two disclosed gpt-4o pilot sides had been scored earlier, during scorer development. Nineteen cassette sides, about 10.6 hours of degraded 1970s-80s interview audio, were reprocessed through the production code path under three prompt arms, crossed with two deployed commercial configurations, gpt-4o-transcribe and gemini-2.5-flash, and scored against operator-corrected verbatim references. For gpt-4o-transcribe the median paired difference between the full-context and no-context arms was +0.6 WER points, with a side-resampled interval of [-1.1, +1.0]; the Gemini estimates were too unstable to support a comparable negative inference. A post-hoc rerun found run-to-run pipeline variability larger than the confirmatory differences, so effects of that size cannot be resolved from one transcription per cell. An implementation audit verified the manipulation was live, and sequence-alignment analysis found a small improvement on complete context-listed phrases, too small to materially change side-level WER, and for Gemini coexisting with worsened unlisted-token error. Evaluating context mechanisms therefore requires sequence-aligned term-level, insertion, and speaker-label measures alongside aggregate accuracy.
Record
| Number | A4A-TR-2026-003 |
|---|---|
| Type | Technical report |
| Released | 2026-08-28 |
| Version | arXiv v1 |
| Status | Not peer reviewed. First released on arXiv before this series began, and numbered in it in October 2026. |
| Reviewer | Anonymous |
| License | CC BY 4.0 |
| arXiv | arXiv:2608.28875v1 |
| DOI | 10.48550/arXiv.2608.28875 |
| Registration | osf.io/ns49b |
| Materials | osf.io/9nsvr |
| SHA-256 | f5f0ae9be4ee08e8c735c431043edc98c73ff748a2215d7d24ffafdeebd69060 |
History
- 2026-08-28: Version 1 posted to arXiv.
Cite as
Cochran, Theodore O., Stephanie Dodson, and Keith Nore. 2026. “No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus.” Technical report A4A-TR-2026-003, AI for Altruism. https://reports.ai4altruism.org/tr-2026-003/