Position Paper · A4A-PP-2026-002

Describe the Instrument, Declare the Plan: Preregistration for Research That Uses Generative AI

Theodore O. Cochran

AI for Altruism

Download PDF (66 KB)

Abstract

Publishers now ask researchers to describe how artificial intelligence was used to conduct their research, usually in the methods section. These requirements address what was used but not when its consequential configuration was fixed, so a description written after the results are known can satisfy them even when that configuration was chosen after the results were seen. The problem grows when a model makes design choices the researcher did not specify, as can happen when an AI assistant writes analysis code. This paper draws on two preregistered evaluations by the author. In both, the registered plan changed what could be reported, and in both, assumptions introduced by an AI coding assistant were found only by later checks. For confirmatory evaluations, it recommends that researchers prospectively specify the configuration of model-based instruments and the material implementation and analysis choices, including assumptions embedded in code, before observing the resulting measurements on evaluation data. It also recommends that journals ask authors to disclose the timing of those choices and any material changes made after observation.

Record

History

• 2026-10-16: Version 1 released.

Cite as

Cochran, Theodore O. 2026. “Describe the Instrument, Declare the Plan: Preregistration for Research That Uses Generative AI.” Position paper A4A-PP-2026-002, AI for Altruism. https://reports.ai4altruism.org/pp-2026-002/