Original
New Evidence and Design Considerations for Repeated Measure Experiments in Survey Research
Abstract
We re-examine recent influential claims that repeated measure experimental designs offer large precision gains without biasing treatment effect estimates in survey research. We test these claims by experimentally varying the design of six classic political science experiments across three distinct large samples of U.S. adults (total $ N=\mathrm{13,163} $ ). In contrast to prior evidence, we observe consistent attenuation of treatment effects in repeated measure designs. However, we show in simulations that this average design effect is small enough, and the precision gains large enough, that we recommend repeated measure designs for broad application—though (large-N) post-only designs may be preferable when research priorities include estimating the precise magnitude of a treatment effect. We additionally explore how several design considerations affect the bias-precision trade-off, such as within-subject versus between-groups designs, the relative separation of repeated measures within single surveys, and differences in respondent characteristics across sample types.
中文
调查研究中的重复测量实验：新证据与设计考量
摘要
我们重新审视近期有影响力的主张，即重复测量实验设计在调查研究中的精度增益巨大且不会使处理效应估计产生偏误。我们通过实验改变六个经典政治学实验的设计，并在三个不同的美国成年人大型样本中检验这些主张（总N=13,163）。与先前证据相反，我们在重复测量设计中观察到处理效应的一致衰减。然而，我们通过模拟表明，这一平均设计效应足够小，而精度增益足够大，因此我们建议将重复测量设计广泛适用——不过，当研究优先事项包括估计处理效应的精确大小时，（大样本）仅后测设计可能更可取。我们还探讨了几种设计考量如何影响偏误-精度权衡，例如被试内设计与组间设计、单次调查中重复测量之间的相对间隔，以及不同样本类型中受访者特征的差异。
关键词
重复测量实验、调查实验、实验设计、处理效应、偏误-精度权衡、被试内设计、仅后测设计、样本类型