Original
The Hidden Universe of Operationalizing Natural Experiments: Navigating Extreme Weather
Abstract
Natural or quasi-experimental research designs are based on exogenous variation, oftentimes requiring operationalization before use in statistical tests. For this, researchers regularly lack clear theoretical guidance and need to make threshold decisions. We show for three types of shocks—extreme weather, immigration exposure, and commodity price changes—that this leads to a vast menu of equally plausible measurements, which are used unsystematically in ensuing literatures. This hidden universe of operationalization decisions has consequences: Drawing on geocoded survey and weather panel data for Switzerland, we show that different measurements of extreme weather exposure exhibit vast statistical and geographical variation. Importantly, researchers can support arbitrary conclusions when using singular operationalizations, while, in our case, the whole set indicates robust null effects. These results highlight that measurement decisions induce relevant researcher degrees of freedom rarely made transparent. We consequently discuss how current practice could improve, concluding with a template and recommendations.
中文
自然实验操作化的隐藏宇宙：驾驭极端天气
摘要
自然实验或准实验研究设计基于外生变异，在用于统计检验之前往往需要操作化。对此，研究者通常缺乏明确的理论指导，并需要作出阈值决策。我们针对三类冲击——极端天气、移民暴露和商品价格变化——表明，这会导致大量同样合理的测量方式，而这些测量在后续文献中被不系统地使用。这一操作化决策的隐藏宇宙会带来后果：基于瑞士的地理编码调查和天气面板数据，我们表明极端天气暴露的不同测量在统计和地理上呈现出巨大差异。重要的是，研究者在使用单一操作化时可以支持任意结论，而在我们的案例中，整套测量却表明稳健的零效应。这些结果凸显出，测量决策会引入相关的研究者自由度，而这一点很少被透明化。因此，我们讨论当前实践如何改进，并以一个模板和建议作结。
关键词
自然实验、操作化、极端天气、研究者自由度、测量、准实验设计、外生变异、稳健性