Key Takeaways
- • An autonomous LLM-based replication pipeline reconstructed the main result of 1,328 papers from the Journal of Finance, Journal of Financial Economics, and Review of Financial Studies (2000-2020), with 75% reproducing the published sign and statistical significance in their original samples.
- • Only 42% of key results retain their published sign and significance in later data, rising to 48% after adjusting for shorter post-sample precision; the median coefficient falls to 45% of the published estimate out of sample, versus 93% in sample.
- • Decay concentrates at the end of each original sample: the coefficient ratio drops by about 0.28 at the boundary and stays flat afterward, which the author interprets as more consistent with data snooping than with investor learning or gradual market change.
- • Original t-statistics are the strongest predictor of persistence: about 31% of weakest-quartile results hold up out of sample versus roughly 72% in the strongest quartile.
- • Conventional publication-selection models explain only about one-fifth of the observed decay, implying specification search, changing relationships, or misspecification account for most of the gap.
Summary
A working paper by Dmitriy Muravyev, summarized by Elisabetta Basilico, PhD, CFA on Alpha Architect, separates two questions about published empirical finance research: whether a result can be reproduced using its original sample, and whether it holds when new data arrive. Using an autonomous LLM-based replication pipeline with independent audits of every replication, the author rebuilt the main result of 1,328 papers published in the Journal of Finance, Journal of Financial Economics, and Review of Financial Studies between 2000 and 2020. Of these, 1,005 had enough later data to be tested out of sample.
In sample, reproducibility is relatively strong. Around 75% of key results retained their original sign and statistical significance, the median replicated coefficient equaled 93% of the published magnitude, and only 1.3% became significant with the opposite sign. Most failures involved losing statistical significance rather than reversing direction.
Out of sample, the picture weakens considerably. Only 42% of results kept their sign and significance in later years, or 48% after adjusting for the lower precision of shorter post-sample periods. The median coefficient fell to 45% of the published estimate. Year-by-year re-estimation shows magnitudes stable within the original sample, a sharp decline around the sample's end, and little further change after publication, a pattern the author reads as more consistent with original-sample selection or data snooping than with gradual market change or investor learning.
Original statistical strength is the clearest predictor of durability. In the weakest quartile, about 31% of results retained sign and significance after precision adjustment, with roughly one-third of the original coefficient remaining. In the strongest quartile, about 72% retained their category and the median coefficient ratio was 0.62. Publication selection explains only a limited share of the decline: two fitted selection models predicted around 90% survival of the published coefficient, versus an actual median retention of 45%, implying publication bias accounts for roughly one-fifth of observed decay.
For investment advisors, the study argues that published effect sizes should be treated as starting points rather than permanent constants, particularly when used for expected returns, risk premiums, or factor exposures. The author emphasizes that replication answers only whether a result can be recovered, not whether it remains economically meaningful, and recommends greater weight for results that have been independently replicated and tested in new data. The paper notes its results are hypothetical and not indicative of future performance.
In sample, reproducibility is relatively strong. Around 75% of key results retained their original sign and statistical significance, the median replicated coefficient equaled 93% of the published magnitude, and only 1.3% became significant with the opposite sign. Most failures involved losing statistical significance rather than reversing direction.
Out of sample, the picture weakens considerably. Only 42% of results kept their sign and significance in later years, or 48% after adjusting for the lower precision of shorter post-sample periods. The median coefficient fell to 45% of the published estimate. Year-by-year re-estimation shows magnitudes stable within the original sample, a sharp decline around the sample's end, and little further change after publication, a pattern the author reads as more consistent with original-sample selection or data snooping than with gradual market change or investor learning.
Original statistical strength is the clearest predictor of durability. In the weakest quartile, about 31% of results retained sign and significance after precision adjustment, with roughly one-third of the original coefficient remaining. In the strongest quartile, about 72% retained their category and the median coefficient ratio was 0.62. Publication selection explains only a limited share of the decline: two fitted selection models predicted around 90% survival of the published coefficient, versus an actual median retention of 45%, implying publication bias accounts for roughly one-fifth of observed decay.
For investment advisors, the study argues that published effect sizes should be treated as starting points rather than permanent constants, particularly when used for expected returns, risk premiums, or factor exposures. The author emphasizes that replication answers only whether a result can be recovered, not whether it remains economically meaningful, and recommends greater weight for results that have been independently replicated and tested in new data. The paper notes its results are hypothetical and not indicative of future performance.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.