Nullius in Verba

In this episode, we will discuss multiverse analysis–what it is, why we use it, and how we should use it. We also discuss many analyst projects, and Daniel's new preprint on multiverse analysis. 
 
Shownotes
Auspurg, K. (2025). Robustness is better assessed with a few thoughtful models than with billions of regressions. Proceedings of the National Academy of Sciences, 122(43), e2521917122.
Chamberlin, T. C. (1890). The method of multiple working hypotheses. Science, (366), 92–96.
Dochtermann, N. A., & Jenkins, S. H. (2011). Developing multiple hypotheses in behavioral ecology. Behavioral Ecology and Sociobiology, 65(1), 37–45.
Dreber, A., & Johannesson, M. (2025). The credibility gap: Evaluating and improving empirical research in the social sciences. Routledge.
Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University, 348(1-17), 3.
Knief, U., & Forstmeier, W. (2025). Are our 95% CIs only worth 45% confidence? bioRxiv, 2025-08.
Lakens, D., Rasti, S., & Tunç, M. N. (2026). There is only one correct analysis. (Preprint)
Platt, J. R. (1964). Strong Inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others. Science, 146(3642), 347–353.
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour, 4(11), 1208–1214.
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712.
Wimsatt, W. C. (2007). Re-engineering philosophy for limited beings: Piecewise approximations to reality. Harvard University Press.
 

After a short break due to busy calendars, we are back to discuss the role automation and artificial intelligence can play in science. Is AI evil, or does it just amplify the problems that were already present in science? Can automated AI driven checks help scientists to improve the way they work? If so, where are these tools helpful, and when? Enjoy. 
The Babbage quote is from Babbage in November 1839, recalling events in 1821; quoted in Harry Wilmot Buxton and Anthony Hyman (1988), Memoir of the Life and Labours of the Late Charles Babbage.
 
Alfred Whitehead: An Introduction to Mathematics (1911)
 
Blog about AI references: Evaluating Dr. Cuddy’s Claim that the Debunking of Power Posing is a Myth. https://daniellakens.blogspot.com/2026/05/evaluating-dr-cuddys-claim-that.html
 
Metacheck: DeBruine L, Mesquida C, Werner J, Lakens D (2026). metacheck: Check Research Outputs for Best Practices. doi:10.5281/zenodo.20704754, R package version 0.1.0, https://scienceverse.org/metacheck
 
Cummins, J., Clarke, B., Hussey, I., & Elson, M. (2026). RegCheck: A tool for structured comparisons between study registrations and papers (arXiv:2601.13330). arXiv. https://doi.org/10.48550/arXiv.2601.13330
 

In this episode, we continue our discussion of disagreement in science, shifting the conversation from why it matters to how to do it well.
 
Shownotes
Paul Graham. (2008). How to disagree. 
Rapoport's Rules. 
Cass Sunstien. The Rapoport Rules.
Preregistration is redundant, at best. 
An Evidence-Based Critique of the Cass Review 
Fiedler, K., Messner, C., & Bluemke, M. (2006). Unresolved problems with the “I”, the “A”, and the “T”: A logical and psychometric critique of the Implicit Association Test (IAT). European Review of Social Psychology, 17(1), 74–147. https://doi.org/10.1080/10463280600681248
Neyman, J. (1961). Silver Jubilee of My Dispute With Fisher. Journal of the Operations Research Society of Japan, 3, 145–154.
 

This is a two-part episode on the role of disagreement in science. In the first part, we discuss the "why," before moving on to the "how" in the next episode. Enjoy. 
 
Shownotes
Dellsén, F., & Baghramian, M. (2021). Disagreement in science: Introduction to the special issue. Synthese, 198(Suppl 25), 6011-6021.
Oreskes, N., & Conway, E. M. (2011). Merchants of doubt: How a handful of scientists obscured the truth on issues from tobacco smoke to global warming. Bloomsbury Publishing USA.
Popper, K. (1959). The Logic of Scientific Discovery. London: Hutchinson.
Seidel, M. (2021). Kuhn’s two accounts of rational disagreement in science: an interpretation and critique. Synthese, 198(25), 6023-6051.
Shaw, J. (2021). Feyerabend and manufactured disagreement: reflections on expertise, consensus, and science policy. Synthese, 198(25), 6053-6084.
 

In this episode, we discuss the problem of miscitation. How often are citations to the scientific literature outright misleading? Do we really need to spell out that people  are supposed to read what they cite? What can we learn from other fields? Or should we just live with the fact that a decent percentage of citations in the literature are wrong? Enjoy. 
 
Careless citations don't just spread scientific myths – they can make them stronger (Nature)
Cobb, C. L., Crumly, B., Montero-Zamora, P., Schwartz, S. J., & Martínez Jr, C. R. (2024). The problem of miscitation in psychological science: Righting the ship. American Psychologist, 79(2), 299–311.
Simmering, M. J., Fuller, C. M., Leonard, S. R., & Simmering, V. R. (2025). Cognitive biases and research miscitations. Applied Psychology, 74(1), e12589.
Qinyue Liu, Amira Barhoumi, Cyril Labbé. (2024). Miscitations in scientific papers: Dataset and detection. International Workshop on Bibliometric-enhanced Information Retrieval. Glasgow, United Kingdom.
Lazonder, A. W., & Janssen, N. (2022). Quotation accuracy in educational research articles. Educational Research Review, 35(1), https://doi.org/10.1016/j.edurev.2021.100430
James, W. (1914). The energies of men. New York : Moffat, Yard and Company. http://archive.org/details/energiesofmen00jameuoft
Beyerstein, B.L. (1999) Whence cometh the myth that we only use ten percent of our brains? In, S. Della Sala (Ed.), Mind Myths: Exploring Everyday Mysteries
Jergas, H., & Baethge, C. (2015). Quotation accuracy in medical journal articles—A systematic review and meta-analysis. PeerJ, 3, e1364. https://doi.org/10.7717/peerj.1364
Bruton, S. V., Macchione, A. L., Brown, M., & Hosseini, M. (2025). Citation Ethics: An Exploratory Survey of Norms and Behaviors. Journal of Academic Ethics, 23(2), 329–346. https://doi.org/10.1007/s10805-024-09539-2
Simkin, M., & Roychowdhury, V. (2006). Do You Sincerely Want to Be Cited? Or: Read Before You Cite. Significance, 3(4), 179–181. https://doi.org/10.1111/j.1740-9713.2006.00202.x
Simmering, M. J., Fuller, C. M., Leonard, S. R., & Simmering, V. R. (2025). Cognitive biases and research miscitations. Applied Psychology, 74(1), e12589. https://doi.org/10.1111/apps.12589
Bluebook: https://www.legalbluebook.com
 

In this two-part episode, we discuss incentives in science and academia. We discuss the various incentives in science, including recognition, citations, money, and the kick in the discovery.
 
Shownotes
Cole, S., & Cole, J. R. (1967). Scientific output and recognition: a study in the operation of the reward system in science. American Sociological Review, 377–390.
Crane, D. (1965). Scientists at major and minor universities: A study of productivity and recognition. American Sociological Review, 699–714.
Merton, R. K. (1963). Resistance to the systematic study of multiple discoveries in science. European Journal of Sociology/Archives Européennes de Sociologie, 4(2), 237–282.
Stephan, P. (2015). How economics shapes science. Harvard University Press.
Tal Yarkoni - No, it’s not The Incentives—it’s you
Tom Leher - Lobachevsky (1953)
 

In this two-part episode, we discuss incentives in science and academia. We discuss the various incentives in science, including recognition, citations, money, and the kick in the discovery.
 
Shownotes
Cole, S., & Cole, J. R. (1967). Scientific output and recognition: a study in the operation of the reward system in science. American Sociological Review, 377–390.
Crane, D. (1965). Scientists at major and minor universities: A study of productivity and recognition. American Sociological Review, 699–714.
Merton, R. K. (1963). Resistance to the systematic study of multiple discoveries in science. European Journal of Sociology/Archives Européennes de Sociologie, 4(2), 237–282.
Stephan, P. (2015). How economics shapes science. Harvard University Press.
 

In this episode, we discuss the problems associated with vague concepts in psychological science. We talk about the jingle-jangle fallacy, the trade-off between broad concepts and more precise concepts, if we should generate databases of conceptual definitions, and how the reward structures can get in the way of specifying concepts clearly.
 
Shownotes
Aikins, H. A. (1902). The principles of logic. H. Holt and Company. https://www.loc.gov/resource/gdcmassbookdig.principlesoflogi00aiki/
Chalmers, D. J. (2025). What is conceptual engineering and what should it be? Inquiry, 68(9), 2902–2919. https://doi.org/10.1080/0020174X.2020.1817141
Gerring, J. (1999). What Makes a Concept Good? A Criterial Framework for Understanding Concept Formation in the Social Sciences. Polity, 31(3), 357–393. https://doi.org/10.2307/3235246
Hirsch, P. M., & Levin, D. Z. (1999). Umbrella Advocates Versus Validity Police: A Life-Cycle Model. Organization Science, 10(2), 199–212. https://doi.org/10.1287/orsc.10.2.199
Sartori, G. (1970). Concept Misformation in Comparative Politics. The American Political Science Review, 64(4), 1033–1053. https://doi.org/10.2307/1958356
Thorndike, E. L. (Edward L. (with University of California Libraries). (1904). An introduction to the theory of mental and social measurements. New York : Science Press. http://archive.org/details/theoryofmentalso00thor
Truman Lee Kelley, (1927). Interpretation of education measurements. World book company. http://archive.org/details/bwb_P9-AQI-186
APA Dictionary of Psychology: https://dictionary.apa.org/ 
 

In this episode, we continue our discussion of scientism. We talk about 6 problems with scientism that have been raised by Susan Haack, if we should feel bad about having some sympathy for scientism, and whether the contributions of all scientifici disciplines deserved the label of 'knowledge'. Enjoy. 
 
References:
 
Haack, S. (2012). Six Signs of Scientism. Logos & Episteme, 3(1), 75–95. https://doi.org/10.5840/logos-episteme20123151
Brown, N. J. L., Sokal, A. D., & Friedman, H. L. (2013). The complex dynamics of wishful thinking: The critical positivity ratio. American Psychologist, 68(9), 801–813. https://doi.org/10.1037/a0032850
Peels, R. (2023). Scientism and scientific fundamentalism: What science can learn from mainstream religion. Interdisciplinary Science Reviews, 48(2), 395–410. https://doi.org/10.1080/03080188.2022.2152246
de Ridder, Jeroen. “Science and Scientism in Popular Science Writing.” Social Epistemology Review and Reply Collective 3, no. 12 (2014): 23-39. https://social-epistemology.com/2014/11/03/science-and-scientism-in-popular-science-writing-jeroen-de-ridder/ 
Meehl, P. E. (2004). Cliometric metatheory III: Peircean consensus, verisimilitude and asymptotic method. The British Journal for the Philosophy of Science, 55(4), 615–643.
Mizrahi, M. (2017). What’s so bad about scientism? Social Epistemology, 31(4), 351–367. https://doi.org/10.1080/02691728.2017.1297505
Hayek, F. A. (1952). The Counter-Revolution of Science: Studies on the Abuse of Reason. Glencoe, Illinois: The Free Press. 
Rulkens, C. C. S., Peels, R., Stols-Witlox, M., Meloni, S., Lechner, I. M., & Bouter, L. (2025). The attribution of two portraits of Rembrandt revisited: A replication study in art history. Humanities and Social Sciences Communications, 12(1), 1347. https://doi.org/10.1057/s41599-025-05523-2
 

In this two-part episode, we delve into the topic of scientism. Is science the best way to generate knowledge? Or are we giving too much deference to science if we believe this? In this first part, we discuss what scientism is, what - if anything - is wrong with scientism, and whether it is bad to be a scien-ti-sim-ist?
 
References:
 
Haack, S. (2012). Six Signs of Scientism. Logos & Episteme, 3(1), 75–95. https://doi.org/10.5840/logos-episteme20123151
Brown, N. J. L., Sokal, A. D., & Friedman, H. L. (2013). The complex dynamics of wishful thinking: The critical positivity ratio. American Psychologist, 68(9), 801–813. https://doi.org/10.1037/a0032850
Peels, R. (2023). Scientism and scientific fundamentalism: What science can learn from mainstream religion. Interdisciplinary Science Reviews, 48(2), 395–410. https://doi.org/10.1080/03080188.2022.2152246
de Ridder, Jeroen. “Science and Scientism in Popular Science Writing.” Social Epistemology Review and Reply Collective 3, no. 12 (2014): 23-39. https://social-epistemology.com/2014/11/03/science-and-scientism-in-popular-science-writing-jeroen-de-ridder/ 
Meehl, P. E. (2004). Cliometric metatheory III: Peircean consensus, verisimilitude and asymptotic method. The British Journal for the Philosophy of Science, 55(4), 615–643.
Mizrahi, M. (2017). What’s so bad about scientism? Social Epistemology, 31(4), 351–367. https://doi.org/10.1080/02691728.2017.1297505
Hayek, F. A. (1952). The Counter-Revolution of Science: Studies on the Abuse of Reason. Glencoe, Illinois: The Free Press. 
Rulkens, C. C. S., Peels, R., Stols-Witlox, M., Meloni, S., Lechner, I. M., & Bouter, L. (2025). The attribution of two portraits of Rembrandt revisited: A replication study in art history. Humanities and Social Sciences Communications, 12(1), 1347. https://doi.org/10.1057/s41599-025-05523-2