Comment on Jakob, Dechamps & Maier
How to Read a Paper
DOI:
https://doi.org/10.31156/jaex.25420Keywords:
Bayesian t test, optional stopping, multiple testing, psychokinesis, mind-matter interaction, replication crisis, psiAbstract
In this issue of JAEX, Jakob et al. (2024) report results from an experiment to test whether an implicit psychokinetic effect related to unconscious intention can alter the outcome of quantum events. The experiment is well conceived and executed. However, the claim of evidence to support the experimental hypothesis is undermined by errors of analysis and of omission. Some are not obvious and require a careful reading to discern. I explain the nature of the mistakes, where and how they arise, and the consequences for the inferences the authors’ draw. I emphasize that a helpful way to detect such problems is by following the chain of inference in experimental reports. Editors, reviewers, and particularly authors can protect the integrity of the scientific literature by reading carefully in this way.
References
Bancel, P. A. (2017). Searching for global consciousness: A 17-year exploration. EXPLORE, 13(2), 94–101. https://doi.org/10.1016/j.explore.2016.12.003
Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425. https://doi.org/10.1037/a0021524
Bem, D., Tressoldi, P., Rabeyron, T., & Duggan, M. (2016). Feeling the future: A meta-analysis of 90 experiments on the anomalous anticipation of random future events. F1000Research, 4, 1188. https://doi.org/10.12688/f1000research.7177.2
Berry, D. A., & Hochberg, Y. (1999). Bayesian perspectives on multiple comparisons. Journal of Statistical Planning and Inference, 82(1), 215–227. https://doi.org/10.1016/S0378-3758(99)00044-0
Ehrlich, R. (2021). Eight preposterous propositions. Princeton University Press. https://doi.org/10.1515/9780691228402
Gelman, A., Hill, J., & Yajima, M. (2012). Why we (usually) don’t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness, 5(2), 189–211. https://doi.org/10.1080/19345747.2011.618213
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
Jahn, R., Dunne, B., Bradish, G., Dobyns, Y., Lettieri, A., Nelson, R., …. & Walter, B. (2000). Mind/ Machine interaction consortium: portREG replication experiments. Journal of Scientific Exploration, 14(4), 499–555.
Kekecs, Z., Palfi, B., Szaszi, B., Szecsi, P., Zrubka, M., Kovacs, M., … Aczel, B. (2023). Raising the value of research studies in psychological science by increasing the credibility of research reports: The transparent Psi project. Royal Society Open Science, 10(2), 191375. https://doi.org/10.1098/rsos.191375
Kruschke, J. K. (2021). Bayesian analysis reporting guidelines. Nature Human Behaviour, 5(10), Article 10. https://doi.org/10.1038/s41562-021-01177-7
Makin, T. R., & Orban de Xivry, J.-J. (2019). Ten common statistical mistakes to watch out for when writing or reviewing a manuscript. eLife, 8, e48175. https://doi.org/10.7554/eLife.48175
Muhmenthaler, M. C., Dubravac, M., & Meier, B. (2022). The future failed: No evidence for precognition in a large scale replication attempt of Bem (2011). Psychology of Consciousness: Theory, Research, and Practice. https://doi.org/10.1037/cns0000342
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14(9), Article 9. https://doi.org/10.1038/nn.2886
Palfi, B., & Dienes, Z. (2020). Why Bayesian “evidence for H1” in one condition and Bayesian “evidence for H0” in another condition does not mean good-enough Bayesian evidence for a difference between the conditions. Advances in Methods and Practices in Psychological Science, 3(3), 300–308. https://doi.org/10.1177/2515245920913019
Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2), 25152459211007467. https://doi.org/10.1177/25152459211007467
Schlitz, M., Bem, D., Marcusson-Clavertz, D., Cardeña, E., Lyke, J., Grover, R., … & Delorme, A. (2021). Two replication studies of a time-reversed (psi) priming task and the role of expectancy in reaction times. Journal of Scientific Exploration, 35(1), Article 1. https://doi.org/10.31275/20211903
Schmalz, X., Biurrun Manresa, J., & Zhang, L. (2023). What is a Bayes factor? Psychological Methods, 28(3), 705–718. https://doi.org/10.1037/met0000421
Schönbrodt, F. D., & Wagenmakers, E.-J. (2018). Bayes factor design analysis: Planning for compelling evidence. Psychonomic Bulletin & Review, 25(1), 128–142. https://doi.org/10.3758/s13423-017-1230-y
Sjölander, A., & Vansteelandt, S. (2019). Frequentist versus Bayesian approaches to multiple testing. European Journal of Epidemiology, 34(9), 809–821. https://doi.org/10.1007/s10654-019-00517-2
Varvoglis, M., & Bancel, P. A. (2015). Micro-psychokinesis. In In E. Cardeña, J. Palmer, & D. Marcusson-Clavertz (Eds.), Parapsychology: A handbook for the 21st century (pp. 266–281). McFarland.
Varvoglis, M., & Bancel, P. A. (2016). Micro-psychokinesis: Exceptional or universal? Journal of Parapsychology, 80(1), 37–44.
Wagenmakers, E.-J., Sarafoglou, A., Aarts, S., Albers, C., Algermissen, J., Bahník, Š., … Aczel, B. (2021). Seven steps toward more transparency in statistical practice. Nature Human Behaviour, 5(11), Article 11. https://doi.org/10.1038/s41562-021-01211-8
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019). Moving to a world beyond “ p < 0.05.” The American Statistician, 73(sup1), 1–19. https://doi.org/10.1080/00031305.2019.1583913
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Peter Bancel
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in JAEX are open access, freely and universally accessible online, and archived in the open journal’s Lund University website (https://journals.lub.lu.se). Articles in JAEX can be distributed under the terms of the Creative Commons Attribution 4.0 International License CCBY (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, with appropriate credit to the original author(s) and the source, a link to the Creative Commons license, and an indication if changes were made.
