This repository contains the slides of the symposium The Multiverse of Multi-labs. Methodological and Statistical Aspects of Multi-Lab and Multiverse Studies during the Italian Psychological Association (AIP) conference in Noto (SR), Sicily 2024. The book of abstracts can be found here.
Modern research in psychology is adopting several tools to face the replicability and reproducibility crisis. In this context, multi-lab research combined with the multiverse approach is a very powerful yet complex approach. The symposium aims to present a modern overview of editorial aspects, statistical methods, and data management in the era of multi-lab and multiverse studies highlighting strengths, limitations, and challenges. The first talk by Crepaldi provides an introduction to multi-lab and multiverse studies focusing on the editorial aspects. The talk by Calignano integrated the multiverse approach into the multi-lab methodology focusing on data pre-processing showing the amount of researchers’ degrees of freedom and the impact on the final results. Summarising and presenting the results of a multiverse analysis requires appropriate descriptive and inferential tools. Given the lack of proper inferential methods, the talk by Finos proposed an innovative, flexible, and powerful inferential approach to summarise the results of a multiverse analysis. Especially for multi-lab studies, adopting open science practices in terms of transparency, preregistration, and data sharing, is becoming a new standard. However, data sharing is also a controversial, delicate, and often overlooked topic. The final talk by Scandola illustrates the problem of data management and sharing considering privacy policies and modern open science practices.
There’s no doubt that large-scale collaboration efforts are improving the precision and reliability of our science (wannabe) and, ultimately, the contribution of experimental psychology to the community at large. However, this radically new way of carrying out psychological research brings along several challenges. Some of these challenges are quite obvious and pervasive in our day-to-day activity (e.g., leading a group vs. working as a lonely wolf); others are more subtle (e.g., how to navigate seniority in a broad community vs. in your own lab, how to define –and promote– consensus). Some of these issues have strong implications on the editorial process – what gets published and what doesn’t. Leveraging on my experience at the British Journal of Psychology and Psychonomic Bulletin and Review, I’ll share some considerations (open questions, really) on these issues from the perspective of the Action Editor.
Open data is fundamental to open science, enhancing the trustworthiness and reproducibility of research alongside pre-registration. Many journals and European grants require data sharing and encourage compliance with the FAIR guidelines. However, despite its benefits, data sharing faces barriers, particularly around privacy. GDPR mandates strict privacy protections that challenge the feasibility of anonymisation, especially in niche populations such as those in clinical or neurodivergent settings. This talk will describe the inherent risks of data sharing, and present strategies to improve data anonymisation. It will explore potential solutions within the legal framework, acknowledging the complexities and ambiguities therein. In addressing these challenges, we aim to strike a balance between the imperatives of open science and the rights of individuals to privacy and data protection.
In this multi-lab study, we explored goal attribution in infants using various data analysis pipelines for pupillometry data from seven laboratories in Europe, the USA, and Canada. The main focus was on the multiverse approach, where data preprocessing is conceptualised as a garden of forking paths, each of which can dramatically affect subsequent statistical outcomes. This method shifts the focus from merely testing for statistical significance to exploring the robustness of findings, which can be influenced by how data are preprocessed. The present blind yet collaborative analysis aimed to examine the robustness of specific interactions (Target × Path in goal attribution). That is, the hypothesis that infants may deploy more cognitive resources when a hand reaches a familiar goal via a new path, as opposed to reaching for either a new or the familiar goal via a previously known path. However, instead of advocating for a one-best-method for data processing, the multiverse analysis embraces uncertainty by presenting the outcomes of various plausible preprocessing pipelines and exploring how they might impact the results. This approach supports the ongoing shift toward more open scientific practices that prioritise transparency over making groundbreaking but potentially fragile claims. In particular, the combination of the multiverse approach applied to a multi-lab effort acknowledges how different traditions in handling data can lead to opposite conclusions. This contribution stresses the importance of a collaborative strategy that empowers the findings from individual labs with broader, community-based insights.
When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this contribution, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the familywise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect.