class: title-slide, center, middle .title[Multivariate Meta-analysis in the Multiverse 🚀] </br> </br> </br> .author[Chiara Montuori, Filippo Gambarota, Gianmarco Altoè and Barbara Arfè] </br> </br> </br> </br> </br> </br> </br> .event[University of Padova @Psicostat] </br> .event[04/03/2022] <img src="data:image/png;base64,#img/logo_psicostat.png" width="10%" style="display: block; margin: auto;" /> --- class: inverse, center, middle # Meta-analysis in 2 minutes 😱🕐 --- # 1. Changing the statistical unit When we do a meta-analysis we are **switching the statistical unit** from e.g. participants to studies with multiple participants -- <img src="data:image/png;base64,#img/statistical-unit.svg" style="display: block; margin: auto;" /> --- class: white-bg # 2. Summarizing with Effect Sizes Usually (but not always) we use a standardized effect size measure (e.g., Cohen's *d* or Pearson Correlation) in order to compare studies with different designs, dependent measure (e.g., Accuracy and Reaction Times) .pull-left[ Cohen's *d* <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-3-1.png" width="3000" style="display: block; margin: auto;" /> ] .pull-right[ Correlation <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-4-1.png" width="3000" style="display: block; margin: auto;" /> ] --- class: white-bg # 3. Weighting by precision In order do a meta-analysis we need to pool together multiple studies taking into account that some studies should have more weight (e.g., higher sample size). In the simplest form, a meta-analysis is essentially a weighted average. <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-6-1.png" width="80%" style="display: block; margin: auto;" /> --- class: white-bg # 3. Weighting by precision In order do a meta-analysis we need to pool together multiple studies taking into account that some studies should have more weight (e.g., higher sample size). In the simplest form, a meta-analysis is essentially a weighted average. <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-7-1.png" width="80%" style="display: block; margin: auto;" /> --- # 4. Fixed effect vs Random effect This is an essential (*and often misunderstood*) step: -- .pull-left[ The **fixed-effect** model assume a single **population-level** effect/parameter to be estimated `\(\mu_{fixed}\)`. Observed variability between effects is due to **sampling error** only. <img src="data:image/png;base64,#img/fixed_effect.svg" width="70%" style="display: block; margin: auto;" /> ] -- .pull-right[ The **random-effect** model assume a distribution of **population-level** effects where the **true effect can vary**. We need to estimate the mean `\(\theta_{random}\)` and the variance `\(\tau^2\)` <img src="data:image/png;base64,#img/random_effect.svg" width="70%" style="display: block; margin: auto;" /> ] --- # 5. Complex data structure In some situations we need to take into account multilevel and/or multivariate situations: - multiple studies within the same paper (multilevel structure) - multiple effects (dependent variables) measured on the same pool of participants (e.g., Accuracy and Reaction Times) -- <img src="data:image/png;base64,#img/data-structure.svg" style="display: block; margin: auto;" /> --- class: inverse, center, middle # The present work --- # Coding and Executive Functions The impact of **coding training** on children (~5-10 age) executive functions (**outcomes**). We selected only **randomized-control trials**. <br/> <img src="data:image/png;base64,#img/research-design.svg" style="display: block; margin: auto;" /> --- # First problem: **Effect size** For PPC designs one of the mostly used effect size is the `\(dpcc\)` by Morris (2008). In particular the `\(dpcc_2\)`: </br> `$$d_{pcc_2} = c_p \frac{(M_{T,post} - M_{T,pre}) - (M_{C,post} - M_{C,pre})}{SD_{pooled,pre}}$$` </br> With sampling variance: </br> `$$\sigma^2(d_{pcc_2}) = c^2_p(1-\rho)(\frac{n_t+n_c}{n_t n_c})(\frac{n_t+n_c-2}{n_t+n_c-4})(\frac{1 + \Delta^2}{2(1-\rho)(\frac{n_t+n_c}{n_t n_c})})$$` </br> The critical component is the `\(\rho\)` i.e. the **pre-post** correlation that is often **not reported**! --- # Second problem: **Multiple Effect Sizes** When measuring a certain cognitive function (e.g., **working memory**) different authors could use different measures. We decided to recode the **raw** test measure `\(y_1, y_2,...y_n\)` into the **latent** psychological variable `\(y_i\)`. This create a situation where we have multiple `\(y_i\)` on the same paper. Borenstein et al. (2009) and also the `metafor` package with the `metafor::aggregate.escalc()` function implemented a way to combine multiple dependent effect sizes: </br> .pull-left[ <img src="data:image/png;base64,#img/aggregate_help.png" width="90%" style="display: block; margin: auto;" /> ] .pull-right[ <img src="data:image/png;base64,#img/borenstein-formula.png" width="90%" style="display: block; margin: auto;" /> ] --- # Third problem: **Multiple Outcomes** This is the classical **multivariate situation** where we need to take into account the correlation between different measures on the same pool of participants: </br> <img src="data:image/png;base64,#img/multivariate-structure.svg" style="display: block; margin: auto;" /> -- We need this matrix for each study, creating a **huge** variance-covariance matrix. But most importantly we need the **covariance between effects**! --- # Fourth problem: **Limited amount of studies** Often, for new area of research or not really widespread research topics the amount of available studies is limited. In particular according to our **strict** inclusion criteria we found **9 papers** with several effects within each paper: -- <table class="table table-striped table-condensed" style="width: auto !important; margin-left: auto; margin-right: auto;"> <thead> <tr> <th style="text-align:left;"> Outcome </th> <th style="text-align:right;"> n </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> Cognitive Flexibility Acc. </td> <td style="text-align:right;"> 2 </td> </tr> <tr> <td style="text-align:left;"> Inhibition Acc. </td> <td style="text-align:right;"> 5 </td> </tr> <tr> <td style="text-align:left;"> Planning Acc. </td> <td style="text-align:right;"> 3 </td> </tr> <tr> <td style="text-align:left;"> Problem Solving </td> <td style="text-align:right;"> 7 </td> </tr> <tr> <td style="text-align:left;"> Working Memory Acc. </td> <td style="text-align:right;"> 2 </td> </tr> </tbody> </table> --- # Why is a problem? -- Depending on the model we need to estimate **one or several parameters**: - Williams et al. (2018) clearly demonstraed the biased estimation of `\(\tau\)` with a limited amount of studies impacting also the estimation of `\(\mu\)` especially using the classical DerSimonian and Laird (1986) or REML estimators. - With a multivariate model we estimate several `\(\mu\)` and, in case of the random-effect model, several `\(\tau\)` <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#img/williams-2020-tau.png" alt="Simulated sampling distribution of Tau from Williams et al. (2018)" width="80%" /> <p class="caption">Simulated sampling distribution of Tau from Williams et al. (2018)</p> </div> --- class: inverse, center, middle # Our solution? ...a Multiverse approach! 🚀 --- # Why multiverse? <img src="data:image/png;base64,#img/multiverse-gelman.png" width="80%" style="display: block; margin: auto;" /> <img src="data:image/png;base64,#img/multiverse-voracek.png" width="80%" style="display: block; margin: auto;" /> --- # Why multiverse? <br/> <br/> <br/> <br/> .blockquote[ We suggest that instead of performing only one analysis, researchers could perform a multiverse analysis [...] A multiverse analysis offers an idea of **how much the conclusions change because of arbitrary choices in data construction** and gives pointers as to **which choices are most consequential** in the fragility of the result. ] --- # Our choice...Fixed-effect multivariate model! <br/> `$$\begin{pmatrix} y_{i1} \\ \vdots \\ y_{ij} \end{pmatrix} \sim MVN \Bigg( \begin{pmatrix} \mu_{i1} \\ \vdots \\ \mu_{ij} \end{pmatrix}, \begin{pmatrix} \sigma^2_{i1} & \dots & \sigma_{i1,ij}\\ \vdots & \ddots & \vdots\\ \sigma_{i1,ij} & \dots & \sigma^2_{ij}\\ \end{pmatrix}\Bigg)$$` Where each study `\(y_i\)` can have multiple outcomes `\(j\)` and come from a multivariate normal distribution with means the vector of effects and the variance-covariance matrix. -- - Estimating an effect size for each outcome (as series of univariate analysis) - No `\(\tau\)` estimation (compared to the random-effect model) - Takes into account the multivariate data structure (compared to univariate or multilevel analysis) - More appropriate with a limited amount of studies (see Cai & Fan, 2020) --- # But our Multiverse... - Fixed-effect or random-effect Model? - Multivariate or Univariate? - Which correlations to use? - A `\(\rho_{pre-post}\)` of 0.5, 0.7 and 0.9 - A `\(\rho_{agg}\)` of 0.3, 0.5, 0.7 - A `\(\rho_{multi}\)` of 0.3, 0.5 and 0.7 We have a total of 108 meta-analysis to compute! <iframe src="https://giphy.com/embed/75ZaxapnyMp2w" width="3000" height="300" data-external="1" style="border: none;"></iframe> --- class: inverse, center, middle # The main results... --- # The main results... <br/> <br/> <template id="717cf376-f4d1-4cf0-bb14-8b029901ce4c"><style> .tabwid table{ border-spacing:0px !important; border-collapse:collapse; line-height:1; margin-left:auto; margin-right:auto; border-width: 0; display: table; margin-top: 1.275em; margin-bottom: 1.275em; border-color: transparent; } .tabwid_left table{ margin-left:0; } .tabwid_right table{ margin-right:0; } .tabwid td { padding: 0; } .tabwid a { text-decoration: none; } .tabwid thead { background-color: transparent; } .tabwid tfoot { background-color: transparent; } .tabwid table tr { background-color: transparent; } </style><div class="tabwid"><style>.cl-3b2402da{}.cl-3b17a56c{font-family:'Arial';font-size:11pt;font-weight:bold;font-style:normal;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;}.cl-3b17a56d{font-family:'Arial';font-size:11pt;font-weight:bold;font-style:italic;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;}.cl-3b17a56e{font-family:'Arial';font-size:11pt;font-weight:normal;font-style:normal;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;}.cl-3b17a56f{font-family:'Arial';font-size:6.6pt;font-weight:normal;font-style:normal;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;position: relative;bottom:3.3pt;}.cl-3b17a570{font-family:'Arial';font-size:6pt;font-weight:normal;font-style:normal;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;position: relative;bottom:3pt;}.cl-3b17a571{font-family:'Arial';font-size:10pt;font-weight:normal;font-style:italic;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;}.cl-3b17a572{font-family:'Arial';font-size:10pt;font-weight:normal;font-style:normal;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;}.cl-3b17a573{font-family:'Arial';font-size:6pt;font-weight:normal;font-style:normal;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;position: relative;top:3pt;}.cl-3b17f38c{margin:0;text-align:center;border-bottom: 0 solid rgba(0, 0, 0, 1.00);border-top: 0 solid rgba(0, 0, 0, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);padding-bottom:5pt;padding-top:5pt;padding-left:5pt;padding-right:5pt;line-height: 1;background-color:transparent;}.cl-3b17f38d{margin:0;text-align:left;border-bottom: 0 solid rgba(0, 0, 0, 1.00);border-top: 0 solid rgba(0, 0, 0, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);padding-bottom:5pt;padding-top:5pt;padding-left:5pt;padding-right:5pt;line-height: 1;background-color:transparent;}.cl-3b1842e2{width:91.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0 solid rgba(0, 0, 0, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842e3{width:54.2pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0 solid rgba(0, 0, 0, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842e4{width:139.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0 solid rgba(0, 0, 0, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842e5{width:57.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0 solid rgba(0, 0, 0, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842e6{width:48.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0 solid rgba(0, 0, 0, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842e7{width:57.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842e8{width:48.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842e9{width:91.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842ea{width:54.2pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842eb{width:139.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1842ec{width:91.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869ca{width:57.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869cb{width:48.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869cc{width:139.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869cd{width:54.2pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869ce{width:91.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869cf{width:57.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869d0{width:54.2pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869d1{width:48.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869d2{width:139.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0.5pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869d3{width:91.5pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1869d4{width:48.1pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890da{width:54.2pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890db{width:139.1pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890dc{width:57.5pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 0.5pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890dd{width:57.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0 solid rgba(255, 255, 255, 0.00);border-top: 0 solid rgba(255, 255, 255, 0.00);border-left: 0 solid rgba(255, 255, 255, 0.00);border-right: 0 solid rgba(255, 255, 255, 0.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890de{width:91.5pt;background-color:transparent;vertical-align: middle;border-bottom: 0 solid rgba(255, 255, 255, 0.00);border-top: 0 solid rgba(255, 255, 255, 0.00);border-left: 0 solid rgba(255, 255, 255, 0.00);border-right: 0 solid rgba(255, 255, 255, 0.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890df{width:48.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0 solid rgba(255, 255, 255, 0.00);border-top: 0 solid rgba(255, 255, 255, 0.00);border-left: 0 solid rgba(255, 255, 255, 0.00);border-right: 0 solid rgba(255, 255, 255, 0.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890e0{width:54.2pt;background-color:transparent;vertical-align: middle;border-bottom: 0 solid rgba(255, 255, 255, 0.00);border-top: 0 solid rgba(255, 255, 255, 0.00);border-left: 0 solid rgba(255, 255, 255, 0.00);border-right: 0 solid rgba(255, 255, 255, 0.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890e1{width:139.1pt;background-color:transparent;vertical-align: middle;border-bottom: 0 solid rgba(255, 255, 255, 0.00);border-top: 0 solid rgba(255, 255, 255, 0.00);border-left: 0 solid rgba(255, 255, 255, 0.00);border-right: 0 solid rgba(255, 255, 255, 0.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890e2{width:48.1pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 2pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890e3{width:54.2pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 2pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b1890e4{width:91.5pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 2pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b18b7ea{width:139.1pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 2pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-3b18b7eb{width:57.5pt;background-color:transparent;vertical-align: middle;border-bottom: 2pt solid rgba(102, 102, 102, 1.00);border-top: 2pt solid rgba(102, 102, 102, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}</style><table class='cl-3b2402da'><thead><tr style="overflow-wrap:break-word;"><td class="cl-3b18b7ea"><p class="cl-3b17f38c"><span class="cl-3b17a56c">Outcome</span></p></td><td class="cl-3b1890e2"><p class="cl-3b17f38c"><span class="cl-3b17a56c">β</span></p></td><td class="cl-3b1890e2"><p class="cl-3b17f38c"><span class="cl-3b17a56c">SE</span></p></td><td class="cl-3b1890e4"><p class="cl-3b17f38c"><span class="cl-3b17a56c">95% CI</span></p></td><td class="cl-3b1890e3"><p class="cl-3b17f38c"><span class="cl-3b17a56c">z</span></p></td><td class="cl-3b18b7eb"><p class="cl-3b17f38c"><span class="cl-3b17a56d">p</span></p></td></tr></thead><tbody><tr style="overflow-wrap:break-word;"><td class="cl-3b1842e4"><p class="cl-3b17f38d"><span class="cl-3b17a56e">Cognitive Flexibility Acc.</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842e6"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.123</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842e6"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.096</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842e2"><p class="cl-3b17f38c"><span class="cl-3b17a56e">[-0.065, 0.311]</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842e3"><p class="cl-3b17f38c"><span class="cl-3b17a56e">1.281</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842e5"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.2</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td></tr><tr style="overflow-wrap:break-word;"><td class="cl-3b1842eb"><p class="cl-3b17f38d"><span class="cl-3b17a56e">Inhibition Acc.</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842e8"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.177</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842e8"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.057</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842e9"><p class="cl-3b17f38c"><span class="cl-3b17a56e">[0.065, 0.289]</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842ea"><p class="cl-3b17f38c"><span class="cl-3b17a56e">3.098</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842e7"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.002</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td></tr><tr style="overflow-wrap:break-word;"><td class="cl-3b1869cc"><p class="cl-3b17f38d"><span class="cl-3b17a56e">Planning Acc.</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869cb"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.377</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869cb"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.073</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1842ec"><p class="cl-3b17f38c"><span class="cl-3b17a56e">[0.234, 0.519]</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869cd"><p class="cl-3b17f38c"><span class="cl-3b17a56e">5.187</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869ca"><p class="cl-3b17f38c"><span class="cl-3b17a56e">&lt; 0.001</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td></tr><tr style="overflow-wrap:break-word;"><td class="cl-3b1869d2"><p class="cl-3b17f38d"><span class="cl-3b17a56e">Problem Solving</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869d1"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.929</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869d1"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.070</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869ce"><p class="cl-3b17f38c"><span class="cl-3b17a56e">[0.792, 1.066]</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869d0"><p class="cl-3b17f38c"><span class="cl-3b17a56e">13.308</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869cf"><p class="cl-3b17f38c"><span class="cl-3b17a56e">&lt; 0.001</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td></tr><tr style="overflow-wrap:break-word;"><td class="cl-3b1890db"><p class="cl-3b17f38d"><span class="cl-3b17a56e">Working Memory Acc.</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869d4"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.204</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869d4"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.079</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1869d3"><p class="cl-3b17f38c"><span class="cl-3b17a56e">[0.049, 0.358]</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1890da"><p class="cl-3b17f38c"><span class="cl-3b17a56e">2.583</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td><td class="cl-3b1890dc"><p class="cl-3b17f38c"><span class="cl-3b17a56e">0.01</span><span class="cl-3b17a56f"></span><span class="cl-3b17a56f"></span></p></td></tr></tbody><tfoot><tr style="overflow-wrap:break-word;"><td colspan="6"class="cl-3b1890e1"><p class="cl-3b17f38d"><span class="cl-3b17a570"></span><span class="cl-3b17a571">Omnibus Test</span><span class="cl-3b17a572"> </span><span class="cl-3b17a572">χ</span><span class="cl-3b17a573">5</span><span class="cl-3b17a572">=</span><span class="cl-3b17a572"> </span><span class="cl-3b17a572">181.9</span><span class="cl-3b17a572"> </span><span class="cl-3b17a572">p &lt; 0.001</span></p></td></tr><tr style="overflow-wrap:break-word;"><td colspan="6"class="cl-3b1890e1"><p class="cl-3b17f38d"><span class="cl-3b17a570"></span><span class="cl-3b17a572">ρ</span><span class="cl-3b17a573">pre-post</span><span class="cl-3b17a572"> = </span><span class="cl-3b17a572">0.7</span><span class="cl-3b17a572">, </span><span class="cl-3b17a572">ρ</span><span class="cl-3b17a573">agg</span><span class="cl-3b17a572"> = </span><span class="cl-3b17a572">0.5</span><span class="cl-3b17a572">, </span><span class="cl-3b17a572">ρ</span><span class="cl-3b17a573">multi</span><span class="cl-3b17a572"> = </span><span class="cl-3b17a572">0.5</span></p></td></tr></tfoot></table></div></template> <div class="flextable-shadow-host" id="9f6ae949-f41e-4166-a82a-f492b965a845"></div> <script> var dest = document.getElementById("9f6ae949-f41e-4166-a82a-f492b965a845"); var template = document.getElementById("717cf376-f4d1-4cf0-bb14-8b029901ce4c"); var caption = template.content.querySelector("caption"); if(caption) { caption.style.cssText = "display:block;text-align:center;"; var newcapt = document.createElement("p"); newcapt.appendChild(caption) dest.parentNode.insertBefore(newcapt, dest.previousSibling); } var fantome = dest.attachShadow({mode: 'open'}); var templateContent = template.content; fantome.appendChild(templateContent); </script> --- class: clear # The main results... <br/> <br/> <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-21-1.png" width="80%" style="display: block; margin: auto;" /> --- class: inverse, center, middle # Our multiverse results! --- class: clear, full-image .panelset[ <!-- BEGIN PANEL --> .panel[.panel-name[Cognitive Flexibility] <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-22-1.png" width="90%" style="display: block; margin: auto;" /> ] .panel[.panel-name[Inhibition] <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-23-1.png" width="90%" style="display: block; margin: auto;" /> ] .panel[.panel-name[Planning] <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-24-1.png" width="90%" style="display: block; margin: auto;" /> ] .panel[.panel-name[Problem Solving] <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-25-1.png" width="90%" style="display: block; margin: auto;" /> ] .panel[.panel-name[Working Memory] <img src="data:image/png;base64,#psicostat-multivariate-multiverse_files/figure-html/unnamed-chunk-26-1.png" width="90%" style="display: block; margin: auto;" /> ] <!-- END PANEL --> ] --- class: inverse, center, middle # Take Home Message --- # Take Home Message <br/> <br/> -- .blockquote[Data analysis is not **easy** and **cannot be oversimplified**] <br/> <br/> -- .blockquote[Always have to make a choice from **multiverse** of possibilities in terms of statistical models or values to impute] <br/> <br/> -- .blockquote[ Doing one analysis is **FINE**. Doing Multiple analyses is **FUN** (and **useful ** 😉) ] --- # References .bib[ Borenstein, M., L. V. Hedges, J. P. T. Higgins, et al. (2009). _Introduction to Meta-Analysis_. DOI: [10.1002/9780470743386](https://doi.org/10.1002%2F9780470743386). Cai, Z. and X. Fan (2020). "A Comparison of Fixed-Effects and Random-Effects Models for Multivariate Meta-Analysis Using an SEM Approach". En. In: _Multivariate Behav. Res._ 55.6, pp. 839-854. ISSN: 0027-3171, 1532-7906. DOI: [10.1080/00273171.2019.1689348](https://doi.org/10.1080%2F00273171.2019.1689348). Mavridis, D. and G. Salanti (2013). "A practical introduction to multivariate meta-analysis". En. In: _Stat. Methods Med. Res._ 22.2, pp. 133-158. ISSN: 0962-2802, 1477-0334. DOI: [10.1177/0962280211432219](https://doi.org/10.1177%2F0962280211432219). Morris, S. B. (2008). "Estimating Effect Sizes From Pretest-Posttest-Control Group Designs". In: _Organizational Research Methods_ 11.2, pp. 364-386. ISSN: 1094-4281. DOI: [10.1177/1094428106291059](https://doi.org/10.1177%2F1094428106291059). Steegen, S., F. Tuerlinckx, A. Gelman, et al. (2016). "Increasing Transparency Through a Multiverse Analysis". En. In: _Perspect. Psychol. Sci._ 11.5, pp. 702-712. ISSN: 1745-6916, 1745-6924. DOI: [10.1177/1745691616658637](https://doi.org/10.1177%2F1745691616658637). Viechtbauer, W. (2010). _Conducting meta-analyses in R with the metafor package_. DOI: [10.18637/jss.v036.i03](https://doi.org/10.18637%2Fjss.v036.i03). Voracek, M., M. Kossmeier, and U. S. Tran (2019). "Which Data to Meta-Analyze, and How?" In: _Zeitschrift für Psychologie_ 227.1, pp. 64-82. ISSN: 2190-8370. DOI: [10.1027/2151-2604/a000357](https://doi.org/10.1027%2F2151-2604%2Fa000357). Williams, D. R., P. Rast, and P. -. C. Bürkner (2018). "Bayesian meta-analysis with weakly informative prior distributions". DOI: [10.31234/osf.io/7tbrm](https://doi.org/10.31234%2Fosf.io%2F7tbrm). ] --- # Useful links - [Doing Meta-Analysis with R: A Hands-On Guide](https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R): Amazing resource - [Meta-analysis mailing list](https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis): A lot of Q&A - [Metafor](https://www.metafor-project.org/doku.php): Not only the most important package for meta-analysis in R but also a collection of tutorial and practical solutions. - [Handbook of Meta-Analysis - 2020](https://www.routledge.com/Handbook-of-Meta-Analysis/Schmid-Stijnen-White/p/book/9781498703987): The most complete and recent book on meta-analysis --- class: title-slide, center, middle .email[<svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg"> <path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"></path></svg> [filippo.gambarota@phd.unipd.it](mailto:filippo.gambarota@gmail.com)] <br/> <br/> <br/> <br/> .social[<svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path></svg> [@fgambarota](https://twitter.com/fgambarota)] <br/> .social[<svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"></path></svg> [filippogambarota](https://github.com/filippogambarota)]