A gentle intro to Meta-analysis 🚀

# A gentle intro to Meta-analysis 🚀
### Filippo Gambarota
### University of Padova - <span class="citation">@Lab</span> Meeting
### 01/03/2021

---

# Contents

---

# Contents

* ### General Intro
* ### Meta-analysis steps
  * Research question
  * Literature search
  * Extracting data
  * Effect Size computation
  * Effects Size standard error
* ### Fixed and Random-effect models
  * Model fitting
  * Reporting

---

# General Intro

---

# General Intro

* ### A meta-analysis is a **quantitative** way to summarize research result about a specific topic, research question or research area.

* ### Shares the **systematic literature search** with narrative or systematic reviews

* ### Beyond "simply" quantifying a phenomenon, the meta-analysis can be considered as a **radiography** of the published literature

* ### Gives insights about holes in literature, could suggest new research topics or support a new experiment

---

# The Meta perspective

.footnote[Source: [Doing meta-analysis in R](https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/mlma.html)]

---
class: inverse, center, middle

# Meta-analysis steps

---

# Research Question

* ### Not all topics can be managed using a meta-analytic approach
  * **Too broad**: *Efficacy of psychotherapy*
  * **Too narrow**: Not a problem, but probably a little amount of available papers

* ### *Mixing apple and oranges*
  * Does makes sense to pull together these *n* articles?

---

# Literature Search

* ### This is a very crucial and complicated step
  * Keywords
  * Systematic research across *databases*
  * Ideally performed at least by **2 peoples**
  * A lot of guidelines such as the **PRISMA** approach
]

]

---
# Extracting Data

* ### The first step towards the statistical modeling
* ### Extract relevant data in order to:
  * Compute **outcome** (e.g., effect size) measures
  * Select relevant **moderators** for the meta-regression
* ### These moderators need to be theoretically relevant and should be selected before starting the meta-analysis. At the same time exploratory analysis is perfectly fine.

---

# Effect Size Computation

The effect size is number that quantify the **strength of an effect** in the population that is estimated from our sample

---

# Effect Size Computation

### Correlation

`$$r =\frac{ \sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}}$$`

]

### Standardized Mean Difference (e.g., Cohen's `$d$`)

`$$d = \frac{M_1 - M_2}{SD_{pooled}}$$`
]

---

# Effect Size Standard Error

In statistics, every population parameter estimation has an associated uncertainty measure: **Standard Error**. Given that our sample size is always a subset of the population, our estimation is not perfect.

Example: estimation of the males height in Italy

* `$\theta$` = real population height (175cm; but unknown in real research)
* We took a sample `$S$` of size `$n$`, with observed mean `$\bar{X}$` and standard deviation `$s$`
* From statistical theory we know that `$\bar{X}$` is a good estimator of `$\theta$` but with a given uncertainty:

`$$SE = \frac{s}{\sqrt{n}}$$`
---

# Effect Size Standard Error

* The estimated mean is the same but the **uncertainty vary as a function of sample size**
* A study with an **higher sample size** has a **greater estimation precision**

]

.pull-right[
<img src="meta_analysis_presentation_files/figure-html/unnamed-chunk-4-1.png" width="504" style="display: block; margin: auto;" />
]

---

# Quick Recap

### We have:

* Defined our research question
  * Found relevant literature
  * Gathered important data
  * For each study:
    * Outcome measure
    * Measure uncertainty (i.e., Standard Error)

---

# Meta-analysis models

---

# Meta-analysis models

* ### The basic meta-analytic model can be considered a weighted average where all information is combined giving **more weight** to **more precise** studies
* ### From a linear regression point of view, is the simplest model (aka null model) where only the (weighted) mean is estimated
* ### Two main meta-analytic models:
  * Fixed-effect or Equal effect model 
  * Random-effect model

---

# Equal-effect model

---

# Equal-effect model

Model:

`$$y_i = \theta + \epsilon_i$$`

`$$\epsilon_i \sim Normal(0, v_i)$$`

`$$\hat{\theta} = \frac{\sum{w_iy_i}}{\sum{w_i}}$$`
`$$w_i = \frac{1}{v_i}$$`
]

* Weighted average where the weight is the inverse of the study precision (i.e., inverse variance weight)
* More precision = more weight

]

---

# Equal-effect model

### The statistical assumption: the **true** (latent) effect is fixed and observed variability is due to different study precision

* Only 1 parameter to estimate and the associated standard error
* No real between-study variability
* Each study is considered like a replication of the same effect
* Meta-regression is not considered --> no variability to explain

### Often (in psychology) not appropriate!!

---

# Random-effect Model

---

# Random-effect Model

Model:

`$$y_i = \mu + \theta_i + \epsilon_i$$`

`$$\theta_i \sim Normal(0, \tau)$$`

`$$\hat{\mu} = \frac{\sum{w_iy_i}}{\sum{w_i}}$$`
`$$w_i = \frac{1}{v_i + \hat{\tau}}$$`
]

* Weighted average where the weight is the inverse of the study precision (i.e., inverse variance weight) and the between-study heterogeneity
* More precision = more weight BUT extreme studies (very low/high precison) are smoothed in terms of final weight

]

---

# Equal-effect model

### The statistical assumption: the **true** (latent) effect is fixed and observed variability is due to different study precision

* Given that we are estimating a distribution of effect and not a single value:
  * Estimation of `$\mu$`
  * Estimation of `$\tau$`
* `$\tau$` is the between-study heterogeneity that can be explained using moderators

---
# Fixed vs Random-effect model

## Important!

### These two models are not estimating the same quantity from both the statistical and theoretical point of view:
* **Fixed-effect model**: We are estimating the **true underlying effect**
* **Random-effect model**: We are estimating the **true average effect** along the effect variability. The same effect, under specific conditions could be higher or lower/absent

---
class: inverse, center, middle

# Practical Example

---
# Analysis Steps

- ### Effect size computation
- ### Model fitting
  - Model diagnostic
  - Parameters Intepretation
  - Forest Plot
- ### Publication Bias
- ### Meta-regression

---

# Data

<table class="table" style="margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> study </th>
   <th style="text-align:right;"> minutes </th>
   <th style="text-align:right;"> yi </th>
   <th style="text-align:right;"> vi </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Chang et al. (2002) </td>
   <td style="text-align:right;"> 30 </td>
   <td style="text-align:right;"> 0.77 </td>
   <td style="text-align:right;"> 0.072 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Chin (1999) </td>
   <td style="text-align:right;"> 10 </td>
   <td style="text-align:right;"> -0.51 </td>
   <td style="text-align:right;"> 0.049 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Delaney et al. (2002) </td>
   <td style="text-align:right;"> 20 </td>
   <td style="text-align:right;"> 0.21 </td>
   <td style="text-align:right;"> 0.134 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Diego et al. (2002) </td>
   <td style="text-align:right;"> 40 </td>
   <td style="text-align:right;"> 0.46 </td>
   <td style="text-align:right;"> 0.205 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Diego et al. (2001) </td>
   <td style="text-align:right;"> 20 </td>
   <td style="text-align:right;"> 0.56 </td>
   <td style="text-align:right;"> 0.173 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Field et al. (2002) </td>
   <td style="text-align:right;"> 30 </td>
   <td style="text-align:right;"> 0.44 </td>
   <td style="text-align:right;"> 0.205 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Field et al. (1996) </td>
   <td style="text-align:right;"> 15 </td>
   <td style="text-align:right;"> 0.62 </td>
   <td style="text-align:right;"> 0.084 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Groer et al. (1994) </td>
   <td style="text-align:right;"> 10 </td>
   <td style="text-align:right;"> -0.21 </td>
   <td style="text-align:right;"> 0.128 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Hernandez-Reif et al. (1998) </td>
   <td style="text-align:right;"> 45 </td>
   <td style="text-align:right;"> 1.53 </td>
   <td style="text-align:right;"> 0.215 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Hernandez-Reif et al. (2001) </td>
   <td style="text-align:right;"> 30 </td>
   <td style="text-align:right;"> 0.44 </td>
   <td style="text-align:right;"> 0.171 </td>
  </tr>
</tbody>
</table>

---
# Data

---
# Model - Metafor

### Fixed Effect Model

```r
fit.fe <- rma(yi, vi, method = "FE", data = massage)
```

### Random Effect Model

```r
fit.re <- rma(yi, vi, method = "REML", data = massage)
```

---
# Fixed Effect

```
## 
## Fixed-Effects Model (k = 16)
## 
## I^2 (total heterogeneity / total variability):   63.62%
## H^2 (total variability / sampling variability):  2.75
## 
## Test for Heterogeneity:
## Q(df = 15) = 41.2360, p-val = 0.0003
## 
## Model Results:
## 
## estimate      se    zval    pval   ci.lb   ci.ub 
##   0.3542  0.0843  4.2002  <.0001  0.1889  0.5194  *** 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
```

---

# Random Effect

```
## 
## Random-Effects Model (k = 16; tau^2 estimator: REML)
## 
## tau^2 (estimated amount of total heterogeneity): 0.1813 (SE = 0.1126)
## tau (square root of estimated tau^2 value):      0.4258
## I^2 (total heterogeneity / total variability):   60.96%
## H^2 (total variability / sampling variability):  2.56
## 
## Test for Heterogeneity:
## Q(df = 15) = 41.2360, p-val = 0.0003
## 
## Model Results:
## 
## estimate      se    zval    pval   ci.lb   ci.ub 
##   0.4468  0.1397  3.1992  0.0014  0.1731  0.7205  ** 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
```

---
# Forest Plot

---
# Publication Bias

### Published articles are only a (biased) subset of the research conducted on a particular phenomenon
* Only "significant" results are published
* Replication studies are not catchy

### Approaches to Publication Bias
* Graphical evaluation
* Test for the presence of publication bias
* Effect size correction

---
# Publication Bias - Funnel Plot

---
# Publication Bias - Funnel Plot

---
# Meta-regression

---
# Meta-regression

### The impact of **minutes** on the estimated effect size:

```r
fit.meta <- rma(yi, vi, method = "REML", data = massage,
                mods = ~minutes)
```

---
# Meta-regression

```
## 
## Mixed-Effects Model (k = 16; tau^2 estimator: REML)
## 
## tau^2 (estimated amount of residual heterogeneity):     0.0187 (SE = 0.0493)
## tau (square root of estimated tau^2 value):             0.1366
## I^2 (residual heterogeneity / unaccounted variability): 13.63%
## H^2 (unaccounted variability / sampling variability):   1.16
## R^2 (amount of heterogeneity accounted for):            89.71%
## 
## Test for Residual Heterogeneity:
## QE(df = 14) = 13.0270, p-val = 0.5244
## 
## Test of Moderators (coefficient 2):
## QM(df = 1) = 22.9524, p-val < .0001
## 
## Model Results:
## 
##          estimate      se     zval    pval    ci.lb    ci.ub 
## intrcpt   -0.4028  0.1881  -2.1415  0.0322  -0.7715  -0.0341    * 
## minutes    0.0314  0.0066   4.7909  <.0001   0.0186   0.0443  *** 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
```

---
class: inverse, center, middle

# Final considerations

---
# Final considerations

* ### We know the average effect of interest
* ### We know which moderators explain our effect variance
* ### We know which effect we should expect for a future study (i.e., power analysis)
* ### We know the literature structure about our phenomenon:
  * missing studies?
  * future studies?

---

# What else?

---
# What else?

* ### Bayesian meta-analysis
* ### More complex models (multilevel, multivariate)
* ### Effect size calculation from computed statistics, p-values, etc.
* ### Diagnostic: outliers, influential points
* ### Power of meta-analysis

---
class: inverse, center, middle

# Some Materials

---
# Some Materials

* ### [notion.so/filippogambarota/Meta-analysis](https://www.notion.so/filippogambarota/Meta-analysis-5c503baa375a4fdeacd1f9685dc14888)

* ### [Metafor Website](https://www.metafor-project.org/doku.php)

* ### [Wolfgang Viechtbauer Course](https://www.wvbauer.com/doku.php/course_ma)

* ### [Doing Meta-analysis in R](https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/)

---
class: inverse, center, middle

<br/>

### .large[[filippo.gambarota@phd.unipd.it](mailto:filippo.gambarota@phd.unipd.it)]

<br/>

.tiny[Slides made with the [Xaringan](https://github.com/yihui/xaringan) package by [Yihui Xie](https://yihui.name/)]