1.1.1 Goals of this section

• Multiple measures of the same thing or related things as an outcome
  • Possibly over time
• Want the variables separate: Not PCA / FA
• In this section:
  • MANOVA (this week)
  • Repeated measures ANOVA (this week)
  • Mixed models (next week)
  • Mediation (2 weeks)

1.1.2 Goals of this lecture

• Multivariate Analysis of Variance (MANOVA)
  • Outcome is multivariate: Several outcome variables
• Repeated measures ANOVA (RM ANOVA)
  • Univariate: Single outcome variable, measured multiple times
  • Multivariate: Multiple outcome variables
• Punchline: MANOVA is almost never a good choice
  • But multivariate RM ANOVA is a decent approach

2.1.1 Extending ANOVA to multiple outcomes

• Frequently interested in more than 1 outcome at a time
  • Anxiety
    • Test anxiety, minor stressor anxiety, general anxiety
  • Children’s school achievement
    • Reading ability, reasoning ability, math ability
  • Performance on a task
    • Speed and accuracy

2.1.2 Could do GLM on each outcome but…

• …you (often) shouldn’t
  • Inflated type I error due to multiple tests on correlated outcomes
  • Sometimes only the combination of the outcomes shows an effect
  • Ignore relations between DVs

2.1.3 Structure of this section

• Review (univariate) between-subjects ANOVA
  • One outcome
• Extend to multivariate version
  • Multiple related outcomes

2.1.4 Univariate analysis of variance (ANOVA)

• Independent variables (IVs) are categorical groups
  • e.g., treatment and control
• Independent variables are called factors
  • Not to be confused with latent factors
• Single outcome variable (DV)
  • Continuous, normally distributed

2.1.5 ANOVA hypotheses are about the means

• One factor ANOVA
  • \(k\) levels of the independent variable
  • Null hypothesis: All \(k\) group means are equal
    • \(H_0: \mu_1 = \mu_2 = \dots = \mu_k\)

2.1.6 ANOVA hypotheses are about the means

• Two factor ANOVA
  • \(k\) levels of one IV, \(m\) levels of other IV
  • 3 null hypotheses
    • Main effect 1: All \(k\) means across factor 1 are equal
    • Main effect 2: All \(m\) means across factor 2 are equal
    • Interaction: All cell means are equal

2.1.7 Partitioned variation

• Partition the variation in scores into:
  • between-subject portion (group differences, \(SS_{between}\))
  • within-subject portion (error, \(SS_{within}\))
  • \(SS_{total} = SS_{between} + SS_{within}\)
• Calculate based on observed scores, group means, grand mean
  • \(X_{fi}\) = score for subject \(f\) in condition \(i\)
  • \(\bar{T}_{i}\) = mean for scores in condition \(i\)
  • \(\bar{G}\) = grand mean of all scores in the study

2.1.8 Partitioned variation

• Between group variation:

\[SS_{between} = n\Sigma(\bar{T}_i - \bar{G})^2 =\] \[n[(\bar{T}_1 - \bar{G})^2 +(\bar{T}_2 - \bar{G})^2 + \dots + (\bar{T}_k - \bar{G})^2]\]

• Within group variation:

\[SS_{within} = \Sigma(X_{fi} - \bar{T}_i)^2 =\] \[(X_{1i} - \bar{T}_i)^2 +(X_{2i} - \bar{T}_i)^2 + \dots + (X_{ni} - \bar{T}_i)^2\]

2.1.9 Testing the hypothesis

\[MS_{between} = \frac{SS_{between}}{k - 1}\]

\[MS_{within} = \frac{SS_{within}}{k(n−1)}\]

\[F = \frac{MS_{between}}{MS_{within}}\]

• Compare observed \(F\) to critical \(F(k - 1, k(n-1))\)
  • Significant test = at least one of the \(k\) groups is different from the other groups

2.2.1 Multivariate analysis of variance (MANOVA)

• Independent variables are categorical groups
  • e.g., treatment and control
• Independent variables are called factors
  • Not to be confused with latent factors
• Multiple outcome variables
  • \(p\) outcome variables
  • Continuous, normally distributed

2.2.2 What does MANOVA do with all those outcomes?

• MANOVA creates a linear combination of the \(p\) outcome variables
  • Constructed to separate the \(k\) groups as much as possible
  • “Maximally discriminating linear combination”
• Look for group differences on the linear combination
• If you can’t find differences on the maximally discriminating linear combination of all the DVs, then there really really aren’t group differences on the DVs

2.2.3 MANOVA questions

• Do the groups differ at all?
  • On the maximally discriminating linear combination
• If yes, post hoc:
  • Which DVs have groups differences?
  • Which groups differ on those DVs?

2.2.4 Covariation matrix of outcomes \(\textbf{P}\)

• Covariation matrix of the \(p\) DVs: \(p \times p\) matrix
  • Multivariate extension of \(SS_{total}\)
• Just like ANOVA: Partitions into between (\(\textbf{H}\)) and within (\(\textbf{E}\))

\[\textbf{P} = \begin{bmatrix} SS_1 & SP_{12} & \cdots & SP_{1p}\\ SP_{21} & SS_2 & \cdots & SP_{2p}\\ \vdots & \vdots & \ddots & \vdots\\ SP_{p1} & SP_{p2} & \cdots & SS_p\\ \end{bmatrix}\]

2.2.5 Hypothesis matrix \(\textbf{H}\)

• Multivariate extension of \(SS_{between}\): \(p \times p\) matrix
  • Diagonal: between-group variation of each DV
  • Off-diagonal: covariation between means for pairs of DVs

\[\textbf{H} = \begin{bmatrix} SS_{H,1} & SP_{H,12} & \cdots & SP_{H,1p}\\ SP_{H,21} & SS_{H,2} & \cdots & SP_{H,2p}\\ \vdots & \vdots & \ddots & \vdots\\ SP_{H,p1} & SP_{H,p2} & \cdots & SS_{H,p}\\ \end{bmatrix}\]

2.2.6 Aside: H matrix for two-factor MANOVA

• For a one-factor MANOVA, there is a single \(\textbf{H}\) matrix
• For a two-factor MANOVA, there is a single \(\textbf{H}\) matrix
  • BUT it can be further partitioned into 3 matrices reflecting:
    • Main effect 1
    • Main effect 2
    • Interaction effect

2.2.7 Error matrix E

• Multivariate extension of \(SS_{within}\): \(p \times p\) matrix
  • Diagonal: within-group variation of each DV, added across \(k\) grp
  • Off-diagonal: error covariation, added across \(k\) groups
• No between-group information in this matrix

\[\textbf{E} = \begin{bmatrix} SS_{E,1} & SP_{E,12} & \cdots & SP_{E,1p}\\ SP_{E,21} & SS_{E,2} & \cdots & SP_{E,2p}\\ \vdots & \vdots & \ddots & \vdots\\ SP_{E,p1} & SP_{E,p2} & \cdots & SS_{E,p}\\ \end{bmatrix}\]

2.2.8 Partitioned variation

• ANOVA
  • \(SS_{total} = SS_{between} + SS_{within}\)
• MANOVA
  • Total variation = between-group variation + within-group variation
  • One factor: \(\textbf{P} = \textbf{H} + \textbf{E}\)
  • Two factor: \(\textbf{P} = \textbf{H}_{factor1} + \textbf{H}_{factor2} + \textbf{H}_{factor1*factor 2} + \textbf{E}\)

2.2.9 Multivariate hypothesis tests (omnibus)

• ANOVA
  • Divide \(SS\) by their degrees of freedom to produce \(MS\) (variances)
  • \(F\)-statistic is ratio of \(MS\)s (variances)
• MANOVA
  • Use matrix equivalent of variance: Determinant
    • Determinant is “generalized variance” for a matrix
  • Create analogues to \(F\)-statistics
  • Unfortunately, it’s not straight-forward

2.2.10 Multivariate hypothesis tests

• Four commonly used multivariate tests
  • Different ratio of determinants or eigenvalues
• Wilks’ lambda: within / total
• Pillai’s trace: between / total
• Hotelling’s trace: between / within
• Roy’s largest characteristic root: between / total

2.2.11 Wilks’ lambda

• \(\Lambda = \dfrac{\vert \textbf{E} \vert}{\vert \textbf{H} + \textbf{E} \vert}=\dfrac{\vert \textbf{E} \vert}{\vert \textbf{P} \vert}\)
  • where \(\vert \textbf{E} \vert\) is the determinant of \(\textbf{E}\)
• \(H_0\): no between-group variation, so \(\textbf{H}\) is all zeroes and ratio is 1
  • As group differences increase, \(\Lambda \rightarrow 0\)
• Effect size = eta squared = \(\eta^2 = 1 - \Lambda\)
  • \(\eta^2\) = variance accounted for by the best linear combination of DVs

2.2.12 Pillai’s trace

• Pillai’s trace = \(trace\left[\textbf{H}(\textbf{H} + \textbf{E})^{-1}\right]\)
  • where the trace of a matrix is the sum of the diagonal elements
• Conceptually:
  • Matrix representing proportion of variation that is between-group
  • Sum of eigenvalues from that matrix

2.2.13 Hotelling’s trace

• Hotelling’s trace = \(trace\left[\textbf{H}(\textbf{E})^{-1}\right]\)
  • where the trace of a matrix is the sum of the diagonal elements
• Conceptually:
  • Matrix representing ratio of between- to within-group variation
  • Sum of eigenvalues from that matrix

2.2.14 Roy’s largest characteristic root

• Roy’s greatest characteristic root = first eigenvalue of \(\textbf{H}(\textbf{H} + \textbf{E})^{-1}\)

• Conceptually:

  • Matrix representing proportion of variation that is between-group
  • First eigenvalue from that matrix

2.2.15 Summary of multivariate tests

These tests are similar, but they differ in terms of power and robustness to violations of assumptions

2.2.16 Assumptions of MANOVA

• GLM: Multivariate normality of outcomes, linearity, etc

• “Homogeneity of variance-covariance matrices”

  • Error matrix is same in all groups and \(\textbf{E}\) is average
  • Multivariate extension of homogeneity of variance assumption

• Box’s M test to test this assumption

  • Significant test means that assumption is violated
  • Sensitive: use p<.001, ignore unless \(n\)s very different across groups

2.2.17 Which test should I use???

• One factor MANOVA with k = 2 groups: All tests are identical
• Recommended: Pillai’s trace
  • Robust to assumptions, powerful when DVs not highly corr
• Recommended: Wilks’ lambda
  • Good power, relatively robust when assumptions probably met
• Maybe use: Roy’s greatest characteristic root
  • Powerful when DVs highly corr, not robust to assumptions
• Not recommended: Hotelling’s trace
  • OK when sample size is very large

2.3.1 MANOVA

• Extends ANOVA to multiple outcomes
  • Many omnibus test options
  • Many follow-up options
  • Maximally discriminating linear combination?
  • Missing data, ANOVA framework only, time
• Quantitude says MANOVA must die

2.3.2 MANOVA questions

• Do the groups differ at all (on max discriminating linear comb.)?
  • This is what Pillai’s trace, etc are testing
• If yes, post hoc:
  • Which DVs have groups differences?
  • Which groups differ on those DVs?
  • Enders, C. K. (2003). Performing multivariate group comparisons following a statistically significant MANOVA. Measurement and Evaluation in Counseling and Development, 36, 40-56.

2.3.3 When to use MANOVA?

• DVs are highly negatively correlated
  • Time to complete a task and number of errors on task
• DVs are all moderately correlated in either direction
  • Around \(\pm 0.6\) correlation
  • Not really high enough to support a latent factor
  • Repeated measures

2.3.4 When not to use MANOVA?

• DVs are not really correlated
  • MANOVA is unnecessarily complicated and wasteful
  • You don’t gain anything by analyzing them together
    
• DVs are all highly positively correlated
  • MANOVA is unnecessarily complicated and wasteful
  • The variables are all basically the same thing

2.3.5 Alternatives to MANOVA

• Repeated-measures DVs:
  • Repeated measures ANOVA
  • Mixed / multilevel / hierarchical linear models
  • Latent growth models
• Separate univariate ANOVAs: esp uncorrelated DVs
• SEM / path model with multiple DVs
• Latent factor: esp highly correlated DVs

3.1.1 Between-subjects ANOVA

• Different subjects in each condition or cell of the design
  • 2 dimensions: subjects and variables

3.1.2 Between-subjects ANOVA: Partitioning

• Partition the variation in scores into:
  • between-subject portion (group differences, \(SS_{between}\))
  • within-subject portion (error, \(SS_{within}\))
  • \(SS_{total} = SS_{between} + SS_{within}\)

3.1.3 Repeated-measures

1. Measure the same DV over time
   • e.g., anxiety level at 1 wk intervals after starting medication
2. Measure the same DV in each of a set of related conditions
   • e.g., anxiety level after CBT, after medication, etc.

• Multiple outcome measures that are related
  • Measured on the same person (not independent)
  • MANOVA: related dependent variables

3.1.4 Repeated-measures ANOVA

• Subjects are repeatedly measured / same subject in all conditions
  • 3 dimensions: subjects, variables (\(Y_1\)), treatment or time (\(T\))

3.1.5 Two ways to do repeated-measures ANOVA

• Univariate:
  • Standard repeated measures ANOVA
  • Treats the outcome as one variable that is measured repeatedly
• Multivariate:
  • Treats the outcome as a multivariate outcome
    • Single outcome made up of several (related) variables
      • Sound familiar?

3.1.6 Univariate: \(n\) subjects, \(k\) repeated measures

• Single outcome variable \(Y\)
  • “Univariate”
• \(T\) (time or treatment) is a predictor
  • Specific levels: \(1, 2, \dots, k\)
• Also called “tall” or “stacked” data format
  • Used in mixed models (next week)

3.1.7 Univariate: \(n\) subjects, \(k\) repeated measures

3.1.8 Multivariate: \(n\) subjects, \(k\) repeated measures

• Several related outcome variables \(Y\)
  • “Multivariate”
• \(T\) (time or treatment) is not an explicit predictor
  • Treated like waves
• Also called “wide” data format
  • Used in MANOVA

3.1.9 Multivariate: \(n\) subjects, \(k\) repeated measures

3.2.1 Univariate approach to repeated measures

• Partition variation in scores into:
  • Between-subject variation
  • Within-subject variation, which is further partitioned into:
    • Treatment (or time) effects for individuals
    • Residual or random error

3.2.2 Univariate approach to repeated measures

• \(Y_{ij}\) = score for person \(i\) at time or treatment \(j\)
• \(\bar{T}_j\) = mean score for treatment or time \(j\)
  • Up to \(k\) treatments or times
• \(\bar{P}_i\) = mean score for person \(i\)
  • Up to \(n\) subjects
• \(\bar{G}\) = grand mean of all scores

3.2.3 Between-subjects variation

• Individual subjects’ variation around the grand mean

\[SS_{between\;subject} = k \sum_{i=1}^{n} (\overline{P}_i - \overline{G}) ^2\]

\[=k [ (\overline{P}_1 - \overline{G}) ^2 + (\overline{P}_2 - \overline{G}) ^2 + \dots + (\overline{P}_k - \overline{G}) ^2]\]

• Similar to between-groups variation in ANOVA, but no groups here
  • People are “groups”

3.2.4 Within-subjects variation

• Individual subjects’ variation around their mean

• For person \(i\):

\[SS_{within\;person\;i} = \sum_{j=1}^{k} (Y_{ij} - \overline{P}_i) ^2\]

\[(Y_{i1} - \overline{P}_i) ^2 + (Y_{i2} - \overline{P}_i) ^2 + \dots + (Y_{ik} - \overline{P}_i) ^2\]

• Add up across all \(n\) subjects: \(SS_{within\;subject} = \sum_{i=1}^{n} \sum_{j=1}^{k} (Y_{ij} - \overline{P}_i) ^2\)

3.2.5 Within-subjects variation

• Within-subjects variation = time (or treatment) + residual

• Time variation = timepoint mean variation around grand mean

\[SS_{time} = n \sum_{j=1}^{k} (\overline{T}_j - \overline{G}) ^2\]

\[=n [ (\overline{T}_1 - \overline{G}) ^2 + (\overline{T}_2 - \overline{G}) ^2 + \dots + (\overline{T}_k - \overline{G}) ^2]\]

3.2.6 Within-subjects variation

• Within-subjects variation = time (or treatment) + residual

• Residual variation = any remaining variation

\[SS_{residual} = SS_{time \times subject} = SS_{within\;subject} - SS_{time}\]

3.2.8 Mixed effects ANOVA

• Between-subjects + within-subjects = “mixed ANOVA”
  • Unfortunate: too easy to confuse with “mixed models”
    • Also have several other names: Next week
• You can have BOTH within and between subjects factors in ANOVA
  • e.g., group (between) and time (within)
• Also look at the interaction
  • Does time effect vary across groups? Or vice versa?

3.2.9 Assumptions of univariate RM ANOVA

• About the covariance matrix of the outcomes

\[\textbf{S}_{YY}=\begin{bmatrix}\sigma_1^2 & \sigma_{12} & \sigma_{13} & \sigma_{14} \\ & \sigma_2^2 & \sigma_{23} & \sigma_{24} \\ & & \sigma_3^2 & \sigma_{34} \\ & & & \sigma_4^2 \end{bmatrix}\]

• \(\sigma_1^2\) = variance of outcome at time 1 / treatment 1
• \(\sigma_{12}\) = covariance between outcome at time /treatment 1 and outcome at time / treatment 2

3.2.10 Compound symmetry and sphericity

• Compound symmetry of the covariance matrix of outcomes
  • Homogeneity of variances (i.e., variances are all the same):
    • \(\sigma_1^2 = \sigma_2^2 = \sigma_3^2 = \sigma_4^2\)
  • Homogeneity of covariances (i.e., covariances are all the same):
    • \(\sigma_{12} = \sigma_{13} = \sigma_{14} = \sigma_{23}= \sigma_{24}= \sigma_{34}\)
• Actual assumption: Sphericity
  • Compound symmetry holds for differences between pairs of scores
  • Slightly weaker assumption

3.2.11 Plausibility of sphericity assumption

• \(T_1\) through \(T_k\) are different trials or conditions in a single session
  • Sphericity may be plausible
• \(T_1\) through \(T_k\) are different time points
  • Say, 9th, 10th, 11th, and 12th grades
  • Probably expect T1 and T2 to be more alike that T1 and T4
  • Sphericity is probably not very plausible

3.2.12 Violations of Assumptions

• Even if sphericity is plausible, it still may be violated
  • Very small violations can greatly increase type I error rate
• How to deal with violation of sphericity?
  • Adjust for violations of sphericity to return alpha to .05
  • Use multivariate test of repeated measures (next section)

3.2.13 Adjusting for sphericity violations

• Lower bound correction: Most conservative
  • Ignore repeated measures, treat as between subjects
    
  • Use critical \(F\)(1, n − 1)
• Greenhouse-Geisser: Middle of the road
  • \(\hat{\epsilon}\) ranges from \(\frac{1}{k-1}\) (severe violation) to 1 (sphericity)
  • Multiply degrees of freedom by \(\hat{\epsilon}\)
• Huynh-Feldt: Least conservative, smallest adjustment
  • Multiply degrees of freedom by \(\tilde{\epsilon}\) (function of \(\hat{\epsilon}\))

3.3.1 Mutlivariate approach to repeated measures

• Multivariate extension of paired t-test
• Basically a MANOVA on specific set of difference scores
  • Multivariate tests (i.e., Wilks’ lambda) as in MANOVA

3.3.2 Vector of differences

• \(k − 1\) differences between combinations of \(k\) repeated scores
  • Must be linearly independent
  • Most common: Differences between adjacent pairs of means
• For a single subject \(i\):

\(\underline{Y}'_{id} = \begin{bmatrix} d_{i1}\\ d_{i2}\\ d_{i3}\\ \vdots\\ d_{i,k-1}\\ \end{bmatrix} = \begin{bmatrix} Y_{i1} - Y_{i2}\\ Y_{i2} - Y_{i3}\\ Y_{i3} - Y_{i4}\\ \vdots\\ Y_{i,k-1} - Y_{ik}\\ \end{bmatrix}\)

3.3.3 Matrix of difference scores

• \(n × (k − 1)\) matrix of difference scores is matrix of outcomes
  • Rows are subjects, columns are difference scores
  • \(k\) repeated measures: \(k-1\) difference scores

\(\textbf{Y}_d = \begin{bmatrix}d_{11} & d_{12} & \dots & d_{1,k-1}\\ d_{21} & d_{22} & \dots & d_{2,k-1}\\ \vdots & \vdots & \ddots & \vdots\\ d_{n1} & d_{n2} & \dots & d_{n,k-1}\\ \end{bmatrix}\)

3.3.4 Covariance matrix of differences

• \((k-1) \times (k-1)\) covariance matrix of differences
  • \(s_{d_1}^2\) is the variance of the (\(T1 - T2\)) scores across \(n\) subjects
  • \(s_{d_1d_2}\) is the covariance between (\(T1 - T2\)) and (\(T2 - T3\))
  • Unlike univariate test, no assumptions about this matrix
• For 4 timepoints, this is a \(3 \times 3\) matrix: \[\textbf{S}_{d}=\begin{bmatrix}s_{d_1}^2 & s_{d_1d_2} & s_{d_1d_3} \\ & s_{d_2}^2 & s_{d_2d_3} \\ & & s_{d_3}^2 \end{bmatrix}\]

3.3.5 Null hypothesis

• \(H_0\): \(k − 1\) vectors of mean differences are simultaneously
  • All equal to each other AND all equal to 0
• NS test = no differences over time
  • All mean differences are 0
  • No adjacent differences are different from one another
• Significant test = differences over time
  • Some of the mean differences are not 0
  • Some adjacent differences are different from one another

3.3.6 Multivariate hypothesis tests

• Perform a MANOVA on the difference score matrix
  • Multivariate hypothesis tests:
    • Wilks’ lambda
    • Pillai’s trace
    • Hotelling’s trace
    • Roy’s largest characteristic root

3.4.1 Summary

• Univariate RM ANOVA
  • Single, repeatedly measured outcome
  • Sphericity assumption
• Multivariate RM ANOVA
  • Multiple, related outcomes
  • No sphericity assumption

3.4.2 Comparison

• Univariate approach: \(F(k - 1, (n - 1)(k - 1))\)
  • Assumptions about covariance matrix (sphericity)
    • But can adjust if assumptions not met
  • Missing data results in loss of entire subject
• Multivariate approach: \(F(k - 1, n - k + 1)\)
  • No assumptions about structure of covariance matrix
    • (except that \(n \ge k\) so it is invertable)
  • Missing data results in loss of entire subject

3.4.3 Recommendations: univariate vs. multivariate

• Univariate is preferred with small sample sizes
  • Sphericity holds (rare): More powerful, simpler, correct \(\alpha\)
  • ALWAYS use univariate (with correction) if \(n < k\)
• Multivariate is preferred with large sample sizes
  • If sphericity doesn’t hold (common): correct \(\alpha\)
  • Do not use unless \(n \ge k\)
    • With BS factors: \(n\) in each BS group needs to be \(\ge k\)

4.1.1 Summary of this week

• MANOVA is a way to analyze multiple outcomes in one model
  • Almost never a good choice
  • Limited utility for repeated measures
• RM ANOVA has univariate and multivariate versions
  • Univariate has some easily violated assumptions
  • Multivariate is good but still limited
  • Missing data, ANOVA framework only, time

4.1.2 Next few weeks

• RM ANOVA (both) have shortcomings
  • Best for short-term or single-session studies
  • Does not capture the TIME aspect of longitudinal data
  • Requires same # of repeated measures for each subject
  • Not informative about individual growth
  • Focus on average differences over time and group differences
  • ANOVA framework, so only categorical predictors
• Mixed models, latent growth models solve many of these issues