Familiarize you with classic and modern multivariate statistics
Make sure that you understand how to conduct these analyses and interpret the results
Prepare you for further study in applied statistics
Give you enough background to understand current applied statistics research
Introduce matrix algebra
Review basic descriptive statistics
Convert expressions for those statistics into matrix format
Variate \(\approx\) Variable
Univariate = one variable
Mean, variance
Bivariate = two variables
Correlation between two variables
Multivariate = multiple variables
How several outcomes are related to one another or to predictor(s)
Multivariate data
Multivariate statistics
| Technique | Predictor (IV) | Outcome (DV) |
|---|---|---|
| \(t\) test | 1 discrete, 2 levels | 1 |
| One-way ANOVA | 1 discrete, >2 levels | 1 |
| Factorial ANOVA | \(\ge\) 2 discrete | 1 |
| Correlation | 1 continuous | 1 |
| Regression | \(\ge\) 2 continuous | 1 |
| ANCOVA | Discrete, continuous | 1 |
From Harris, R.N. (1985). A Primer on Multivariate Statistics
| Technique | Predictor (IV) | Outcome (DV) |
|---|---|---|
| Discriminant analysis | 1 discrete, 2 levels | \(\ge\) 2 |
| MANOVA | 1 discrete, >2 levels | \(\ge\) 2 |
| Canonical correlation | \(\ge\) 2 continuous | \(\ge\) 2 |
| PCA | \(\ge\) 2 continuous | |
| FA | \(\ge\) 2 continuous |
From Harris, R.N. (1985). A Primer on Multivariate Statistics
Models for one outcome variable
Models for multiple indicators of a construct
Models for multiple outcomes
Why do we start here?
Statistics is applied math
Matrices help us organize a lot of numbers
Matrix algebra lets us manipulate a lot of numbers at once
Matrix algebra is the language of statistics
A scalar is an “ordinary” number
The algebra of scalars is arithmetic
Doubly ordered arrangement of scalars
Denoted by capital letters (often bold capital letters)
The order (size) of the matrix is (# rows \(\times\) # columns)
The elements in the matrix are identified by subscripts
\[\begin{matrix} \textbf{X} \\ (4,3) \end{matrix} = \begin{bmatrix} 2 & 6 & 5 \\ 8 & 1 & 4 \\ 9 & 3 & 6 \\ 2 & 0 & 5 \end{bmatrix}\]
\[\begin{matrix} \textbf{A} \\ (2,5) \end{matrix} = \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} & a_{15} \\ a_{21} & a_{22} & a_{23} & a_{24} & a_{25} \end{bmatrix}\]
A row vector is a matrix of order \(1 \times q\)
A column vector is a matrix of order \(p \times 1\)
Denoted by lower case, underlined letters
\(\begin{matrix} \underline{x} \\ (1,q) \end{matrix} = \begin{bmatrix} x_1 & x_2 & \cdots & x_q \end{bmatrix}\)
\(\begin{matrix} \underline{y} \\ (p,1) \end{matrix} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_p \end{bmatrix}\)
Matrix algebra is the set of rules for performing mathematical operations on matrices (and vectors)
\(\begin{matrix} \textbf{A} \\ (2,3) \end{matrix} = \begin{bmatrix} 6 & 2 & 4 \\ 8 & 1 & 0 \end{bmatrix}\)
\(\begin{matrix} \textbf{A'} \\ (3,2) \end{matrix} = \begin{matrix} \textbf{A}^{T} \\ (3,2) \end{matrix} = \begin{bmatrix} 6 & 8 \\ 2 & 1\\ 4 & 0 \end{bmatrix}\)
\(\begin{matrix} \underline{x} \\ (4,1) \end{matrix} = \begin{bmatrix} 2\\ 8\\ 9\\ 2 \end{bmatrix}\)
\(\begin{matrix} \underline{x}' \\ (1,4) \end{matrix} = \begin{matrix} \underline{x}^{T} \\ (1,4) \end{matrix} = \begin{bmatrix} 2 & 8 & 9 & 2 \end{bmatrix}\)
Matrices must be of the same order to be conformable for addition
\(\begin{bmatrix} 3 & 1 & 5 \\ 2 & 4 & 6 \end{bmatrix}\) + \(\begin{bmatrix} 2 & 4 & 7 \\ 8 & 11 & 4 \end{bmatrix}\) = \(\begin{bmatrix} 5 & 5 & 12 \\ 10 & 15 & 10 \end{bmatrix}\)
Matrices must be of the same order to be conformable for subtraction
\(\begin{bmatrix} 3 & 1 & 5 \\ 2 & 4 & 6 \end{bmatrix}\) – \(\begin{bmatrix} 2 & 4 & 7 \\ 8 & 11 & 4 \end{bmatrix}\) = \(\begin{bmatrix} 1 & -3 & -2 \\ -6 & -7 & 2 \end{bmatrix}\)
Rules for matrix multiplication
\(\color{OrangeRed}{\begin{matrix} \textbf{A} \\ (2,3) \end{matrix}}\) \(\color{blue}{\begin{matrix} \textbf{B} \\ (3,2) \end{matrix}}\) = \(\begin{matrix}\textbf{C} \\ (2,2) \end{matrix}\)
\(\color{OrangeRed}{\begin{bmatrix} 2 & 4 & 1 \\ 3 & 0 & 4 \end{bmatrix}}\) \(\color{blue}{\begin{bmatrix} 1 & 3 \\ 2 & 0 \\ 4 & 2 \end{bmatrix}}\) = \(\begin{bmatrix} 14 & 8 \\ 19 & 17 \end{bmatrix}\)
Row 1, Column 1: \(({\color{OrangeRed}2} \times {\color{blue}1}) + ({\color{OrangeRed}4} \times {\color{blue}2}) + ({\color{OrangeRed}1} \times {\color{blue}4}) = 14\)
Row 1, Column 2: \(({\color{OrangeRed}2} \times {\color{blue}3}) + ({\color{OrangeRed}4} \times {\color{blue}0}) + ({\color{OrangeRed}1} \times {\color{blue}2}) = 8\)
Row 2, Column 1: \(({\color{OrangeRed}3} \times {\color{blue}1}) + ({\color{OrangeRed}0} \times {\color{blue}2}) + ({\color{OrangeRed}4} \times {\color{blue}4}) = 19\)
Row 2, Column 2: \(({\color{OrangeRed}3} \times {\color{blue}3}) + ({\color{OrangeRed}0} \times {\color{blue}0}) + ({\color{OrangeRed}4} \times {\color{blue}2}) = 17\)
\(\color{OrangeRed}{\begin{matrix} \textbf{A} \\ (2,3) \end{matrix}}\) \(\color{blue}{\begin{matrix} \underline{b} \\ (3,1) \end{matrix}}\) = \(\begin{matrix} \underline{c} \\ (2,1) \end{matrix}\)
\(\color{OrangeRed}{\begin{bmatrix} 2 & 4 & 1 \\ 3 & 0 & 4 \end{bmatrix}}\) \(\color{blue}{\begin{bmatrix} 1 \\ 2 \\ 4 \end{bmatrix}}\) = \(\begin{bmatrix} 14 \\ 19 \end{bmatrix}\)
Row 1, Column 1: \(({\color{OrangeRed}2} \times {\color{blue}1}) + ({\color{OrangeRed}4} \times {\color{blue}2}) + ({\color{OrangeRed}1} \times {\color{blue}4}) = 14\)
Row 2, Column 1: \(({\color{OrangeRed}3} \times {\color{blue}1}) + ({\color{OrangeRed}0} \times {\color{blue}2}) + ({\color{OrangeRed}4} \times {\color{blue}4}) = 19\)
\(\color{OrangeRed}{\begin{matrix} \textbf{X} \\ (5,3) \end{matrix}}\) \(\color{blue}{\begin{matrix} \underline{b} \\ (3,1) \end{matrix}}\) = \(\begin{matrix} \underline{c} \\ (5,1) \end{matrix}\)
\(\color{OrangeRed}{\begin{bmatrix} X_{11} & X_{12} & X_{13} \\ X_{21} & X_{22} & X_{23} \\ X_{31} & X_{32} & X_{33} \\ X_{41} & X_{42} & X_{43} \\ X_{51} & X_{52} & X_{53} \end{bmatrix}}\) \(\color{blue}{\begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix}}\) = \(\begin{bmatrix}\color{blue}{b_1}\color{OrangeRed}{X_{11}} + \color{blue}{b_2}\color{OrangeRed}{X_{12}} + \color{blue}{b_3}\color{OrangeRed}{X_{13}} \\ \color{blue}{b_1}\color{OrangeRed}{X_{21}} + \color{blue}{b_2}\color{OrangeRed}{X_{22}} + \color{blue}{b_3}\color{OrangeRed}{X_{23}}\\ \color{blue}{b_1}\color{OrangeRed}{X_{31}} + \color{blue}{b_2}\color{OrangeRed}{X_{32}} + \color{blue}{b_3}\color{OrangeRed}{X_{33}} \\ \color{blue}{b_1}\color{OrangeRed}{X_{41}} + \color{blue}{b_2}\color{OrangeRed}{X_{42}} + \color{blue}{b_3}\color{OrangeRed}{X_{43}}\\ \color{blue}{b_1}\color{OrangeRed}{X_{51}} + \color{blue}{b_2}\color{OrangeRed}{X_{52}} + \color{blue}{b_3}\color{OrangeRed}{X_{53}} \end{bmatrix}\)
\(\color{OrangeRed}{\begin{matrix} \underline{a'} \\ (1,3) \end{matrix}}\) \(\color{blue}{\begin{matrix} \underline{b} \\ (3,1) \end{matrix}}\) = \(\begin{matrix} \underline{c} \\ (1,1) \end{matrix}\)
\(\color{OrangeRed}{\begin{bmatrix} 3 & 1 & 5 \end{bmatrix}}\) \(\color{blue}{\begin{bmatrix} 2 \\ 4 \\ 9 \end{bmatrix}}\) = \(\begin{bmatrix} 55 \end{bmatrix}\)
\(({\color{OrangeRed}3} \times {\color{blue}2}) + ({\color{OrangeRed}1} \times {\color{blue}4}) + ({\color{OrangeRed}5} \times {\color{blue}9}) = 55\)
\(\color{OrangeRed}{\begin{matrix} \underline{b} \\ (3,1) \end{matrix}}\) \(\color{blue}{\begin{matrix} \underline{a'} \\ (1,3) \end{matrix}}\) = \(\begin{matrix}\textbf{C} \\ (3,3) \end{matrix}\)
\(\color{OrangeRed}{\begin{bmatrix} 2 \\ 4 \\ 9 \end{bmatrix}}\) \(\color{blue}{\begin{bmatrix} 3 & 1 & 5 \end{bmatrix}}\) = \(\begin{bmatrix} 6 & 2 & 10 \\ 12 & 4 & 20 \\ 27 & 9 & 45 \end{bmatrix}\)
Row 1, Column 1: \(({\color{OrangeRed}2} \times {\color{blue}3}) = 6\)
Row 1, Column 2: \(({\color{OrangeRed}2} \times {\color{blue}1}) = 2\)
Row 1, Column 3: \(({\color{OrangeRed}2} \times {\color{blue}5}) = 10\)
Row 2, Column 1: \(({\color{OrangeRed}4} \times {\color{blue}3}) = 12\)
Row 2, Column 2: \(({\color{OrangeRed}4} \times {\color{blue}1}) = 4\)
Row 2, Column 3: \(({\color{OrangeRed}4} \times {\color{blue}5}) = 20\)
Row 3, Column 1: \(({\color{OrangeRed}9} \times {\color{blue}3}) = 27\)
Row 3, Column 2: \(({\color{OrangeRed}9} \times {\color{blue}1}) = 9\)
Row 3, Column 3: \(({\color{OrangeRed}9} \times {\color{blue}5}) = 45\)
Multiply every element in the matrix by the scalar
\(3 \times \begin{bmatrix} 3 & 2 & 4 \\ 1 & 2 & 3 \\ 8 & 1 & 1 \end{bmatrix} = \begin{bmatrix} 9 & 6 & 12 \\ 3 & 6 & 9 \\ 24 & 3 & 3 \end{bmatrix}\)
\((\textbf{AB})\textbf{C} = \textbf{A}(\textbf{BC})\)
With respect to addition and subtraction:
\(\textbf{A}(\textbf{B} + \textbf{C}) = \textbf{AB} + \textbf{AC}\)
\(\textbf{AB} \ne \textbf{BA}\)
If \(\textbf{D} = \textbf{A} \textbf{B}\), then \(\textbf{D}' = \textbf{B}' \textbf{A}'\)
Division is not defined for matrices
Instead of dividing, we multiply by the inverse
We do the same in regular arithmetic:
Calculating the inverse of a matrix is complicated (more later)
Multiplying the matrix by its inverse results in the identity matrix:
\(\textbf{A A}^{-1} = \textbf{A}^{-1} \textbf{A} = \textbf{I}\)
The identity matrix (\(\textbf{I}\)) is a special matrix that looks like:
\(\begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 1 \end{bmatrix}\)
\(1\)s on the main diagonal, \(0\)s elsewhere
Multiplying a matrix by \(\textbf{I}\) is like multiplying it by \(1\)
\(\textbf{A} = \color{OrangeRed}{\begin{bmatrix} 1 & 2 & 3 \\ 1 & 3 & 2 \\ 2 & 4 & 1 \end{bmatrix}}\) \(\textbf{A}^{-1} = \color{blue}{\begin{bmatrix} 1 & -2 & 1 \\ -\frac{3}{5} & 1 & -\frac{1}{5} \\ \frac{2}{5} & 0 & -\frac{1}{5} \end{bmatrix}}\)
Row 1, Column 1: \(1 = ({\color{OrangeRed}1} \times {\color{blue}1}) + ({\color{OrangeRed}2} \times {\color{blue}{-\frac{3}{5}}}) + ({\color{OrangeRed}3} \times {\color{blue}{\frac{2}{5}}})\)
Row 1, Column 2: \(0 = ({\color{OrangeRed}1} \times {\color{blue}{-2}}) + ({\color{OrangeRed}2} \times {\color{blue}1}) + ({\color{OrangeRed}3} \times {\color{blue}0})\)
Row 1, Column 3: \(0 = ({\color{OrangeRed}1} \times {\color{blue}1}) + ({\color{OrangeRed}2} \times {\color{blue}{-\frac{1}{5}}}) + ({\color{OrangeRed}3} \times {\color{blue}{-\frac{1}{5}}})\)
Row 2, Column 1: \(0 = ({\color{OrangeRed}1} \times {\color{blue}1}) + ({\color{OrangeRed}3} \times {\color{blue}{-\frac{3}{5}}}) + ({\color{OrangeRed}2} \times {\color{blue}{\frac{2}{5}}})\)
Row 2, Column 2: \(1 = ({\color{OrangeRed}1} \times {\color{blue}{-2}}) + ({\color{OrangeRed}3} \times {\color{blue}1}) + ({\color{OrangeRed}2} \times {\color{blue}0})\)
Row 2, Column 3: \(0 = ({\color{OrangeRed}1} \times {\color{blue}1}) + ({\color{OrangeRed}3} \times {\color{blue}{-\frac{1}{5}}}) + ({\color{OrangeRed}2} \times {\color{blue}{-\frac{1}{5}}})\)
Row 3, Column 1: \(0 = ({\color{OrangeRed}2} \times {\color{blue}1}) + ({\color{OrangeRed}4} \times {\color{blue}{-\frac{3}{5}}}) + ({\color{OrangeRed}1} \times {\color{blue}{\frac{2}{5}}})\)
Row 3, Column 2: \(0 = ({\color{OrangeRed}2} \times {\color{blue}{-2}}) + ({\color{OrangeRed}4} \times {\color{blue}1}) + ({\color{OrangeRed}1} \times {\color{blue}0})\)
Row 3, Column 3: \(1 = ({\color{OrangeRed}2} \times {\color{blue}1}) + ({\color{OrangeRed}4} \times {\color{blue}{-\frac{1}{5}}}) + ({\color{OrangeRed}1} \times {\color{blue}{-\frac{1}{5}}})\)
This section reviews material you should already be familiar with from previous courses
You don’t need to memorize these equations, but you should be comfortable with using them
All of the material in this course (and in all of statistics!) builds on these basic concepts of central tendency, variability, and covariability
\(\mu_{X} = \frac{\sum X}{N}\)
where N is the size of the population
\(\overline{X} = \frac{\sum X}{n}\)
where n is the size of the sample
Sum of squared deviations from mean: sum of squares (SS)
\(SS_X =\sum (X_i - \mu_X)^2 = \sum X^2 - \frac{(\sum X)^2}{N}\)
where \(\mu_X\) is the population mean
\(SS_X =\sum (X_i - \overline{X})^2 = \sum X^2 - \frac{(\sum X)^2}{n}\)
where \(\overline{X}\) is the sample mean
Average squared deviation of scores around the mean
\({\sigma^2}_X = \frac{\sum (X_i - \mu_X)^2}{N}= \frac{\sum X^2 - \frac{(\sum X)^2}{N}}{N}\)
where \(\mu_X\) is the population mean and N is the population size
\({s^2}_X =\frac{\sum (X_i - \overline{X})^2}{n-1}=\frac{\sum X^2 - \frac{(\sum X)^2}{n}}{n-1}\)
where \(\overline{X}\) is the sample mean and n is the sample size
Square root of variance: in the same units as the original variable
\({\sigma}_X = \sqrt{\frac{\sum (X_i - \mu_X)^2}{N}}= \sqrt{ \frac{\sum X^2 - \frac{(\sum X)^2}{N}}{N}}\)
\({s}_X =\sqrt{\frac{\sum (X_i - \overline{X})^2}{n-1}}= \sqrt{\frac{\sum X^2 - \frac{(\sum X)^2}{n}}{n-1}}\)
Analogous to variation: sum of cross-products of the deviations or sum of products (SP)
\(SP_{XY} =\sum (X_i - \mu_X)(Y_i - \mu_Y) = \sum XY - \frac{(\sum X)(\sum Y)}{N}\)
where \(\mu_X\) and \(\mu_Y\) are the population means
\(SP_{XY} =\sum (X_i - \overline{X})(Y_i - \overline{Y}) = \sum XY - \frac{(\sum X)(\sum Y)}{n}\)
where \(\overline{X}\) and \(\overline{Y}\) are the sample means
Analogous to variance: average sum of cross-products of the deviations around the mean
\(\sigma_{XY} =\frac{\sum (X_i - \mu_X)(Y_i - \mu_Y)}{N} = \frac{\sum XY - \frac{(\sum X)(\sum Y)}{N}}{N}\)
\(s_{XY} =\frac{\sum (X_i - \overline{X})(Y_i - \overline{Y})}{n-1} = \frac{\sum XY - \frac{(\sum X)(\sum Y)}{n}}{n-1}\)
Standardized measure of how two variables are related
\(\rho_{XY} =\frac{\sum z_X z_Y}{N} = \frac{SP_{XY}}{\sqrt{SS_X}\sqrt{SS_Y}} = \frac{\sigma_{XY}}{\sigma_X \sigma_Y}\)
\(r_{XY} =\frac{\sum z_X z_Y}{n} = \frac{SP_{XY}}{\sqrt{SS_X}\sqrt{SS_Y}} = \frac{s_{XY}}{s_X s_Y}\)
where \(z_X\) and \(z_Y\) are standard scores (\(z\)-scores):
\(z_X = \frac{X_i - \mu_X}{\sigma_X}\) (population) or \(z_X = \frac{X_i - \overline{X}}{s_X}\) (sample)
Data is usually presented as
The data matrix is an \(n \times p\) matrix
\(\textbf{X} = \begin{bmatrix} X_{11} & \cdots & X_{1j} & \cdots & X_{1p} \\ \vdots & & \vdots & & \vdots \\ X_{i1} & \cdots & X_{ij} & \cdots & X_{ip} \\ \vdots & & \vdots & & \vdots \\ X_{n1} & \cdots & X_{nj} & \cdots & X_{np} \end{bmatrix}\)
The unit vector is a vector filled with \(1\)s
\(\begin{matrix} \underline{1}\\ (n,1) \end{matrix} = \begin{bmatrix} 1\\ 1\\ \vdots \\ 1 \end{bmatrix}\)
Used to add numbers in a matrix together
Same function as \(\sum\) in arithmetic: \(\sum_{i=1}^{n} X = \underline{1}' \underline{x}\)
The unit vector \(\underline{1}\) is typically a column vector but we can also us its transpose \(\underline{1}'\) when we need a row vector
\(\color{OrangeRed}{\begin{matrix} \underline{1}'\\ (1,4) \end{matrix} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ \end{bmatrix}}\) \(\color{blue}{\begin{matrix} \underline{x} \\ (4,1) \end{matrix} = \begin{bmatrix} 4 \\ 3 \\ 8 \\ 2 \\ \end{bmatrix}}\)
\(\color{OrangeRed}{\underline{1}'} \color{blue}{\underline{x}} = (\color{OrangeRed}{1} \times \color{blue}{4}) + (\color{OrangeRed}{1} \times \color{blue}{3}) + (\color{OrangeRed}{1} \times \color{blue}{8}) + (\color{OrangeRed}{1} \times \color{blue}{2}) =\)
\(4 + 3 + 8 + 2 = 17\)
The unit matrix is a matrix filled with \(1\)s
\(\begin{matrix} \textbf{E}\\ (n,n) \end{matrix} = \begin{bmatrix} 1 & 1 & \cdots & 1\\ 1 & 1 & \cdots & 1\\ \vdots & \vdots & \ddots & \vdots\\ 1 & 1 & \cdots & 1\\ \end{bmatrix}\)
Used to add numbers and create products of numbers
For most of our purposes, we’ll use an \(n \times n\) version, but it can be any size / order
Pre-multiply the vector of values by the unit vector and multiply by inverse of \(n\)
\(\overline{X} = \frac{1}{n} \:\underline{1}' \:\underline{x}\)
Example: Variable \(X\) is observed for \(n = 4\) subjects
\(\color{OrangeRed}{\begin{matrix} \underline{1}'\\ (1,4) \end{matrix} = \begin{bmatrix} 1 & 1 & 1 & 1 \end{bmatrix}}\) \(\color{blue}{\begin{matrix} \underline{x} \\ (4,1) \end{matrix} = \begin{bmatrix} 4 \\ 3 \\ 8 \\ 2 \end{bmatrix}}\)
\(\overline{X} =\frac{1}{n} \:\color{OrangeRed}{\underline{1}'} \:\color{blue}{\underline{x}} =\)
\(\frac{1}{4} [(\color{OrangeRed}{1}\times \color{blue}{4}) + (\color{OrangeRed}{1}\times \color{blue}{3}) + (\color{OrangeRed}{1}\times \color{blue}{8}) + (\color{OrangeRed}{1}\times \color{blue}{2})] =\)
\(\frac{1}{4}(4 + 3 + 8 + 2) = \frac{17}{4} = 4.25\)
Example: Variables \(X_1\), \(X_2\), and \(X_3\) for \(n=4\) subjects
\(\color{OrangeRed}{\begin{matrix} \underline{1}'\\ (1,4) \end{matrix} = \begin{bmatrix} 1 & 1 & 1 & 1 \end{bmatrix}}\) \(\color{blue}{\begin{matrix} \textbf{X} \\ (4,3) \end{matrix} = \begin{bmatrix} 4 & 2 & 4 \\ 3 & 1 & 1 \\ 8 & 3 & 2 \\ 2 & 5 & 5 \end{bmatrix}}\)
\(\overline{\underline{x}} =\frac{1}{n}\:\color{OrangeRed}{\underline{1}'}\:\color{blue}{\textbf{X}}= \frac{1}{4} \color{OrangeRed}{\begin{bmatrix} 1 & 1 & 1 & 1\\ \end{bmatrix}} \color{blue}{\begin{bmatrix} 4 & 2 & 4 \\ 3 & 1 & 1 \\ 8 & 3 & 2 \\ 2 & 5 & 5 \end{bmatrix}}=\)
\(\frac{1}{4} \begin{bmatrix} 4+3+8+2 & 2+1+3+5 & 4+1+2+5 \\ \end{bmatrix}=\)
\(\frac{1}{4} \begin{bmatrix} 17 & 11 & 12\\ \end{bmatrix}= \begin{bmatrix} 4.25 & 2.75 & 3 \end{bmatrix}\)
Sum a variable across all subjects:
\(\sum X = \sum_{i=1}^{n} X = \underline{1}' \: \textbf{X} = \textbf{X}' \: \underline{1}\)
Sum THEN square:
\((\sum X)^2 = (\sum_{i=1}^{n} X)^2 = \textbf{X}' \: \underline{1} \: \underline{1}' \: \textbf{X} = \textbf{X}' \: \textbf{E} \: \textbf{X}\)
Square THEN sum:
\(\sum (X^2) = \sum_{i=1}^{n} (X^2) = \textbf{X}' \: \textbf{X}\)
Recall that the sample variation is:
\(SS_X =\sum (X_i - \overline{X})^2 = \sum (X^2) - \frac{(\sum X)^2}{n}\)
Recall that the sample variation is:
\(SS_X =\sum (X_i - \overline{X})^2 = \sum (X^2) - \frac{(\sum X)^2}{n}\)
And that:
\(\sum (X^2) = \textbf{X}' \: \textbf{X}\)
\((\sum X)^2 = \textbf{X}' \: \underline{1} \: \underline{1}' \: \textbf{X} = \textbf{X}' \: \textbf{E} \: \textbf{X}\)
Recall that the sample variation is:
\(SS_X =\sum (X_i - \overline{X})^2 = \sum (X^2) - \frac{(\sum X)^2}{n}\)
And that:
\(\sum (X^2) = \textbf{X}' \: \textbf{X}\)
\((\sum X)^2 = \textbf{X}' \: \underline{1} \: \underline{1}' \: \textbf{X} = \textbf{X}' \: \textbf{E} \: \textbf{X}\)
Substitute matrix expressions:
\(SS_X = \textbf{X}' \: \textbf{X} - \frac{1}{n} \: \big( \textbf{X}' \: \textbf{E} \: \textbf{X} \big)\)
Recall that the sample covariation is:
\(SP_{XY} =\sum (X_i - \overline{X})(Y_i - \overline{Y}) = \sum XY - \frac{(\sum X)(\sum Y)}{n}\)
Recall that the sample covariation is:
\(SP_{XY} =\sum (X_i - \overline{X})(Y_i - \overline{Y}) = \sum XY - \frac{(\sum X)(\sum Y)}{n}\)
And that (extending to the \(X\) and \(Y\) situation):
\(\sum (XY) = \textbf{X}' \: \textbf{Y}\)
\((\sum X)(\sum Y) = \textbf{X}' \: \underline{1} \: \underline{1}' \: \textbf{Y} = \textbf{X}' \: \textbf{E} \: \textbf{Y}\)
Recall that the sample covariation is:
\(SP_{XY} =\sum (X_i - \overline{X})(Y_i - \overline{Y}) = \sum XY - \frac{(\sum X)(\sum Y)}{n}\)
And that (extending to the \(X\) and \(Y\) situation):
\(\sum (XY) = \textbf{X}' \: \textbf{Y}\)
\((\sum X)(\sum Y) = \textbf{X}' \: \underline{1} \: \underline{1}' \: \textbf{Y} = \textbf{X}' \: \textbf{E} \: \textbf{Y}\)
Substitute matrix expressions:
\(SP_{XY} = \textbf{X}' \: \textbf{Y} - \frac{1}{n} \: \big( \textbf{X}' \: \textbf{E} \: \textbf{Y} \big)\)
Involves many variables
Subscripts indicate which variables are involved: \(\textbf{P}_{XX}\), \(\textbf{P}_{XY}\)
Variation along the diagonal, covariation elsewhere
\(\textbf{P}_{XX} = \textbf{X'} \textbf{X} - \frac{1}{n} \textbf{X'} \textbf{E} \textbf{X} = \begin{bmatrix} \color{blue}{SS_{X_1}} & SP_{X_1X_2} & \cdots & SP_{X_1X_p}\\ SP_{X_2X_1} & \color{blue}{SS_{X_2}} & \cdots & SP_{X_2X_p}\\ \vdots & \vdots & \ddots & \vdots\\ SP_{X_pX_1} & SP_{X_pX_2} & \cdots & \color{blue}{SS_{X_p}}\\ \end{bmatrix}\)
Recall that the sample variance is:
\(s_{X}^2 =\frac{variation}{n-1} = \frac{SS_X}{n-1}\)
Multiply the matrix expression for variation by \(\frac{1}{n - 1}\):
\(s_{X}^2 = \frac{1}{n - 1} \Big( \textbf{X}' \: \textbf{X} - \frac{1}{n} \: \big( \textbf{X}' \: \textbf{E} \: \textbf{X} \big) \Big)\)
Recall that the sample covariance is:
\(cov_{XY} = s_{XY} = \frac{covariation}{n-1} = \frac{SP_{XY}}{n-1}\)
Mulitply the matrix expression for covariation by \(\frac{1}{n - 1}\):
\(cov_{XY} = s_{XY} = \frac{1}{n-1} \Big( \textbf{X}' \: \textbf{Y} - \frac{1}{n} \: \big( \textbf{X}' \: \textbf{E} \: \textbf{Y} \big) \Big)\)
Involves many variables
Subscripts indicate which variables are involved: \(\textbf{S}_{XX}\), \(\textbf{S}_{XY}\)
Variance along the diagonal, covariance elsewhere
One of THE most important matrices in statistics
\(\textbf{S}_{XX} = \frac{1}{n-1} \big( \textbf{X'} \textbf{X} - \frac{1}{n} \textbf{X'} \textbf{E} \textbf{X} \big) = \begin{bmatrix} {\color{blue}{s_{X_1}^2}} & s_{X_1X_2} & \cdots & s_{X_1X_p}\\ s_{X_2X_1} & \color{blue}{{s_{X_2}^2}} & \cdots & s_{X_2X_p}\\ \vdots & \vdots & \ddots & \vdots\\ s_{X_pX_1} & s_{X_pX_2} & \cdots & \color{blue}{{s_{X_p}^2}}\\ \end{bmatrix}\)
The correlation between X and Y is:
\(r_{XY} = \frac{SP_{XY}}{\sqrt{SS_X} \sqrt{SS_Y}}\)
Since division for matrices means multiplication by the inverse:
We need the inverse of \(\sqrt{SS_X}\) and \(\sqrt{SS_Y}\)
i.e., \(\sqrt{SS_X}^{-1}\) and \(\sqrt{SS_Y}^{-1}\)
\(\textbf{D}_P\) is a matrix with the square root of variation on the diagonal:
\(\textbf{D}_P = \begin{bmatrix} \sqrt{SS_{X_1}} & 0 & \cdots & 0\\ 0 & \sqrt{SS_{X_2}} & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & \sqrt{SS_{X_p}}\\ \end{bmatrix}\)
The inverse of \(\textbf{D}_P\):
\(\textbf{D}^{-1}_P = \begin{bmatrix} \frac{1}{\sqrt{SS_{X_1}}} & 0 & \cdots & 0 \\ 0 & \frac{1}{\sqrt{SS_{X_2}}} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \frac{1}{\sqrt{SS_{X_p}}} \\ \end{bmatrix}\)
\(\textbf{D}_S\) is a matrix with the square root of variance on the diagonal:
\(\textbf{D}_S = \begin{bmatrix} \sqrt{{s_{X_1}^2}} & 0 & \cdots & 0\\ 0 & \sqrt{{s_{X_2^2}}} & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & \sqrt{{s_{X_p}^2}}\\ \end{bmatrix}\)
The inverse of \(\textbf{D}_S\):
\(\textbf{D}^{-1}_S = \begin{bmatrix} \frac{1}{\sqrt{{s_{X_1}^2}}} & 0 & \cdots & 0\\ 0 & \frac{1}{\sqrt{{s_{X_2}^2}}} & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & \frac{1}{\sqrt{{s_{X_p}^2}}}\\ \end{bmatrix}\)
Involves many variables
Subscripts indicate which variables are involved: \(\textbf{R}_{XX}\), \(\textbf{R}_{XY}\)
\(1\)s along the diagonal, correlations elsewhere
One of THE most important matrices in statistics
In terms of variation and covariation:
\(\textbf{R}_{XX} = \textbf{D}^{-1}_P \: \textbf{P} \: \textbf{D}^{-1}_P = \begin{bmatrix} 1 & r_{X_1X_2} & \cdots & r_{X_1X_p}\\ r_{X_2X_1} & 1 & \cdots & r_{X_2X_p}\\ \vdots & \vdots & \ddots & \vdots\\ r_{X_pX_1} & r_{X_pX_2} & \cdots & 1\\ \end{bmatrix}\)
In terms of variance and covariance:
\(\textbf{R}_{XX} =\textbf{D}^{-1}_S \: \textbf{S} \: \textbf{D}^{-1}_S = \begin{bmatrix} 1 & r_{X_1X_2} & \cdots & r_{X_1X_p}\\ r_{X_2X_1} & 1 & \cdots & r_{X_2X_p}\\ \vdots & \vdots & \ddots & \vdots\\ r_{X_pX_1} & r_{X_pX_2} & \cdots & 1\\ \end{bmatrix}\)