Site Tools


notes:dan:contrast-matrix

Contrast Matrix Formulation

The constraints on the full model can be written as $$\mathbf{L} ๐›ƒ = \mathbf{c} $$ Using Lagrange multiplier, we can get the estimate for null hypothesis $\widehat{๐›ƒ}_0$ by minimizing $$\mathcal{L}\equiv\left(\mathbf{y}-\mathbf{X}๐›ƒ\right)^{T}\left(\mathbf{y}-\mathbf{X}๐›ƒ\right)+\left(\mathbf{L}๐›ƒ-\mathbf{c}\right)^{T}๐›Œ,$$ where $๐›Œ$ is a column vector of multipliers. The minimization leads to $$ -2\mathbf{X}^{T}\left(\mathbf{y}-\mathbf{X}๐›ƒ_{0}\right)+\mathbf{L}^{T}๐›Œ_{0}=0 $$ and $$ \mathbf{L}๐›ƒ_{0}=\mathbf{c}. $$ These equations give \begin{align*} & \mathbf{L}^{T}๐›Œ_{0}=2\mathbf{X}^{T}\left(\mathbf{y}-\mathbf{X}๐›ƒ_{0}\right)=2\mathbf{X}^{T}\mathbf{y}-2\mathbf{X}^{T}\mathbf{X}๐›ƒ_{0}\\ \Rightarrow\, & \left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}๐›Œ_{0}=2\widehat{๐›ƒ}-2๐›ƒ_{0}\\ \Rightarrow\, & \mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}๐›Œ_{0}=2\mathbf{L}\widehat{๐›ƒ}-2\mathbf{c} \end{align*} or $$ ๐›Œ_{0}=2\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{๐›ƒ}-\mathbf{c}\right), $$ as well as $$ ๐›ƒ_{0}=\widehat{๐›ƒ}-\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}๐›Œ_{0}/2, $$ or \begin{align*} \widehat{๐›ƒ}-๐›ƒ_{0} & = \left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}๐›Œ_{0}/2\\ & = \left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{๐›ƒ}-\mathbf{c}\right). \end{align*}

As the the error of the restricted model is given by the square of the residual, $$ \mathbf{r}_{\mathrm{restr}} = \mathbf{y}-\mathbf{X}๐›ƒ_{0} = \mathbf{r}_{\mathrm{full}}+\mathbf{X}\left(\widehat{๐›ƒ}-๐›ƒ_{0}\right), $$ we have \begin{eqnarray*} \mathrm{Err}_{\mathrm{restr}} & = & \mathbf{r}_{\mathrm{restr}}^{T}\mathbf{r}_{\mathrm{restr}}\\ & = & \mathbf{r}_{\mathrm{full}}^{T}\mathbf{r}_{\mathrm{full}}+\left(\widehat{๐›ƒ}-๐›ƒ_{0}\right)^{T}\color{#00a}{\mathbf{X}^{T}\mathbf{X}\left(\widehat{๐›ƒ}-๐›ƒ_{0}\right)}\\ & = & \mathrm{Err}_{\mathrm{full}}+\color{#800}{\left(\widehat{๐›ƒ}-๐›ƒ_{0}\right)^{T}}\color{#00a}{\mathbf{L}^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{๐›ƒ}-\mathbf{c}\right)}\\ & = & \mathrm{Err}_{\mathrm{full}}+\color{#800}{\left(\mathbf{L}\widehat{๐›ƒ}-\mathbf{c}\right)^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}}\\ & & \times\mathbf{L}^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{๐›ƒ}-\mathbf{c}\right)\\ & = & \mathrm{Err}_{\mathrm{full}}+\left(\mathbf{L}\widehat{๐›ƒ}-\mathbf{c}\right)^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{๐›ƒ}-\mathbf{c}\right) \end{eqnarray*} where the cross terms vanish since $ \mathbf{X}^{T}\mathbf{r}_{\mathrm{full}} = 0 $ by the condition of the full model, and we observe that $ \mathbf{X}^{T}\mathbf{X} $ and $ \mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T} $ are both symmetric matrices.

notes/dan/contrast-matrix.txt ยท Last modified: 2022/03/20 08:37 by cjj