CBN Courses

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
======Contrast Matrix Formulation======

The constraints on the full model can be written as
$$\mathbf{L} 𝛃 = \mathbf{c} $$
Using Lagrange multiplier, we can get the estimate for null hypothesis $\widehat{𝛃}_0$ by minimizing
$$\mathcal{L}\equiv\left(\mathbf{y}-\mathbf{X}𝛃\right)^{T}\left(\mathbf{y}-\mathbf{X}𝛃\right)+\left(\mathbf{L}𝛃-\mathbf{c}\right)^{T}𝛌,$$
where $𝛌$ is a column vector of multipliers.
The minimization leads to
$$ -2\mathbf{X}^{T}\left(\mathbf{y}-\mathbf{X}𝛃_{0}\right)+\mathbf{L}^{T}𝛌_{0}=0 $$
and 
$$ \mathbf{L}𝛃_{0}=\mathbf{c}. $$
These equations give
\begin{align*}
& \mathbf{L}^{T}𝛌_{0}=2\mathbf{X}^{T}\left(\mathbf{y}-\mathbf{X}𝛃_{0}\right)=2\mathbf{X}^{T}\mathbf{y}-2\mathbf{X}^{T}\mathbf{X}𝛃_{0}\\
\Rightarrow\, & \left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}𝛌_{0}=2\widehat{𝛃}-2𝛃_{0}\\
\Rightarrow\, & \mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}𝛌_{0}=2\mathbf{L}\widehat{𝛃}-2\mathbf{c}
\end{align*}
or
$$ 𝛌_{0}=2\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{𝛃}-\mathbf{c}\right), $$
as well as
$$ 𝛃_{0}=\widehat{𝛃}-\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}𝛌_{0}/2, $$
or
\begin{align*}
\widehat{𝛃}-𝛃_{0} & = \left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}𝛌_{0}/2\\
& = \left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{𝛃}-\mathbf{c}\right).
\end{align*}

As the the error of the restricted model is given by the square of the residual,
$$ \mathbf{r}_{\mathrm{restr}} = \mathbf{y}-\mathbf{X}𝛃_{0} = \mathbf{r}_{\mathrm{full}}+\mathbf{X}\left(\widehat{𝛃}-𝛃_{0}\right), $$
we have
\begin{eqnarray*}
\mathrm{Err}_{\mathrm{restr}} & = & \mathbf{r}_{\mathrm{restr}}^{T}\mathbf{r}_{\mathrm{restr}}\\
 & = & \mathbf{r}_{\mathrm{full}}^{T}\mathbf{r}_{\mathrm{full}}+\left(\widehat{𝛃}-𝛃_{0}\right)^{T}\color{#00a}{\mathbf{X}^{T}\mathbf{X}\left(\widehat{𝛃}-𝛃_{0}\right)}\\
 & = & \mathrm{Err}_{\mathrm{full}}+\color{#800}{\left(\widehat{𝛃}-𝛃_{0}\right)^{T}}\color{#00a}{\mathbf{L}^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{𝛃}-\mathbf{c}\right)}\\
 & = & \mathrm{Err}_{\mathrm{full}}+\color{#800}{\left(\mathbf{L}\widehat{𝛃}-\mathbf{c}\right)^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}}\\
 & & \times\mathbf{L}^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{𝛃}-\mathbf{c}\right)\\
 & = & \mathrm{Err}_{\mathrm{full}}+\left(\mathbf{L}\widehat{𝛃}-\mathbf{c}\right)^{T}\left(\mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T}\right)^{-1}\left(\mathbf{L}\widehat{𝛃}-\mathbf{c}\right)
\end{eqnarray*}
where the cross terms vanish since $ \mathbf{X}^{T}\mathbf{r}_{\mathrm{full}} = 0 $ by the condition of the full model, and we observe that $ \mathbf{X}^{T}\mathbf{X} $ and $ \mathbf{L}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{L}^{T} $ are both symmetric matrices.
CBN Courses

User Tools

Site Tools

Page Tools
