Exercises

  1. This question is about the normal distribution, and how it relates to the classification rule provided by linear discriminant analysis.
  1. (1)Write down the density function for a bivariate normal distribution (\(p=2\)), with mean \(\mu_k\) and variance \(\Sigma\).

\[f(x) = \frac{1}{2\pi|\Sigma|}\exp\{-\frac{1}{2}(x-\mu)'\Sigma^{-1}(x-\mu)\}\]
where \(x=(x_1, x_2)\), \(\mu=(\mu_1, \mu_2)\), \(\Sigma=\left[ \begin{array}{cc} \sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2 \end{array} \right]\)

  1. (2)Show that the linear discriminant rule for two groups (\(K=2\)), \(\pi_1=\pi_2\) and

\[\Sigma_1=\Sigma_2 = \Sigma = \left[\begin{array}{cc} \sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2 \end{array}\right]\] where \(\rho\) is the population correlation between the two variables, and \(\sigma_1, \sigma_2\) are the population standard deviations of the two variables, respectively, is equal to: Assign a new observation \(x_0\) to group 1 if \[x_0'\Sigma^{-1}(\mu_1-\mu_2) > \frac{1}{2}(\mu_1 + \mu_2)'\Sigma^{-1}(\mu_1-\mu_2)\] The rule is to assign observation to the group with the highest probability, so work from the density functions for the tewo groups. Let

\(x=(x_1, x_2)\), \(\mu_1=(\mu_{11}, \mu_{12})\), \(\mu_2=(\mu_{21}, \mu_{22})\) then

\[\frac{1}{2\pi|\Sigma|}\exp\{-\frac{1}{2}(x-\mu_1)'\Sigma^{-1}(x-\mu_1)\}>\frac{1}{2\pi|\Sigma|}\exp\{-\frac{1}{2}(x-\mu_2)'\Sigma^{-1}(x-\mu_2)\}\]

\[\leadsto ~~~ (x-\mu_1)'\Sigma^{-1}(x-\mu_1) > (x-\mu_2)'\Sigma^{-1}(x-\mu_2)\] \[\leadsto ~~~~~~~~~~~~~~~ x'\Sigma^{-1}x -x'\Sigma^{-1}\mu_1 - \mu_1'\Sigma^{-1}x+\mu_1'\Sigma^{-1}\mu_1 > x'\Sigma^{-1}x -x'\Sigma^{-1}\mu_2 - \mu_2'\Sigma^{-1}x+\mu_2'\Sigma^{-1}\mu_2\] \[\leadsto ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-x'\Sigma^{-1}\mu_1 - \mu_1'\Sigma^{-1}x+\mu_1'\Sigma^{-1}\mu_1> -x'\Sigma^{-1}\mu_2 - \mu_2'\Sigma^{-1}x+\mu_2'\Sigma^{-1}\mu_2 \] \[\leadsto ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x'\Sigma^{-1}\mu_1 +x'\Sigma^{-1}\mu_2 + \mu_1'\Sigma^{-1}x + \mu_2'\Sigma^{-1}x > - \mu_1'\Sigma^{-1}\mu_1 + \mu_2'\Sigma^{-1}\mu_2 \] \[\leadsto ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~2x'\Sigma^{-1}(\mu_1-\mu_2) > (\mu_1+\mu_2)'\Sigma^{-1}(\mu_1-\mu_2)\] \[\leadsto ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x'\Sigma^{-1}(\mu_1-\mu_2) > \frac{1}{2}(\mu_1+\mu_2)'\Sigma^{-1}(\mu_1-\mu_2)\] c. (2)By generating a grid of values, draw the boundary between two groups, in the 2D space. Use these values for \(\mu_1, \mu_2\) and \(\sigma\). \[\mu_1 = \left[\begin{array}{r} -2 \\ 2 \end{array}\right], ~~~\mu_2 = \left[\begin{array}{r} 2 \\ -2 \end{array}\right]\] \[\Sigma = \left[\begin{array}{rr} 4 & 3 \\ 3 & 5 \end{array}\right]\]