Title: Visualizing Linear Models
Tags: background-linear-algebra blue green background-random-vectors red linear-models background-normality least-squares-linear-regression normal-errors

Number: Exercise 1.01
Question: Let [latex]${\mathbf{0}}$[/latex] be the zero vector, and let [latex]$a$[/latex] be a scalar. Show that [latex]$a {\mathbf{0}} = {\mathbf{0}}$[/latex].
Answer: By definiton, a vector space has the property that [latex]${\mathbf{v}} - {\mathbf{v}} = {\mathbf{0}}$[/latex] for any vector [latex]${\mathbf{v}}$[/latex]. In particular, [latex]${\mathbf{0}} - {\mathbf{0}} = {\mathbf{0}}$[/latex]. [latex]\begin{align*} a {\mathbf{0}} &= a ({\mathbf{0}} - {\mathbf{0}})\\ &= a {\mathbf{0}} - a {\mathbf{0}}\\ &= {\mathbf{0}} \end{align*}[/latex] We distributed scalar multiplication then used the [latex]${\mathbf{v}} - {\mathbf{v}} = {\mathbf{0}}$[/latex] property again.

Number: Exercise 1.02
Question: Let [latex]${\mathbf{v}}$[/latex] be a vector, and let [latex]$0$[/latex] be the zero scalar. Show that [latex]$0 {\mathbf{v}}$[/latex] equals the zero vector.
Answer: The zero scalar satisfies [latex]$a-a=0$[/latex] for every [latex]$a$[/latex] in the scalar field; in particular, [latex]$0-0=0$[/latex]. We make this substitution, then distribute and invoke the fact that [latex]${\mathbf{v}} - {\mathbf{v}}$[/latex] equals the zero vector [latex]${\mathbf{0}}$[/latex] for any vector [latex]${\mathbf{v}}$[/latex]. [latex]\begin{align*} 0 {\mathbf{v}} &= (0 - 0) {\mathbf{v}}\\ &= 0 {\mathbf{v}} - 0 {\mathbf{v}}\\ &= {\mathbf{0}} \end{align*}[/latex]

Number: Exercise 1.03
Question: Show that the span of [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] is a subspace.
Answer: Consider two vectors in the span, say [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}_1$[/latex] and [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}_2$[/latex]. For a pair of scalars [latex]$a_1, a_2$[/latex] the linear combination [latex]\begin{align*} a_1 {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}_1 + a_2 {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}_2 &= {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} (a_1 {\mathbf{b}}_1 + a_2 {\mathbf{b}}_2) \end{align*}[/latex] is also in the span, so the span satisfies the definition of a subspace.

Number: Exercise 1.04
Question: Show that the span of [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] is the same as the span of [latex]${\mathbf{v}}_1 + a_1 {\mathbf{v}}_m, \ldots, {\mathbf{v}}_{m-1} + a_{m-1} {\mathbf{v}}_m, {\mathbf{v}}_m$[/latex] for any scalars [latex]$a_1, \ldots, a_{m-1}$[/latex].
Answer: We’ll show that an arbitrary linear combination of [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] can also be represented as a linear combination of the altered vectors by adding and subtracting the appropriately scaled versions of [latex]${\mathbf{v}}_m$[/latex]. [latex]\begin{align*} b_1 {\mathbf{v}}_1 + \ldots + b_{m-1} {\mathbf{v}}_{m-1} + b_m {\mathbf{v}}_m &= b_1 ({\mathbf{v}}_1 + a_1 {\mathbf{v}}_m) + \ldots + b_{m-1} ({\mathbf{v}}_{m-1} + a_{m-1} {\mathbf{v}}_m) + (b_m - b_1 a_1 - \ldots - b_{m-1} a_{m-1}) {\mathbf{v}}_m \end{align*}[/latex] Similarly, an arbitrary linear combination of the altered vectors becomes a linear combination of [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] by distributing the coefficients and regrouping the terms. [latex]\begin{align*} b_1 ({\mathbf{v}}_1 + a_1 {\mathbf{v}}_1) + \ldots + b_{m-1} ({\mathbf{v}}_{m-1} + a_{m-1} {\mathbf{v}}_m) + b_m {\mathbf{v}}_m &= b_1 {\mathbf{v}}_1 + \ldots + b_{m-1} {\mathbf{v}}_{m-1} + (b_m + b_1 a_1 + \ldots + b_{m-1} a_{m-1}) {\mathbf{v}}_m \end{align*}[/latex]

Number: Exercise 1.05
Question: Let [latex]${\mathcal{F}}$[/latex] be a field, and suppose [latex]${\mathbb{T}}$[/latex] is a linear operator from [latex]${\mathcal{F}}^m$[/latex] to [latex]${\mathcal{V}}$[/latex]. Show that [latex]${\mathbb{T}}$[/latex] can be represented as a mapping of [latex]${\mathbf{b}} \in {\mathcal{F}}^m$[/latex] to [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}$[/latex] for some [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex].
Answer: First, consider that [latex]${\mathbb{T}}$[/latex] and [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}}$[/latex] would have to agree on where to map the standard basis vectors. We see that [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}}$[/latex] maps [latex]${\mathbf{e}}_1 := (1, 0, \ldots, 0)$[/latex] to [latex]${\mathbf{v}}_1$[/latex], so [latex]${\mathbf{v}}_1$[/latex] must be [latex]${\mathbb{T}} {\mathbf{e}}_1$[/latex]. Likewise [latex]${\mathbf{v}}_2$[/latex] would need to [latex]${\mathbb{T}} {\mathbf{e}}_2$[/latex], and so on. Let’s check that this proposal [latex]$\begin{bmatrix} {\mathbb{T}} {\mathbf{e}}_1 & \ldots & {\mathbb{T}} {\mathbf{e}}_m \end{bmatrix} {\mathbf{b}}$[/latex] is the same as [latex]${\mathbb{T}} {\mathbf{b}}$[/latex] for an arbitrary vector [latex]${\mathbf{b}}$[/latex]. [latex]\begin{align*} \begin{bmatrix} {\mathbb{T}} {\mathbf{e}}_1 & \ldots & {\mathbb{T}} {\mathbf{e}}_m \end{bmatrix} {\mathbf{b}} &= \begin{bmatrix} {\mathbb{T}} {\mathbf{e}}_1 & \ldots & {\mathbb{T}} {\mathbf{e}}_m \end{bmatrix} (b_1 {\mathbf{e}}_1 + \ldots + b_m {\mathbf{e}}_m)\\ &= b_1 \begin{bmatrix} {\mathbb{T}} {\mathbf{e}}_1 & \ldots & {\mathbb{T}} {\mathbf{e}}_m \end{bmatrix} {\mathbf{e}}_1 + \ldots + b_m \begin{bmatrix} {\mathbb{T}} {\mathbf{e}}_1 & \ldots & {\mathbb{T}} {\mathbf{e}}_m \end{bmatrix} {\mathbf{e}}_m\\ &= b_1 {\mathbb{T}} {\mathbf{e}}_1 + \ldots + b_m {\mathbb{T}} {\mathbf{e}}_m\\ &= {\mathbb{T}} (b_1 {\mathbf{e}}_1 + \ldots + b_m {\mathbf{e}}_m)\\ &= {\mathbb{T}} {\mathbf{b}} \end{align*}[/latex]

Number: Exercise 1.06
Question: Prove that the null space of [latex]${\mathbb{T}}$[/latex] is a subspace.
Answer: Let [latex]${\mathbf{b}}_1$[/latex] and [latex]${\mathbf{b}}_2$[/latex] be in the null space. Given any scalars [latex]$a_1, a_2$[/latex], consider the vector of scalars [latex]$a_1 {\mathbf{b}}_1 + a_2 {\mathbf{b}}_2$[/latex]. [latex]\begin{align*} {\mathbb{T}} (a_1 {\mathbf{b}}_1 + a_2 {\mathbf{b}}_2) &= a_1 \underbrace{{\mathbb{T}} {\mathbf{b}}_1}_{\mathbf{0}} + a_2 \underbrace{{\mathbb{T}} {\mathbf{b}}_2}_{\mathbf{0}}\\ &= {\mathbf{0}} \end{align*}[/latex] Since [latex]$a_1 {\mathbf{b}}_1 + a_2 {\mathbf{b}}_2$[/latex] is also mapped to [latex]${\mathbf{0}}$[/latex], it’s in the null space as well; the null space therefore satisfies the definition of a subspace.

Number: Exercise 1.07
Question: Show that if [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] are linearly independent then [latex]$(b_1, \ldots, b_m) = (0, \ldots, 0)$[/latex] is the only vector of scalars for which [latex]$b_1 {\mathbf{v}}_1 + \ldots + b_m {\mathbf{v}}_m = {\mathbf{0}}$[/latex].
Answer: Assume there exist scalars [latex]$b_1, \ldots, b_m$[/latex] for which [latex]$b_1 {\mathbf{v}}_1 + \ldots + b_m {\mathbf{v}}_m = 0$[/latex] with [latex]$b_1 \neq 0$[/latex] (without loss of generality). Then the rearranged equation [latex]\begin{align*} {\mathbf{v}}_1 = \left(-\tfrac{b_2}{b_1}\right) {\mathbf{v}}_2 + \ldots + \left(-\tfrac{b_m}{b_1}\right) {\mathbf{v}}_m \end{align*}[/latex] shows that [latex]${\mathbf{v}}_1$[/latex] is a linear combination of the other vectors, contradicting the assumption of linear independence.

Number: Exercise 1.08
Question: Show that if [latex]$(b_1, \ldots, b_m) = (0, \ldots, 0)$[/latex] is the only vector of scalars for which [latex]$b_1 {\mathbf{v}}_1 + \ldots + b_m {\mathbf{v}}_m = {\mathbf{0}}$[/latex], then [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] are linearly independent.
Answer: Assume that [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] aren’t linearly independent; in particular, and without loss of generality, assume [latex]${\mathbf{v}}_m = c_1 {\mathbf{v}}_1 + \ldots + c_{m-1} {\mathbf{v}}_{m-1}$[/latex] for some scalars [latex]$c_1, \ldots, c_{m-1}$[/latex]. Then subtracting [latex]${\mathbf{v}}_m$[/latex] from both sides, we see that [latex]\begin{align*} {\mathbf{0}} &= c_1 {\mathbf{v}}_1 + \ldots + c_{m-1} {\mathbf{v}}_{m-1} + (-1){\mathbf{v}}_m \end{align*}[/latex] provides a linear combination of zero in which not all of the scalar coefficients are zeros.

Number: Exercise 1.09
Question: Let [latex]${\mathbf{z}}$[/latex] be in the null space of [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}}$[/latex]. Given any vector of scalars [latex]${\mathbf{b}}$[/latex], show that the linear combination of [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] produced by the entries of [latex]${\mathbf{b}} + {\mathbf{z}}$[/latex] is exactly the same as that produced by [latex]${\mathbf{b}}$[/latex].
Answer: With [latex]${\mathbf{z}}$[/latex] in the null space, the [latex]${\mathbf{b}} + {\mathbf{z}}$[/latex] linear combination results in [latex]\begin{align*} {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} ({\mathbf{b}} + {\mathbf{z}}) &= {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}} + \underbrace{{\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} \mathbf{z}}_{\mathbf{0}}\\ &= {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}. \end{align*}[/latex]

Number: Exercise 1.10
Question: Show that if [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}} = {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{c}}$[/latex] then [latex]${\mathbf{c}}$[/latex] must equal [latex]${\mathbf{b}} + {\mathbf{z}}$[/latex] for some [latex]${\mathbf{z}}$[/latex] in the null space of [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}}$[/latex].
Answer: Trivially [latex]${\mathbf{c}} = {\mathbf{b}} + ({\mathbf{c}}-{\mathbf{b}})$[/latex]; we show that the second term is in the null space. [latex]\begin{align*} {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} ({\mathbf{c}} - {\mathbf{b}}) &= {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{c}} - {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}\\ &= {\mathbf{0}} \end{align*}[/latex] because the two linear combinations are equal.

Number: Exercise 1.11
Question: Show that [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] are linearly independent if and only if [latex]${\mathbf{b}} \neq {\mathbf{c}}$[/latex] implies [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}} \neq {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{c}}$[/latex], that is, every vector in the span corresponds to a unique vector of scalar coefficients.
Answer: With [latex]${\mathcal{N}}$[/latex] representing the null space of [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}}$[/latex], the set of coefficient vectors producing [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}$[/latex] is exactly [latex]$\{{\mathbf{b}} + {\mathbf{z}} : {\mathbf{z}} \in {\mathcal{N}}\}$[/latex]; this set has a single element ([latex]${\mathbf{b}}$[/latex]) if and only if the null space has only the zero vector. We’ve already established that this condition is equivalent to linear independence.

Number: Exercise 1.12
Question: Let [latex]$\{{\mathbf{v}_1, \ldots, \mathbf{v}_m}\}$[/latex] be a basis for [latex]${\mathcal{V}}$[/latex]. How do you know that [latex]$\{{\mathbf{v}}_2, \ldots, {\mathbf{v}}_m\}$[/latex] is not also a basis for [latex]${\mathcal{V}}$[/latex]?
Answer: By linear independence, no linear combination of [latex]${\mathbf{v}}_2, \ldots, {\mathbf{v}}_m$[/latex] equals [latex]${\mathbf{v}}_1 \in {\mathcal{V}}$[/latex]. Therefore, [latex]${\mathbf{v}}_2, \ldots, {\mathbf{v}}_m$[/latex] do not span [latex]${\mathcal{V}}$[/latex] and thus do not satisfy the definition of a basis.

Number: Exercise 1.13
Question: Let [latex]$\{{\mathbf{v}_1, \ldots, \mathbf{v}_m}\}$[/latex] be a basis for [latex]${\mathcal{V}}$[/latex], and let [latex]${\mathcal{S}}$[/latex] be a proper subspace of [latex]${\mathcal{V}}$[/latex]. Explain why at least one of [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] is not in [latex]${\mathcal{S}}$[/latex].
Answer: Any [latex]${\mathbf{w}}$[/latex] in [latex]${\mathcal{V}}$[/latex] has a unique representation as [latex]$b_1 {\mathbf{v}}_1 + \ldots b_m {\mathbf{v}}_m$[/latex] for some scalar coefficients. Assume every one of the basis vectors were in [latex]${\mathcal{S}}$[/latex]. Because [latex]${\mathcal{S}}$[/latex] is a subspace it contains all linear combinations of its vectors which would include [latex]${\mathbf{w}}$[/latex]. Therefore, if [latex]${\mathcal{S}}$[/latex] is a subspace of [latex]${\mathcal{V}}$[/latex] that includes an entire basis for [latex]${\mathcal{V}}$[/latex], then it would have to be equal to [latex]${\mathcal{V}}$[/latex].

Number: Exercise 1.14
Question: Suppose [latex]$\{{\mathbf{v}_1, \ldots, \mathbf{v}_m}\}$[/latex] is a basis for [latex]${\mathcal{V}}$[/latex]. Prove that every basis for [latex]${\mathcal{V}}$[/latex] has exactly [latex]$m$[/latex] vectors.
Answer: Suppose that [latex]$\{{\mathbf{w}}_1, \ldots, {\mathbf{w}}_k\}$[/latex] is also a basis for [latex]${\mathcal{V}}$[/latex] and that [latex]$k < m$[/latex]. The set [latex]$\{{\mathbf{v}}_2, \ldots, {\mathbf{v}}_m\}$[/latex] doesn’t span [latex]${\mathcal{V}}$[/latex], so there must be some vector in [latex]$\{{\mathbf{w}}_1, \ldots, {\mathbf{w}}_k\}$[/latex] that isn’t in the span of [latex]$\{{\mathbf{v}}_2, \ldots, {\mathbf{v}}_m\}$[/latex]. Assume without loss of generality that [latex]${\mathbf{w}}_1$[/latex] isn’t in the span of [latex]$\{{\mathbf{v}}_2, \ldots, {\mathbf{v}}_m\}$[/latex]. Then [latex]$\{{\mathbf{w}}_1, {\mathbf{v}}_2, \ldots, {\mathbf{v}}_m\}$[/latex] is linearly independent. By repeating this logic with [latex]${\mathbf{v}}_2$[/latex] and continuing through [latex]${\mathbf{v}}_k$[/latex], we end up claiming that [latex]$\{{\mathbf{w}}_1, \ldots, {\mathbf{w}}_k, {\mathbf{v}}_{k+1}, \ldots, {\mathbf{v}}_m\}$[/latex] is a linearly independent set. However, since [latex]$\{{\mathbf{w}}_1, \ldots, {\mathbf{w}}_k\}$[/latex] is a basis for [latex]${\mathcal{V}}$[/latex], the remaining vectors [latex]$\{{\mathbf{v}}_{k+1}, \ldots, {\mathbf{v}}_m\} \subset {\mathcal{V}}$[/latex] have to be in their span which is a contradiction. This proves that one basis for [latex]${\mathcal{V}}$[/latex] can’t be larger than another.

Number: Exercise 1.15
Question: Let [latex]${\mathcal{V}}$[/latex] be an [latex]$m$[/latex]-dimensional vector space. Prove that any set of [latex]$m$[/latex] linearly independent vectors in [latex]${\mathcal{V}}$[/latex] must be basis for [latex]${\mathcal{V}}$[/latex].
Answer: Let [latex]$V := \{{\mathbf{v}_1, \ldots, \mathbf{v}_m}\}$[/latex] be a set of linearly independent vectors in [latex]${\mathcal{V}}$[/latex], and let [latex]$W := \{{\mathbf{w}_1, \ldots, \mathbf{w}_m}\}$[/latex] be a basis for [latex]${\mathcal{V}}$[/latex]. Assume that the span of [latex]$V$[/latex] is not [latex]${\mathcal{V}}$[/latex]. From Exercise 1.13 there exists at least one vector in [latex]$W$[/latex] that isn’t in the span of [latex]$V$[/latex]; without loss of generality, let it be [latex]${\mathbf{w}}_1$[/latex]. By continuing to add one vector at a time from [latex]$V$[/latex] to [latex]$W$[/latex] in this way, you would maintain linear independence and you would eventually produce a set that does have [latex]${\mathcal{V}}$[/latex] as its span. But that set will have more than [latex]$m$[/latex] vectors, which contradicts the fact from Exercise 1.14 that every basis for [latex]${\mathcal{V}}$[/latex] has [latex]$m$[/latex] vectors.

Number: Exercise 1.16
Question: Let [latex]${\mathcal{S}}$[/latex] be a subspace of an [latex]$n$[/latex]-dimensional vector space [latex]${\mathcal{V}}$[/latex]. Prove that a basis for [latex]${\mathcal{S}}$[/latex] exists.
Answer: We’ll describe a constructive proof. Take any vector of [latex]${\mathcal{S}}$[/latex] and call it [latex]${\mathbf{v}}_1$[/latex]. If [latex]${\mathcal{S}}$[/latex] equals the span of [latex]${\mathbf{v}}_1$[/latex], then [latex]${\mathbf{v}}_1$[/latex] is a basis. Otherwise, take a vector from [latex]${\mathcal{S}}$[/latex] that is outside of the span of [latex]${\mathbf{v}}_1$[/latex] and call it [latex]${\mathbf{v}}_2$[/latex]. If [latex]${\mathcal{S}}$[/latex] equals the span of [latex]$\{{\mathbf{v}}_1, {\mathbf{v}}_2\}$[/latex], then they form a basis. Otherwise, continue to repeat this process of adding one more linearly independent vector at a time until the vectors span [latex]${\mathcal{S}}$[/latex]. The algorithm is guaranteed to terminate with no more than [latex]$n$[/latex] vectors, because any set of [latex]$n$[/latex] linearly independent vectors is a basis for [latex]${\mathcal{V}}$[/latex] according to Exercise 1.15.

Number: Exercise 1.17
Question: Let [latex]${\mathcal{F}}$[/latex] be a field. Find the dimension of [latex]${\mathcal{F}}^m$[/latex] as defined in Section 1.2.
Answer: Consider the [latex]$n$[/latex] vectors [latex]${\mathbf{e}}_1 := (1, 0, \ldots, 0), \ldots, {\mathbf{e}}_m := (0, \ldots, 0, 1)$[/latex]. A given vector [latex]$(c_1, \ldots, c_m) \in {\mathcal{F}}^m$[/latex] has the unique representation [latex]$c_1 {\mathbf{e}}_1 + \ldots + c_m {\mathbf{e}}_m$[/latex] with respect to these vectors, so they comprise a basis (known as the standard basis). This tells us that the dimension of [latex]${\mathcal{F}}^m$[/latex] is [latex]$m$[/latex].

Number: Exercise 1.18
Question: Suppose [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}}$[/latex] and [latex]${\begin{bmatrix} \mathbf{w}_1 & \cdots & \mathbf{w}_m \end{bmatrix}}$[/latex] have the exact same behavior on a basis [latex]$B = \{{\mathbf{b}}_1, \ldots, {\mathbf{b}}_m\}$[/latex] for the vector space of scalar coefficients, that is, [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}_j = {\begin{bmatrix} \mathbf{w}_1 & \cdots & \mathbf{w}_m \end{bmatrix}} {\mathbf{b}}_j$[/latex] for every [latex]$j \in \{1, \ldots, m\}$[/latex]. Show that [latex]${\mathbf{v}}_j$[/latex] must equal [latex]${\mathbf{w}}_j$[/latex] for every [latex]$j \in \{1, \ldots, m\}$[/latex].
Answer: We’ll first show that [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}}$[/latex] and [latex]${\begin{bmatrix} \mathbf{w}_1 & \cdots & \mathbf{w}_m \end{bmatrix}}$[/latex] must have the exact same behavior on every vector in [latex]${\mathcal{F}}^m$[/latex] (where [latex]${\mathcal{F}}$[/latex] is the scalar field) by representing an arbitrary vector [latex]${\mathbf{x}}$[/latex] with respect to [latex]$U$[/latex]. Letting [latex]${\mathbf{x}} = a_1 {\mathbf{b}}_1 + \ldots + a_m {\mathbf{b}}_m$[/latex], [latex]\begin{align*} {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{x}} &= {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} (a_1 {\mathbf{b}}_1 + \ldots + a_m {\mathbf{b}}_m)\\ &= a_1 {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}_1 + \ldots + a_m {\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}} {\mathbf{b}}_m\\ &= a_1 {\begin{bmatrix} \mathbf{w}_1 & \cdots & \mathbf{w}_m \end{bmatrix}} {\mathbf{b}}_1 + \ldots + a_m {\begin{bmatrix} \mathbf{w}_1 & \cdots & \mathbf{w}_m \end{bmatrix}} {\mathbf{b}}_m\\ &= {\begin{bmatrix} \mathbf{w}_1 & \cdots & \mathbf{w}_m \end{bmatrix}} (a_1 {\mathbf{b}}_1 + \ldots + a_m {\mathbf{b}}_m)\\ &= {\begin{bmatrix} \mathbf{w}_1 & \cdots & \mathbf{w}_m \end{bmatrix}} {\mathbf{x}}. \end{align*}[/latex] In particular, the fact that [latex]${\begin{bmatrix} \mathbf{v}_1 & \cdots & \mathbf{v}_m \end{bmatrix}}$[/latex] and [latex]${\begin{bmatrix} \mathbf{w}_1 & \cdots & \mathbf{w}_m \end{bmatrix}}$[/latex] map [latex]$(1, 0,\ldots, 0)$[/latex] to the same vector means that [latex]${\mathbf{v}}_1$[/latex] must equal [latex]${\mathbf{w}}_1$[/latex]. Such an argument holds for every column in turn.

Number: Exercise 1.19
Question: Let [latex]$\lambda$[/latex] be an eigenvalue for [latex]${\mathbb{T}}$[/latex]. Show that the eigenspace of [latex]$\lambda$[/latex] is a subspace.
Answer: Suppose that [latex]${\mathbf{q}}_1$[/latex] and [latex]${\mathbf{q}}_2$[/latex] are both in the eigenspace. For any scalars [latex]$a_1, a_2$[/latex], [latex]\begin{align*} {\mathbb{T}}(a_1 {\mathbf{q}}_1 + a_2 {\mathbf{q}}_2) &= a_1 {\mathbb{T}} {\mathbf{q}}_1 + a_2 {\mathbb{T}} {\mathbf{q}}_2\\ &= a_1 \lambda {\mathbf{q}}_1 + a_2 \lambda {\mathbf{q}}_2\\ &= \lambda (a_1 {\mathbf{q}}_1 + a_2 {\mathbf{q}}_2) \end{align*}[/latex] which confirms that [latex]$a_1 {\mathbf{q}}_1 + a_2 {\mathbf{q}}_2$[/latex] is also an eigenvector for [latex]${\mathbb{T}}$[/latex] with eigenvalue [latex]$\lambda$[/latex].

Number: Exercise 1.20
Question: Suppose [latex]${\mathbb{T}}$[/latex] has eigenvalues [latex]$\lambda_1, \ldots, \lambda_m$[/latex] with corresponding eigenvectors [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_m$[/latex]. Let [latex]$a$[/latex] be a non-zero scalar. Identify eigenvalues and eigenvectors of [latex]$a {\mathbb{T}}$[/latex], i.e. the function that maps any vector [latex]${\mathbf{v}}$[/latex] to [latex]$a$[/latex] times [latex]${\mathbb{T}} {\mathbf{v}}$[/latex].
Answer: Consider the action of [latex]$a {\mathbb{T}}$[/latex] on [latex]${\mathbf{q}}_j$[/latex]. [latex]\begin{align*} [a {\mathbb{T}}] ({\mathbf{q}}_j) &= a ({\mathbb{T}} {\mathbf{q}}_j)\\ &= a \lambda_j {\mathbf{q}}_j \end{align*}[/latex] So [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_m$[/latex] remain eigenvectors, and their eigenvalues are [latex]$a \lambda_1, \ldots, a \lambda_m$[/latex]. Furthermore, no additional eigenvectors for [latex]$a {\mathbb{T}}$[/latex] are introduced because clearly they would also have been eigenvectors for [latex]${\mathbb{T}}$[/latex].

Number: Exercise 1.21
Question: Explain why any linear operator that has [latex]$0$[/latex] as an eigenvalue doesn’t have an inverse function.
Answer: The corresponding eigenspace is a subspace (with dimension at least [latex]$1$[/latex]) that the linear operator maps to [latex]${\mathbf{0}}$[/latex]. Because it maps multiple elements of its domain to the same value, it can’t be invertible.

Number: Exercise 1.22
Question: Let [latex]${\mathbb{T}}^{-1}$[/latex] be the inverse of a linear operator [latex]${\mathbb{T}}$[/latex], that is, [latex]${\mathbb{T}}^{-1} {\mathbb{T}} {\mathbf{v}} = {\mathbf{v}}$[/latex] for every [latex]${\mathbf{v}}$[/latex] in the domain of [latex]${\mathbb{T}}$[/latex]. Show that [latex]${\mathbb{T}}$[/latex] is also the inverse of [latex]${\mathbb{T}}^{-1}$[/latex], that is, [latex]${\mathbb{T}} {\mathbb{T}}^{-1} {\mathbf{w}} = {\mathbf{w}}$[/latex] for every [latex]${\mathbf{w}}$[/latex] in the domain of [latex]${\mathbb{T}}^{-1}$[/latex] (which we’ve defined to be the range of [latex]${\mathbb{T}}$[/latex]).
Answer: We know that [latex]${\mathbf{w}} = {\mathbb{T}} {\mathbf{v}}$[/latex] for some [latex]${\mathbf{v}}$[/latex]; by making this substitution, [latex]\begin{align*} {\mathbb{T}} {\mathbb{T}}^{-1} {\mathbf{w}} &= {\mathbb{T}} {\mathbb{T}}^{-1} {\mathbb{T}} {\mathbf{v}}\\ &= {\mathbb{T}} {\mathbf{v}}\\ &= {\mathbf{w}}. \end{align*}[/latex]

Number: Exercise 1.23
Question: Suppose a linear operator [latex]${\mathbb{T}}$[/latex] has an inverse [latex]${\mathbb{T}}^{-1}$[/latex]. Show that [latex]${\mathbb{T}}^{-1}$[/latex] is also linear: [latex]\begin{align*} {\mathbb{T}}^{-1}(a_1 {\mathbf{w}}_1 + a_2 {\mathbf{w}}_2) &= a_1 {\mathbb{T}}^{-1} {\mathbf{w}}_1 + a_2 {\mathbb{T}}^{-1} {\mathbf{w}}_2 \end{align*}[/latex] for all vectors [latex]${\mathbf{w}}_1, {\mathbf{w}}_2$[/latex] and scalars [latex]$a_1, a_2$[/latex].
Answer: As an inverse function, [latex]${\mathbb{T}}^{-1}$[/latex] maps from [latex]${\mathbb{T}}$[/latex]’s range to its domain. [latex]${\mathbf{w}}_1$[/latex] and [latex]${\mathbf{w}}_2$[/latex] are in the range of [latex]${\mathbb{T}}$[/latex], so we know that they can be represented by [latex]${\mathbf{w}}_1 = {\mathbb{T}} {\mathbf{v}}_1$[/latex] and [latex]${\mathbf{w}}_2 = {\mathbb{T}} {\mathbf{v}}_2$[/latex] for some vectors [latex]${\mathbf{v}}_1$[/latex] and [latex]${\mathbf{v}}_2$[/latex]. [latex]\begin{align*} {\mathbb{T}}^{-1}(a_1 {\mathbf{w}}_1 + a_2 {\mathbf{w}}_2) &= {\mathbb{T}}^{-1}(a_1 {\mathbb{T}} {\mathbf{v}}_1 + a_2 {\mathbb{T}} {\mathbf{v}}_2)\\ &= {\mathbb{T}}^{-1}{\mathbb{T}} (a_1 {\mathbf{v}}_1 + a_2 {\mathbf{v}}_2)\\ &= a_1 {\mathbf{v}}_1 + a_2 {\mathbf{v}}_2\\ &= a_1 {\mathbb{T}}^{-1} {\mathbf{w}}_1 + a_2 {\mathbb{T}}^{-1} {\mathbf{w}}_2 \end{align*}[/latex]

Number: Exercise 1.24
Question: Let [latex]${\mathbb{T}}$[/latex] be a linear operator that has non-zero eigenvalues [latex]$\lambda_1, \ldots, \lambda_n$[/latex] with eigenvectors [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex]. Suppose [latex]${\mathbb{T}}$[/latex] is invertible. Show that [latex]${\mathbb{T}}^{-1}$[/latex] also has [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex] as eigenvectors, and find the corresponding eigenvalues.
Answer: Consider the behavior of the inverse on [latex]${\mathbf{q}}_j$[/latex]. We know that the inverse is supposed to undo the behavior of [latex]${\mathbb{T}}$[/latex], so [latex]${\mathbb{T}}^{-1} {\mathbb{T}} {\mathbf{q}}_j$[/latex] should equal [latex]${\mathbf{q}}_j$[/latex]. [latex]\begin{align*} {\mathbb{T}}^{-1} {\mathbb{T}} {\mathbf{q}}_j &= {\mathbb{T}}^{-1} (\lambda_j {\mathbf{q}}_j)\\ &= \lambda_j {\mathbb{T}}^{-1} {\mathbf{q}}_j \end{align*}[/latex] For [latex]$\lambda_j {\mathbb{T}}^{-1} {\mathbf{q}}_j$[/latex] to equal [latex]${\mathbf{q}}_j$[/latex], we can see that [latex]${\mathbf{q}}_j$[/latex] must be an eigenvector of [latex]${\mathbb{T}}^{-1}$[/latex] with eigenvalue [latex]$1/\lambda_j$[/latex]. Thus [latex]${\mathbb{T}}^{-1}$[/latex] has eigenvalues [latex]$1/\lambda_1, \ldots, 1/\lambda_n$[/latex] with eigenvectors [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex].

Number: Exercise 1.25
Question: Show that inner products are also additive in their second argument: [latex]\begin{align*} \langle {\mathbf{v}}, {\mathbf{w}} + {\mathbf{x}} \rangle = \langle {\mathbf{v}}, {\mathbf{w}} \rangle + \langle {\mathbf{v}}, {\mathbf{x}} \rangle. \end{align*}[/latex]
Answer: The complex conjugate of a sum is equal to the sum of the complex conjugates, so [latex]\begin{align*} \langle {\mathbf{v}}, {\mathbf{w}} + {\mathbf{x}} \rangle &= \overline{\langle {\mathbf{w}} + {\mathbf{x}}, {\mathbf{v}} \rangle}\\ &= \overline{\langle {\mathbf{w}}, {\mathbf{v}} \rangle} + \overline{\langle {\mathbf{x}}, {\mathbf{v}} \rangle}\\ &= \langle {\mathbf{v}}, {\mathbf{w}} \rangle + \langle {\mathbf{v}}, {\mathbf{x}} \rangle. \end{align*}[/latex]

Number: Exercise 1.26
Question: Show that inner products are also homogeneous in their second argument when the scalar is real: for every [latex]$a \in {\pmb{\mathbb{R}}}$[/latex], [latex]\begin{align*} \langle {\mathbf{v}}, a {\mathbf{w}} \rangle = a \langle {\mathbf{v}}, {\mathbf{w}} \rangle. \end{align*}[/latex]
Answer: The complex conjugate of a product is equal to the product of the complex conjugates, and the complex conjugate of a real number is itself, so [latex]\begin{align*} \langle {\mathbf{v}}, a {\mathbf{w}} \rangle &= \overline{\langle a {\mathbf{w}}, {\mathbf{v}} \rangle}\\ &= \overline{a \langle {\mathbf{w}}, {\mathbf{v}} \rangle}\\ &= \overline{a} \overline{\langle {\mathbf{w}}, {\mathbf{v}} \rangle}\\ &= a \langle {\mathbf{v}}, {\mathbf{w}}, \rangle. \end{align*}[/latex]

Number: Exercise 1.27
Question: Show that if [latex]${\mathbf{y}}$[/latex] is orthogonal to every one of [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex], then it is also orthogonal to every vector in their span.
Answer: Let [latex]$b_1 {\mathbf{v}}_1 + \ldots b_m {\mathbf{v}}_m$[/latex] represent an arbitrary vector in the span. By linearity of inner products, its inner product with [latex]${\mathbf{y}}$[/latex] is [latex]\begin{align*} \langle b_1 {\mathbf{v}}_1 + \ldots b_m {\mathbf{v}}_m , {\mathbf{y}}\rangle &= b_1 \underbrace{\langle {\mathbf{v}}_1, {\mathbf{y}} \rangle}_0 + \ldots + b_m \underbrace{\langle {\mathbf{v}}_m, {\mathbf{y}} \rangle}_0\\ &= 0 \end{align*}[/latex] because [latex]${\mathbf{y}}$[/latex] is orthogonal to each of the basis vectors.

Number: Exercise 1.28
Question: Show that [latex]$\| a {\mathbf{v}} \| = |a| \| {\mathbf{v}} \|$[/latex] for any scalar [latex]$a$[/latex].
Answer: From steps in our solution to Exercise 1.26, we can realize that [latex]$\langle {\mathbf{x}}, a {\mathbf{y}} \rangle = \overline{a} \langle {\mathbf{x}}, {\mathbf{y}} \rangle$[/latex]. Using the definition of norm, [latex]\begin{align*} \left\| a {\mathbf{v}} \right\| &= \sqrt{\left\langle a {\mathbf{v}}, a {\mathbf{v}} \right\rangle}\\ &= \sqrt{a \overline{a}} \sqrt{\langle {\mathbf{v}}, {\mathbf{v}} \rangle}\\ &= |a| \| {\mathbf{v}} \|. \end{align*}[/latex] Note that with [latex]$a = b + ic$[/latex], the squared absolute value [latex]$|a|^2$[/latex] is defined to be [latex]$b^2 + c^2$[/latex].

Number: Exercise 1.29
Question: Suppose [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex] are orthogonal to each other and none of them is the zero vector. Show that they must be linearly independent.
Answer: Without loss of generality, we will consider whether or not [latex]${\mathbf{v}}_m$[/latex] can be equal to some linear combination [latex]$b_1 {\mathbf{v}}_1 + \ldots + b_{m-1} {\mathbf{v}}_{m-1}$[/latex]. The inner product of this linear combination with [latex]${\mathbf{v}}_m$[/latex] equals [latex]\begin{align*} \langle b_1 {\mathbf{v}}_1 + \ldots + b_{m-1} {\mathbf{v}}_{m-1}, {\mathbf{v}}_m \rangle &= b_1 \underbrace{\langle {\mathbf{v}}_1, {\mathbf{v}}_m \rangle}_0 + \ldots + b_{m-1} \underbrace{\langle {\mathbf{v}}_{m-1}, {\mathbf{v}}_m \rangle}_0\\ &= 0. \end{align*}[/latex] But the inner product of [latex]${\mathbf{v}}_m$[/latex] with itself is equal to its squared length, which must be greater than zero by the assumption that [latex]${\mathbf{v}}_m$[/latex] isn’t the zero vector. Therefore, no such linear combination can be equal to [latex]${\mathbf{v}}_m$[/latex].

Number: Exercise 1.30
Question: Suppose [latex]$\langle {\mathbf{x}}, {\mathbf{v}} \rangle = \langle {\mathbf{y}}, {\mathbf{v}} \rangle$[/latex] for all [latex]${\mathbf{v}}$[/latex]. Show that [latex]${\mathbf{x}}$[/latex] and [latex]${\mathbf{y}}$[/latex] must be the same vector.
Answer: Subtracting, [latex]$\langle {\mathbf{y}}, {\mathbf{v}} \rangle$[/latex] from both sides of the assumption, [latex]\begin{align*} 0 &= \langle {\mathbf{x}}, {\mathbf{v}} \rangle - \langle {\mathbf{y}}, {\mathbf{v}} \rangle\\ &= \langle {\mathbf{x}} - {\mathbf{y}}, {\mathbf{v}} \rangle \end{align*}[/latex] for all [latex]${\mathbf{v}}$[/latex]. In particular, substitute [latex]${\mathbf{x}} - {\mathbf{y}}$[/latex] for [latex]${\mathbf{v}}$[/latex] to see that [latex]$\| {\mathbf{x}} - {\mathbf{y}} \|^2 = 0$[/latex] which implies that [latex]${\mathbf{x}} - {\mathbf{y}} = {\mathbf{0}}$[/latex], i.e. [latex]${\mathbf{x}} = {\mathbf{y}}$[/latex].

Number: Exercise 1.31
Question: Justify the Pythagorean identity extended to [latex]$m$[/latex] orthogonal vectors [latex]${\mathbf{v}_1, \ldots, \mathbf{v}_m}$[/latex]: [latex]\begin{align*} \| {\mathbf{v}}_1 + \ldots + {\mathbf{v}}_m \|^2 &= \| {\mathbf{v}}_1 \|^2 + \ldots + \| {\mathbf{v}}_m \|^2. \end{align*}[/latex]
Answer: [latex]${\mathbf{v}}_1$[/latex] is orthogonal to [latex]${\mathbf{v}}_2 + \ldots + {\mathbf{v}}_m$[/latex], so by the Pythagorean identity [latex]\begin{align*} \| {\mathbf{v}}_1 + \ldots + {\mathbf{v}}_m \|^2 &= \| {\mathbf{v}}_1 \|^2 + \| {\mathbf{v}}_2 + \ldots + {\mathbf{v}}_m \|^2. \end{align*}[/latex] This logic can be applied repeatedly to bring out one vector at a time leading to the desired result. (For a more formal argument, one can invoke induction.)

Number: Exercise 1.32
Question: Given a non-zero vector [latex]${\mathbf{v}}$[/latex], find the norm of [latex]$\frac{1}{\| {\mathbf{v}} \|} {\mathbf{v}}$[/latex].
Answer: Using Exercise 1.28 and the fact that norms are non-negative, [latex]\begin{align*} \left\| \tfrac{1}{\| {\mathbf{v}} \|} {\mathbf{v}} \right\| &= \tfrac{1}{\| {\mathbf{v}} \|} \| {\mathbf{v}} \|\\ &= 1. \end{align*}[/latex]

Number: Exercise 1.33
Question: Given a unit vector [latex]${\mathbf{u}}$[/latex], find a unique representation of the vector [latex]${\mathbf{y}}$[/latex] as the sum of a vector in the span of [latex]${\mathbf{u}}$[/latex] and a vector orthogonal to the span of [latex]${\mathbf{u}}$[/latex].
Answer: We’ll explicitly construct the desired vector in the span of [latex]${\mathbf{u}}$[/latex]. The vector we seek must equal [latex]${\hat{b}} \, {\mathbf{u}}$[/latex] for some scalar [latex]${\hat{b}} \,$[/latex]. Based on the trivial identity [latex]${\mathbf{y}} = {\hat{b}} \, {\mathbf{u}} + ({\mathbf{y}} - {\hat{b}} \, {\mathbf{u}})$[/latex], we see that we need the second vector [latex]${\mathbf{y}} - {\hat{b}} \, {\mathbf{u}}$[/latex] to be orthogonal to [latex]${\mathbf{u}}$[/latex]. Its inner product with [latex]${\mathbf{u}}$[/latex] is [latex]\begin{align*} \langle {\mathbf{y}} - {\hat{b}} \, {\mathbf{u}}, {\mathbf{u}} \rangle &= \langle {\mathbf{y}}, {\mathbf{u}} \rangle - {\hat{b}} \underbrace{\langle {\mathbf{u}}, {\mathbf{u}} \rangle}_{\| {\mathbf{u}} \|^2=1} \end{align*}[/latex] which is zero precisely when [latex]${\hat{b}} = \langle {\mathbf{y}}, {\mathbf{u}} \rangle$[/latex]. Therefore, [latex]${\mathbf{y}}$[/latex] can be represented as the sum of [latex]$\langle {\mathbf{y}}, {\mathbf{u}} \rangle {\mathbf{u}}$[/latex] which is in the span of [latex]${\mathbf{u}}$[/latex] and [latex]$({\mathbf{y}} - \langle {\mathbf{y}}, {\mathbf{u}} \rangle {\mathbf{u}})$[/latex] which is orthogonal to the span of [latex]${\mathbf{u}}$[/latex].

Number: Exercise 1.34
Question: Given a non-zero vector [latex]${\mathbf{v}}$[/latex], find a unique representation of the vector [latex]${\mathbf{y}}$[/latex] as the sum of a vector in the span of [latex]${\mathbf{v}}$[/latex] and a vector orthogonal to the span of [latex]${\mathbf{v}}$[/latex].
Answer: A vector is in the span of [latex]${\mathbf{v}}$[/latex] if and only if it’s in the span of the unit vector [latex]$\frac{\mathbf{v}}{\| {\mathbf{v}} \|}$[/latex]. Likewise, a vector is orthogonal to the span of [latex]${\mathbf{v}}$[/latex] if and only if it’s orthogonal to the unit vector [latex]$\frac{\mathbf{v}}{\| {\mathbf{v}} \|}$[/latex]. Based on our solution to Exercise 1.33 the part in the span of [latex]${\mathbf{v}}$[/latex] must be [latex]\begin{align*} \left\langle {\mathbf{y}}, \frac{\mathbf{v}}{\| {\mathbf{v}} \|}\right\rangle \frac{\mathbf{v}}{\| {\mathbf{v}} \|} = \frac{\langle {\mathbf{y}}, {\mathbf{v}} \rangle}{\| {\mathbf{v}} \|^2} {\mathbf{v}}. \end{align*}[/latex] Thus the desired representation of [latex]${\mathbf{y}}$[/latex] is [latex]\begin{align*} {\mathbf{y}} &= \underbrace{\frac{\langle {\mathbf{y}}, {\mathbf{v}} \rangle}{\| {\mathbf{v}} \|^2} \mathbf{v}}_{\in \text{span}\{{\mathbf{v}}\}} + \underbrace{\left({\mathbf{y}} - \frac{\langle {\mathbf{y}}, {\mathbf{v}} \rangle}{\| {\mathbf{v}} \|^2} {\mathbf{v}} \right)}_{\perp \text{span}\{{\mathbf{v}}\}}. \end{align*}[/latex]

Number: Exercise 1.35
Question: Let [latex]$\{{\mathbf{u}_1, \ldots, \mathbf{u}_m}\}$[/latex] be an orthonormal basis for [latex]${\mathcal{V}}$[/latex]. Find a unique representation of [latex]${\mathbf{y}} \in {\mathcal{V}}$[/latex] as a linear combination of the basis vectors.
Answer: The correct coefficients can be readily determined thanks to the orthogonality of the terms: [latex]\begin{align*} {\mathbf{y}} = \underbrace{{\hat{b}}_1 {\mathbf{u}}_1}_{\in \text{span}\{{\mathbf{u}}_1\}} + \underbrace{{\hat{b}}_2 {\mathbf{u}}_2 + \ldots + {\hat{b}}_m {\mathbf{u}}_m}_{\perp \text{span}\{{\mathbf{u}}_1\}}. \end{align*}[/latex] By comparison to Exercise 1.33, the first term has to be [latex]$\langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1$[/latex], so its coefficient has to be [latex]${\hat{b}}_1 = \langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle$[/latex]. By reasoning similarly for each of the basis vectors, we conclude that [latex]${\mathbf{y}}$[/latex] must have the unique representation [latex]\begin{align*} {\mathbf{y}} = \langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1 + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle {\mathbf{u}}_m. \end{align*}[/latex]

Number: Exercise 1.36
Question: Let [latex]$\{{\mathbf{u}_1, \ldots, \mathbf{u}_m}\}$[/latex] be an orthonormal basis for a real vector space [latex]${\mathcal{V}}$[/latex]. Show that the inner product between [latex]${\mathbf{x}}$[/latex] and [latex]${\mathbf{y}}$[/latex] equals the sum of the product of their squared coordinates with respect to [latex]${\mathbf{u}_1, \ldots, \mathbf{u}_m}$[/latex]: [latex]\begin{align*} \langle {\mathbf{x}}, {\mathbf{y}} \rangle &= \sum_i (\langle {\mathbf{x}}, {\mathbf{u}}_i \rangle \langle {\mathbf{y}}, {\mathbf{u}}_i \rangle). \end{align*}[/latex]
Answer: We’ll use the orthonormal basis representation (Exercise 1.35) to expand [latex]${\mathbf{y}}$[/latex] use linearity of inner products. [latex]\begin{align*} \langle {\mathbf{x}}, {\mathbf{y}} \rangle &= \left\langle {\mathbf{x}}, \langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1 + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle {\mathbf{u}}_m \right\rangle\\ &= \left\langle {\mathbf{x}}, \langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1 \right\rangle + \ldots + \left\langle {\mathbf{x}}, \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle {\mathbf{u}}_m \right\rangle\\ &= \langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle \langle {\mathbf{x}}, {\mathbf{u}}_1 \rangle + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle \langle {\mathbf{x}}, {\mathbf{u}}_m \rangle. \end{align*}[/latex]

Number: Exercise 1.37
Question: Let [latex]$\{{\mathbf{u}_1, \ldots, \mathbf{u}_m}\}$[/latex] be an orthonormal basis for a real vector space [latex]${\mathcal{V}}$[/latex], and let [latex]${\mathbf{y}} \in {\mathcal{V}}$[/latex]. Consider the approximation [latex]${\mathbf{\hat{y}}} := \langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1 + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_k \rangle {\mathbf{u}}_k$[/latex] with [latex]$k \leq m$[/latex]. Use Parseval’s identity to derive a simple formula for the squared norm of [latex]${\mathbf{y}} - {\mathbf{\hat{y}}}$[/latex], which we might call the squared approximation error.
Answer: Representing [latex]${\mathbf{y}}$[/latex] with respect to the orthonormal basis, we find that the difference between the vectors is [latex]\begin{align*} {\mathbf{y}} - {\mathbf{\hat{y}}} &= (\langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1 + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle {\mathbf{u}}_m) - (\langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1 + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_k \rangle {\mathbf{u}}_k)\\ &= \langle {\mathbf{y}}, {\mathbf{u}}_{k+1} \rangle {\mathbf{u}}_{k+1} + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle {\mathbf{u}}_m. \end{align*}[/latex] Its squared norm is the sum of its squared coordinates, so [latex]\begin{align*} \| {\mathbf{y}} - {\mathbf{\hat{y}}} \|^2 &= \langle {\mathbf{y}}, {\mathbf{u}}_{k+1} \rangle^2 + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle^2. \end{align*}[/latex]

Number: Exercise 1.38
Question: Let [latex]$\{{\mathbf{u}_1, \ldots, \mathbf{u}_m}\}$[/latex] be an orthonormal basis for a real vector space [latex]${\mathcal{V}}$[/latex], and let [latex]${\mathbf{y}} \in {\mathcal{V}}$[/latex]. Explain which term in the representation [latex]$\langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1 + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle {\mathbf{u}}_m$[/latex] best approximates [latex]${\mathbf{y}}$[/latex] in the sense that it results in the smallest approximation error [latex]$\| {\mathbf{y}} - \langle {\mathbf{y}}, {\mathbf{u}}_j \rangle {\mathbf{u}}_j \|$[/latex].
Answer: Based on Exercise 1.37, the squared approximation error [latex]$\| {\mathbf{y}} - \langle {\mathbf{y}}, {\mathbf{u}}_j \rangle {\mathbf{u}}_j \|^2$[/latex] is equal to the sum of the squares of the other coefficients [latex]$\sum_{i \neq j} \langle {\mathbf{y}}, {\mathbf{u}}_i \rangle^2$[/latex]. Therefore, the approximation error is minimized if we use the term with the largest squared coefficient.

Number: Exercise 1.39
Question: Given a subspace [latex]${\mathcal{S}}$[/latex], show that [latex]${\mathcal{S}}^\perp$[/latex] is also a subspace.
Answer: Let [latex]${\mathbf{v}}_1, {\mathbf{v}}_2 \in {\mathcal{S}}^\perp$[/latex], and let [latex]$b_1$[/latex] and [latex]$b_2$[/latex] be scalars. We need to show that the linear combination [latex]$b_1 {\mathbf{v}}_1 + b_2 {\mathbf{v}}_2$[/latex] is also in [latex]${\mathcal{S}}^\perp$[/latex]. Letting [latex]${\mathbf{w}}$[/latex] be an arbitrary vector in [latex]${\mathcal{S}}$[/latex], [latex]\begin{align*} \langle b_1 {\mathbf{v}}_1 + b_2 {\mathbf{v}}_2, {\mathbf{w}} \rangle &= b_1 \underbrace{\langle {\mathbf{v}}_1, {\mathbf{w}} \rangle}_0 + b_2 \underbrace{\langle {\mathbf{v}}_2, {\mathbf{w}} \rangle}_0\\ &= 0. \end{align*}[/latex]

Number: Exercise 1.40
Question: Let [latex]${\mathbf{\hat{y}}}$[/latex] be the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}$[/latex]. Use the Pythagorean identity to show that the vector in [latex]${\mathcal{S}}$[/latex] that is closest to [latex]${\mathbf{y}}$[/latex] is [latex]${\mathbf{\hat{y}}}$[/latex].
Answer: Let [latex]${\mathbf{v}}$[/latex] be an arbitrary vector in [latex]${\mathcal{S}}$[/latex]. Realizing that [latex]${\mathbf{\hat{y}}} - {\mathbf{v}}$[/latex] is in [latex]${\mathcal{S}}$[/latex] and that [latex]${\mathbf{y}} - {\mathbf{\hat{y}}}$[/latex] is orthogonal to [latex]${\mathcal{S}}$[/latex], we observe a right triangle (Figure 1.3) with sides [latex]${\mathbf{y}} - {\mathbf{v}}$[/latex], [latex]${\mathbf{\hat{y}}} - {\mathbf{v}}$[/latex], and [latex]${\mathbf{y}} - {\mathbf{\hat{y}}}$[/latex]. By the Pythagorean identity, [latex]\begin{align*} \| {\mathbf{y}} - {\mathbf{v}} \|^2 &= \| {\mathbf{y}} - {\mathbf{\hat{y}}} \|^2 + \| {\mathbf{\hat{y}}} - {\mathbf{v}} \|^2. \end{align*}[/latex] The first term on the right doesn’t depend on the choice of [latex]${\mathbf{v}}$[/latex], so the quantity is uniquely minimized by choosing [latex]${\mathbf{v}}$[/latex] equal to [latex]${\mathbf{\hat{y}}}$[/latex] to make the second term zero.

Number: Exercise 1.41
Question: Let [latex]${\mathcal{S}}_1$[/latex] and [latex]${\mathcal{S}}_2$[/latex] be subspaces that are orthogonal to each other, and let [latex]${\mathcal{S}}$[/latex] be the span of their union. If [latex]${\mathbf{\hat{y}}}_1$[/latex] and [latex]${\mathbf{\hat{y}}}_2$[/latex] are the orthogonal projections of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}_1$[/latex] and [latex]${\mathcal{S}}_2$[/latex], show that the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}$[/latex] is [latex]${\mathbf{\hat{y}}}_1 + {\mathbf{\hat{y}}}_2$[/latex].
Answer: For an arbitrary [latex]${\mathbf{v}} \in {\mathcal{S}}$[/latex], we need to establish that [latex]\begin{align*} {\mathbf{y}} - ({\mathbf{\hat{y}}}_1 + {\mathbf{\hat{y}}}_2) \perp {\mathbf{v}}. \end{align*}[/latex] Every vector in the span of [latex]${\mathcal{S}}_1 \cup {\mathcal{S}}_2$[/latex] can be represented as the sum of a vector in [latex]${\mathcal{S}}_1$[/latex] and a vector in [latex]${\mathcal{S}}_2$[/latex]. Making use of this fact, we let [latex]${\mathbf{v}} = {\mathbf{v}}_1 + {\mathbf{v}}_2$[/latex] with [latex]${\mathbf{v}}_1 \in {\mathcal{S}}_1$[/latex] and [latex]${\mathbf{v}}_2 \in {\mathcal{S}}_2$[/latex]. [latex]\begin{align*} \langle {\mathbf{v}}, {\mathbf{y}} - ({\mathbf{\hat{y}}}_1 + {\mathbf{\hat{y}}}_2) \rangle &= \langle {\mathbf{v}}_1 + {\mathbf{v}}_2, {\mathbf{y}} - {\mathbf{\hat{y}}}_1 - {\mathbf{\hat{y}}}_2 \rangle\\ &= \langle {\mathbf{v}}_1, {\mathbf{y}} - {\mathbf{\hat{y}}}_1 - {\mathbf{\hat{y}}}_2 \rangle + \langle {\mathbf{v}}_2, {\mathbf{y}} - {\mathbf{\hat{y}}}_1 - {\mathbf{\hat{y}}}_2 \rangle\\ &= \underbrace{\langle {\mathbf{v}}_1, {\mathbf{y}} - {\mathbf{\hat{y}}}_1 \rangle}_0 + \underbrace{\langle {\mathbf{v}}_2, {\mathbf{y}} - {\mathbf{\hat{y}}}_2 \rangle}_0\\ &= 0 \end{align*}[/latex]

Number: Exercise 1.42
Question: Let [latex]${\mathcal{S}}$[/latex] be a subspace of [latex]${\mathcal{V}}$[/latex], and let [latex]${\mathbf{u}_1, \ldots, \mathbf{u}_m}$[/latex] comprise an orthonormal basis for [latex]${\mathcal{S}}$[/latex]. Given any [latex]${\mathbf{y}} \in {\mathcal{V}}$[/latex], show that [latex]${\mathbf{\hat{y}}} := \langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1 + \ldots + \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle {\mathbf{u}}_m$[/latex] is the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}$[/latex].
Answer: From Exercise 1.41, we understand that the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}$[/latex] equals the sum of its orthogonal projections onto the spans of the orthonormal basis vectors. The representations of these orthogonal projections as [latex]$\langle {\mathbf{y}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1, \ldots, \langle {\mathbf{y}}, {\mathbf{u}}_m \rangle {\mathbf{u}}_m$[/latex] comes from Exercise 1.33.

Number: Exercise 1.43
Question: Suppose [latex]${\mathbf{\hat{y}}}_1$[/latex] and [latex]${\mathbf{\hat{y}}}_2$[/latex] are the orthogonal projections of [latex]${\mathbf{y}}_1$[/latex] and [latex]${\mathbf{y}}_2$[/latex] onto [latex]${\mathcal{S}}$[/latex]. With scalars [latex]$a_1$[/latex] and [latex]$a_2$[/latex], find the orthogonal projection of [latex]$a_1 {\mathbf{y}}_1 + a_2 {\mathbf{y}}_2$[/latex] onto [latex]${\mathcal{S}}$[/latex].
Answer: We can write out each vector in terms of its orthogonal projections onto [latex]${\mathcal{S}}$[/latex] and [latex]${\mathcal{S}}^\perp$[/latex], then regroup the terms. [latex]\begin{align*} a_1 {\mathbf{y}}_1 + a_2 {\mathbf{y}}_2 &= a_1 [{\mathbf{\hat{y}}}_1 + ({\mathbf{y}}_1 - {\mathbf{\hat{y}}}_1)] + a_2 [{\mathbf{\hat{y}}}_2 + ({\mathbf{y}}_2 - {\mathbf{\hat{y}}}_2)]\\ &= \underbrace{(a_1 {\mathbf{\hat{y}}}_1 + a_2 {\mathbf{\hat{y}}}_2)}_{\in \mathcal{S}} + \underbrace{[a_1 ({\mathbf{y}}_1 - {\mathbf{\hat{y}}}_1) + a_2 ({\mathbf{y}}_2 - {\mathbf{\hat{y}}}_2)]}_{\perp \mathcal{S}} \end{align*}[/latex] This shows that [latex]$a_1 {\mathbf{\hat{y}}}_1 + a_2 {\mathbf{\hat{y}}}_2$[/latex] is the orthogonal projection of [latex]$a_1 {\mathbf{y}}_1 + a_2 {\mathbf{y}}_2$[/latex] onto [latex]${\mathcal{S}}$[/latex]. In other words, the orthogonal projection of a linear combination is the linear combination of the orthogonal projections.

Number: Exercise 1.44
Question: Let [latex]${\mathbf{\hat{y}}}$[/latex] be the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}$[/latex]. How do we know that [latex]${\mathbf{y}} - {\mathbf{\hat{y}}}$[/latex] is the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}^\perp$[/latex]?
Answer: We know that [latex]${\mathbf{y}} = {\mathbf{\hat{y}}} + ({\mathbf{y}} - {\mathbf{\hat{y}}})$[/latex] with [latex]${\mathbf{\hat{y}}} \in {\mathcal{S}}$[/latex] and [latex]${\mathbf{y}} - {\mathbf{\hat{y}}} \in {\mathcal{S}}^\perp$[/latex] by definition of orthogonal projection. Of course, by definition of orthogonal complement, [latex]${\mathbf{\hat{y}}} \perp {\mathcal{S}}^\perp$[/latex], so that same representation shows that [latex]${\mathbf{y}} - {\mathbf{\hat{y}}}$[/latex] is the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}^\perp$[/latex].

Number: Exercise 1.45
Question: Let [latex]${\mathcal{S}}$[/latex] be an [latex]$m$[/latex]-dimensional subspace of a [latex]$d$[/latex]-dimensional vector space [latex]${\mathcal{V}}$[/latex]. Verify that the dimension of [latex]${\mathcal{S}}^\perp$[/latex] is [latex]$d-m$[/latex].
Answer: Let [latex]$B$[/latex] be a basis for [latex]${\mathcal{S}}$[/latex]; it contains [latex]$m$[/latex] vectors. Suppose there exist more than [latex]$d-m$[/latex] linearly independent vectors in [latex]${\mathcal{S}}^\perp$[/latex]. All of them are also linearly independent of [latex]$B$[/latex], so the two bases taken together would contain a total of more than [latex]$d$[/latex] linearly independent vectors in [latex]${\mathcal{V}}$[/latex] which is impossible. On the other hand, suppose [latex]${\mathcal{S}}^\perp$[/latex] has a basis of fewer than [latex]$d-m$[/latex] vectors. Then that basis, taken together with [latex]$B$[/latex] would have fewer than [latex]$d$[/latex] vectors, so it wouldn’t span [latex]${\mathcal{V}}$[/latex]. However, this can’t be true because we’ve seen than any vector in [latex]${\mathcal{V}}$[/latex] can be represented as the sum of a vector in [latex]${\mathcal{S}}$[/latex] and a vector in [latex]${\mathcal{S}}^\perp$[/latex].

Number: Exercise 1.46
Question: Let [latex]${\mathbb{H}}$[/latex] be an orthogonal projection operator onto [latex]${\mathcal{S}}$[/latex]. Show that every vector in [latex]${\mathcal{S}}$[/latex] is an eigenvector of [latex]${\mathbb{H}}$[/latex].
Answer: If [latex]${\mathbf{v}}$[/latex] is in [latex]${\mathcal{S}}$[/latex], then clearly [latex]${\mathbf{v}} = {\mathbf{v}} + {\mathbf{0}}$[/latex] is the unique representation of [latex]${\mathbf{v}}$[/latex] as the sum of a vector in [latex]${\mathcal{S}}$[/latex] and a vector orthogonal to [latex]${\mathcal{S}}$[/latex]. Therefore [latex]${\mathbb{H}} {\mathbf{v}} = {\mathbf{v}}$[/latex], which means that [latex]${\mathbf{v}}$[/latex] is an eigenvector with eigenvalue [latex]$1$[/latex].

Number: Exercise 1.47
Question: Let [latex]${\mathbb{H}}$[/latex] be the orthogonal projection operator onto [latex]${\mathcal{S}}$[/latex]. Show that every vector in [latex]${\mathcal{S}}^\perp$[/latex] is an eigenvector of [latex]${\mathbb{H}}$[/latex].
Answer: If [latex]${\mathbf{v}} \perp {\mathcal{S}}$[/latex], then clearly [latex]${\mathbf{v}} = {\mathbf{0}} + {\mathbf{v}}$[/latex] is the unique representation of [latex]${\mathbf{v}}$[/latex] as the sum of a vector in [latex]${\mathcal{S}}$[/latex] and a vector orthogonal to [latex]${\mathcal{S}}$[/latex]. Therefore [latex]${\mathbb{H}} {\mathbf{v}} = {\mathbf{0}}$[/latex], which means that [latex]${\mathbf{v}}$[/latex] is an eigenvector with eigenvalue [latex]$0$[/latex].

Number: Exercise 1.48
Question: Show that every orthogonal projection operator is idempotent.
Answer: Let [latex]${\mathbb{H}}$[/latex] be the orthogonal projection operator onto [latex]${\mathcal{S}}$[/latex], and let [latex]${\mathbf{\hat{y}}}$[/latex] be the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}$[/latex]. Because [latex]${\mathbf{\hat{y}}}$[/latex] is in [latex]${\mathcal{S}}$[/latex], [latex]${\mathbb{H}}$[/latex] maps it to itself. [latex]\begin{align*} [{\mathbb{H}} \circ {\mathbb{H}}] {\mathbf{y}} &= {\mathbb{H}} ({\mathbb{H}} {\mathbf{y}})\\ &= {\mathbb{H}} {\mathbf{\hat{y}}}\\ &= {\mathbf{\hat{y}}} \end{align*}[/latex] The action of [latex]${\mathbb{H}} \circ {\mathbb{H}}$[/latex] is exactly the same as that of [latex]${\mathbb{H}}$[/latex] on every vector, so they’re the same operator.

Number: Exercise 1.49
Question: Let [latex]${\mathbb{H}}_1$[/latex] be the orthogonal projection operator onto [latex]${\mathcal{S}}_1$[/latex], and let [latex]${\mathbb{H}}_0$[/latex] be the orthogonal projection operator onto [latex]${\mathcal{S}}_0 \subseteq {\mathcal{S}}_1$[/latex]. Explain why [latex]${\mathbb{H}}_1 - {\mathbb{H}}_0$[/latex] is the orthogonal projection operator onto the orthogonal complement of [latex]${\mathcal{S}}_0$[/latex] within [latex]${\mathcal{S}}_1$[/latex].
Answer: The operator [latex]${\mathbb{H}}_1 - {\mathbb{H}}_0$[/latex] evaluated at [latex]${\mathbf{y}}$[/latex] has the value [latex]${\mathbb{H}}_1 {\mathbf{y}} - {\mathbb{H}}_0 {\mathbf{y}}$[/latex]. From our discussion regarding Equation 1.4, we know that this is precisely the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto the orthogonal complement of [latex]${\mathcal{S}}_0$[/latex] within [latex]${\mathcal{S}}_1$[/latex].

Number: Exercise 1.50
Question: Let [latex]${\mathcal{S}}_0 \subseteq {\mathcal{S}}_1$[/latex] be subspaces, and let [latex]${\mathbb{H}}_0$[/latex] and [latex]${\mathbb{H}}_1$[/latex] be orthogonal projection operators onto [latex]${\mathcal{S}}_0$[/latex] and [latex]${\mathcal{S}}_1$[/latex] respectively. Explain why [latex]${\mathbb{H}}_0 \circ {\mathbb{H}}_1 = {\mathbb{H}}_1 \circ {\mathbb{H}}_0 = {\mathbb{H}}_0$[/latex].
Answer: In our discussion regarding Equation 1.4, we realized that the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}_0$[/latex] is the same as the orthogonal projection of [latex]${\mathbb{H}}_1 {\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}_0$[/latex]. In other words, it doesn’t matter which order you compose the operators, you end up at [latex]${\mathbb{H}}_0 {\mathbf{y}}$[/latex] either way.

Number: Exercise 1.51
Question: Let [latex]${\mathbb{M}} \in {\mathcal{F}}^{n \times m}$[/latex] and [latex]${\mathbb{L}} \in {\mathcal{F}}^{m \times n}$[/latex]. Show that the trace of [latex]${\mathbb{M}} {\mathbb{L}}$[/latex] is equal to the trace of [latex]${\mathbb{L}} {\mathbb{M}}$[/latex].
Answer: The [latex]$i$[/latex]th diagonal entry of [latex]${\mathbb{M}} {\mathbb{L}}$[/latex] is [latex]$\sum_{j=1}^m {\mathbb{M}}_{i, j} {\mathbb{L}}_{j, i}$[/latex]. We express the trace as the sum of these diagonals then reverse the order of the summations. [latex]\begin{align*} \text{tr} \, {\mathbb{M}} {\mathbb{L}} &= \sum_{i=1}^n \sum_{j=1}^m {\mathbb{M}}_{i, j} {\mathbb{L}}_{j, i}\\ &= \sum_{j=1}^m \sum_{i=1}^n {\mathbb{L}}_{j, i} {\mathbb{M}}_{i, j}\\ &= \text{tr} \, {\mathbb{L}} {\mathbb{M}} \end{align*}[/latex]

Number: Exercise 1.52
Question: Let [latex]${\mathbb{M}}$[/latex] and [latex]${\mathbb{L}}$[/latex] be matrices such that the product [latex]${\mathbb{M}} {\mathbb{L}}$[/latex] is well-defined. Show that [latex]$({\mathbb{M}} {\mathbb{L}})^T = {\mathbb{L}}^T {\mathbb{M}}^T$[/latex].
Answer: The [latex]$(i,j)$[/latex] entry of [latex]$({\mathbb{M}} {\mathbb{L}})^T$[/latex] is the [latex]$(j, i)$[/latex] entry of [latex]${\mathbb{M}} {\mathbb{L}}$[/latex], which is [latex]$\sum_k {\mathbb{M}}_{j, k} {\mathbb{L}}_{k, i}$[/latex]. The [latex]$(i, j)$[/latex] entry of [latex]${\mathbb{L}}^T {\mathbb{M}}^T$[/latex] works out to be the same: [latex]\begin{align*} \sum_k ({\mathbb{L}}^T)_{i, k} ({\mathbb{M}}^T)_{k, j} &= \sum_k {\mathbb{L}}_{k, i} {\mathbb{M}}_{j, k}. \end{align*}[/latex]

Number: Exercise 1.53
Question: If the columns of [latex]${\mathbb{U}}$[/latex] are orthonormal, show that [latex]${\mathbb{U}}^T {\mathbb{U}}$[/latex] equals the identity matrix [latex]${\mathbb{I}}$[/latex].
Answer: Letting [latex]${\mathbf{u}}_1, \ldots, {\mathbf{u}}_m$[/latex] denote the columns, [latex]\begin{align*} {\mathbb{U}}^T {\mathbb{U}} &= \begin{bmatrix}- & {\mathbf{u}}_1 & -\\ & \vdots & \\ - & {\mathbf{u}}_m & - \end{bmatrix} \begin{bmatrix}| & & |\\ {\mathbf{u}}_1 & \ldots & {\mathbf{u}}_m\\ | & & |\end{bmatrix}\\ &= \begin{bmatrix}{\mathbf{u}}_1^T {\mathbf{u}}_1 & \cdots & {\mathbf{u}}_1^T {\mathbf{u}}_n\\ \vdots & \ddots & \vdots \\ {\mathbf{u}}_n^T {\mathbf{u}}_1 & \cdots & {\mathbf{u}}_n^T {\mathbf{u}}_n \end{bmatrix}\\ &= \begin{bmatrix}1 & & \\ & \ddots & \\ & & 1 \end{bmatrix} \end{align*}[/latex] with every off-diagonal entry equal to zero.

Number: Exercise 1.54
Question: Show that [latex]${\mathbb{M}}^T {\mathbb{M}}$[/latex] is symmetric.
Answer: The transpose of a product of matrices is equal to the product of their transposes multiplied in the reverse order (Exercise 1.52). Thus [latex]\begin{align*} ({\mathbb{M}}^T {\mathbb{M}})^T &= ({\mathbb{M}})^T ({\mathbb{M}}^T)^T\\ &= {\mathbb{M}}^T {\mathbb{M}}. \end{align*}[/latex]

Number: Exercise 1.55
Question: Show that a square matrix is invertible if and only if its columns are linearly independent.
Answer: Let [latex]${\mathbb{M}}$[/latex] be a square matrix. First, assume it’s invertible. Then what linear combinations satisfy [latex]${\mathbb{M}} {\mathbf{b}} = {\mathbf{0}}$[/latex]? Multiplying both sides by the inverse, we see that the coefficients [latex]${\mathbf{b}} = {\mathbb{M}}^{-1} {\mathbf{0}}$[/latex] must be the zero vector.
Next, suppose the [latex]$n$[/latex] columns of [latex]${\mathbb{M}}$[/latex] are linearly independent. Then for each canonical basis vector [latex]${\mathbf{e}}_j$[/latex], there is some coefficient vector [latex]${\mathbf{b}}_j$[/latex] such that [latex]${\mathbb{M}} {\mathbf{b}}_j = {\mathbf{e}}_j$[/latex]. The matrix with these coefficient vectors as its columns is the inverse of [latex]${\mathbb{M}}$[/latex]. [latex]\begin{align*} {\mathbb{M}} \begin{bmatrix} | & & | \\ {\mathbf{b}}_1 & \cdots & {\mathbf{b}}_n \\ | & & | \end{bmatrix} &= \begin{bmatrix} | & & | \\ {\mathbb{M}} {\mathbf{b}}_1 & \cdots & {\mathbb{M}} {\mathbf{b}}_n \\ | & & | \end{bmatrix}\\ &= \underbrace{\begin{bmatrix} | & & | \\ {\mathbf{e}}_1 & \cdots & {\mathbf{e}}_n \\ | & & | \end{bmatrix}}_{{\mathbb{I}}_n} \end{align*}[/latex]

Number: Exercise 1.56
Question: Suppose [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times n}$[/latex] has orthonormal eigenvectors [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex] with eigenvalues [latex]$\lambda_1, \ldots, \lambda_n$[/latex] Show that [latex]${\mathbb{M}}$[/latex] can’t have any other eigenvalues.
Answer: Let us check what would be required for an arbitrary vector [latex]${\mathbf{w}}$[/latex] to be an eigenvector for [latex]${\mathbb{M}}$[/latex]. We can express [latex]${\mathbf{w}}$[/latex] with respect to the eigenvector basis: [latex]\begin{align*} {\mathbb{M}} {\mathbf{w}} &= {\mathbb{M}} ( \langle {\mathbf{w}}, {\mathbf{q}}_1 \rangle {\mathbf{q}}_1 + \ldots + \langle {\mathbf{w}}, {\mathbf{q}}_n \rangle {\mathbf{q}}_n )\\ &= \langle {\mathbf{w}}, {\mathbf{q}}_1 \rangle {\mathbb{M}} {\mathbf{q}}_1 + \ldots + \langle {\mathbf{w}}, {\mathbf{q}}_n \rangle {\mathbb{M}} {\mathbf{q}}_n\\ &= \langle {\mathbf{w}}, {\mathbf{q}}_1 \rangle \lambda_1 {\mathbf{q}}_1 + \ldots + \langle {\mathbf{w}}, {\mathbf{q}}_n \rangle \lambda_n {\mathbf{q}}_n. \end{align*}[/latex] This is only proportional to [latex]${\mathbf{w}} = \langle {\mathbf{w}}, {\mathbf{q}}_1 \rangle {\mathbf{q}}_1 + \ldots + \langle {\mathbf{w}}, {\mathbf{q}}_n \rangle {\mathbf{q}}_n$[/latex] if all of the non-zero terms share the same eigenvalue coefficient. That coefficient, which is one of [latex]$\lambda_1, \ldots, \lambda_n$[/latex], would be the eigenvalue of [latex]${\mathbf{w}}$[/latex].

Number: Exercise 1.57
Question: Let [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex] be an orthonormal basis for [latex]${\pmb{\mathbb{R}}}^n$[/latex]. Show that [latex]${\mathbb{M}}$[/latex] has the spectral decomposition [latex]\begin{align*} {\mathbb{M}} = \lambda_1 {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \lambda_n {\mathbf{q}}_n {\mathbf{q}}_n^T \end{align*}[/latex] if and only if [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex] are eigenvectors for [latex]${\mathbb{M}}$[/latex] with eigenvalues [latex]$\lambda_1, \ldots, \lambda_n$[/latex].
Answer: Let’s figure out the behavior of [latex]$\lambda_1 {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \lambda_n {\mathbf{q}}_n {\mathbf{q}}_n^T$[/latex] on the basis vectors. [latex]\begin{align*} (\lambda_1 {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \lambda_n {\mathbf{q}}_n {\mathbf{q}}_n^T) {\mathbf{q}}_1 &= \lambda_1 {\mathbf{q}}_1 \underbrace{{\mathbf{q}}_1^T {\mathbf{q}}_1}_{\|{\mathbf{q}}_1\|^2=1} + \ldots + \lambda_n {\mathbf{q}}_n \underbrace{{\mathbf{q}}_n^T {\mathbf{q}}_1}_0\\ &= \lambda_1 {\mathbf{q}}_1 \end{align*}[/latex] meaning [latex]${\mathbf{q}}_1$[/latex] is also an eigenvector of this matrix with eigenvalue [latex]$\lambda_1$[/latex]. Likewise for [latex]${\mathbf{q}}_2, \ldots, {\mathbf{q}}_n$[/latex]. By establishing that [latex]${\mathbb{M}}$[/latex] and [latex]$\lambda_1 {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \lambda_n {\mathbf{q}}_n {\mathbf{q}}_n^T$[/latex] behave the same on a basis, we see that they must be the same matrix by Exercise 1.18.

Number: Exercise 1.58
Question: Let [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times n}$[/latex] be a symmetric matrix with non-negative eigenvalues [latex]$\lambda_1, \ldots, \lambda_n$[/latex] and corresponding orthonormal eigenvectors [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex]. Show that the symmetric matrix that has eigenvalues [latex]$\sqrt{\lambda_1}, \ldots, \sqrt{\lambda_n}$[/latex] with eigenvectors [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex] is the square root of [latex]${\mathbb{M}}$[/latex] (denoted [latex]${\mathbb{M}}^{1/2}$[/latex]) in the sense that [latex]${\mathbb{M}}^{1/2} {\mathbb{M}}^{1/2} = {\mathbb{M}}$[/latex].
Answer: Using a spectral decomposition, we multiply the proposed square root matrix by itself: [latex]\begin{align*} {\mathbb{M}}^{1/2} {\mathbb{M}}^{1/2} &= {\mathbb{M}}^{1/2} (\sqrt{\lambda_1} {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \sqrt{\lambda_n} {\mathbf{q}}_n {\mathbf{q}}_n^T)\\ &= \sqrt{\lambda_1} \underbrace{{\mathbb{M}}^{1/2} {\mathbf{q}}_1}_{\sqrt{\lambda_1} {\mathbf{q}}_1} {\mathbf{q}}_1^T + \ldots + \sqrt{\lambda_n} \underbrace{{\mathbb{M}}^{1/2} {\mathbf{q}}_n}_{\sqrt{\lambda_n} {\mathbf{q}}_n} {\mathbf{q}}_n^T\\ &= \lambda_1 {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \lambda_n {\mathbf{q}}_n {\mathbf{q}}_n^T\\ &= {\mathbb{M}}. \end{align*}[/latex]

Number: Exercise 1.59
Question: Let [latex]${\mathbb{M}}$[/latex] be a symmetric and invertible real matrix. Show that [latex]${\mathbb{M}}^{-1}$[/latex] is also a symmetric real matrix.
Answer: Let [latex]$\lambda_1, \ldots, \lambda_n$[/latex] and [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex] be eigenvalues and orthonormal eigenvectors of [latex]${\mathbb{M}}$[/latex]. Based on Exercise 1.24, we can deduce that [latex]${\mathbb{M}}^{-1}$[/latex] has the spectral decomposition [latex]\begin{align*} {\mathbb{M}}^{-1} &= \tfrac{1}{\lambda_1} {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \tfrac{1}{\lambda_n} {\mathbf{q}}_n {\mathbf{q}}_n^T. \end{align*}[/latex] Because [latex]${\mathbb{M}}^{-1}$[/latex] is a linear combination of symmetric real matrices (see Exercise 1.54), it’s clearly a symmetric real matrix as well.

Number: Exercise 1.60
Question: Let [latex]${\mathbb{M}}$[/latex] be a symmetric real matrix. Show that the trace of [latex]${\mathbb{M}}$[/latex] equals the sum of its eigenvalues.
Answer: We’ll use the matrix form of spectral decomposition [latex]${\mathbb{M}} = {\mathbb{Q}} \Lambda {\mathbb{Q}}^T$[/latex] and the cyclic permutation property of trace (Exercise 1.51). [latex]\begin{align*} \text{tr} \, {\mathbb{M}} &= \text{tr} \, ({\mathbb{Q}} \Lambda {\mathbb{Q}}^T)\\ &= \text{tr} \, (\underbrace{{\mathbb{Q}}^T \mathbb{Q}}_{{\mathbb{I}}_n} \Lambda)\\ &= \text{tr} \, \Lambda \end{align*}[/latex]

Number: Exercise 1.61
Question: Let [latex]${\mathbb{M}}$[/latex] be an [latex]$n \times m$[/latex] real matrix. How do you know that the number of terms in a singular value decomposition of [latex]${\mathbb{M}}$[/latex] can’t be more than [latex]$\min(n, m)$[/latex].
Answer: The vectors [latex]${\mathbf{u}}_1, \ldots, {\mathbf{u}}_d \in {\pmb{\mathbb{R}}}^n$[/latex] are linearly independent, so there can’t be more than [latex]$n$[/latex] of them. Likewise, the vectors [latex]${\mathbf{v}}_1, \ldots, {\mathbf{v}}_d \in {\pmb{\mathbb{R}}}^m$[/latex] are linearly independent, so there can’t be more than [latex]$m$[/latex] of them.

Number: Exercise 1.62
Question: Use a singular value decomposition for [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] to find a spectral decomposition of [latex]${\mathbb{M}}^T {\mathbb{M}}$[/latex].
Answer: Writing [latex]${\mathbb{M}} = {\mathbb{U}} {\mathbb{S}} {\mathbb{V}}^T$[/latex], [latex]\begin{align*} {\mathbb{M}}^T {\mathbb{M}} &= ({\mathbb{U}} {\mathbb{S}} {\mathbb{V}}^T)^T ({\mathbb{U}} {\mathbb{S}} {\mathbb{V}}^T)\\ &= {\mathbb{V}} {\mathbb{S}}^T \underbrace{{\mathbb{U}}^T \mathbb{U}}_{\mathbb{I}} {\mathbb{S}} {\mathbb{V}}^T\\ &= {\mathbb{V}} {\mathbb{S}}^2 {\mathbb{V}}^T. \end{align*}[/latex] By comparison to the matrix form of spectral decomposition, we see that [latex]${\mathbb{M}}^T {\mathbb{M}}$[/latex] has eigenvalues equal to the squares of the singular values of [latex]${\mathbb{M}}$[/latex], and the corresponding eigenvectors are the columns of [latex]${\mathbb{V}}$[/latex].

Number: Exercise 1.63
Question: Explain why [latex]${\mathbb{M}}^T {\mathbb{M}}$[/latex] is invertible if and only if the columns of [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] are linearly independent.
Answer: The columns of [latex]${\mathbb{M}}$[/latex] are linearly independent if and only if [latex]$C({\mathbb{M}})$[/latex] is [latex]$m$[/latex]-dimensional; that’s what we’ll check.
First, suppose [latex]$m \leq n$[/latex], and let [latex]${\mathbb{V}}$[/latex] have the singular value decomposition [latex]\begin{align*} {\mathbb{V}} = \sigma_1 {\mathbf{u}}_1 {\mathbf{v}}_1^T + \ldots + \sigma_m {\mathbf{u}}_m {\mathbf{v}}_m^T. \end{align*}[/latex] Then [latex]${\mathbb{V}}^T {\mathbb{V}}$[/latex] has the spectral decomposition [latex]\begin{align*} {\mathbb{V}}^T {\mathbb{V}} = \sigma_1^2 {\mathbf{v}}_1 {\mathbf{v}}_1^T + \ldots + \sigma_m^2 {\mathbf{v}}_m {\mathbf{v}}_m^T. \end{align*}[/latex] We know that [latex]${\mathbb{V}}^T {\mathbb{V}}$[/latex] is invertible if and only if its eigenvalues [latex]$\sigma_1^2, \ldots, \sigma_m^2$[/latex] are positive. The column space of [latex]${\mathbb{V}}$[/latex] is a linear combination of all of [latex]${\mathbf{u}}_1, \ldots, {\mathbf{u}}_m$[/latex] if and only if none of the singular values [latex]$\sigma_1, \ldots, \sigma_m$[/latex] are zero. These conditions are the same.
Otherwise, if [latex]$n < m$[/latex], then the singular decomposition can’t possibly represent a linear combination of [latex]$m$[/latex] column vectors, so [latex]${\mathbb{M}}$[/latex] can’t have linearly independent. Similarly, the squared singular values can’t account for the [latex]$m$[/latex] positive eigenvalues that [latex]${\mathbb{V}}^T {\mathbb{V}}$[/latex] would need to be invertible.

Number: Exercise 1.64
Question: Let [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times n}$[/latex]. Prove that [latex]${\mathbb{M}}$[/latex] is symmetric if and only if [latex]$\langle {\mathbf{v}}, {\mathbb{M}} {\mathbf{w}} \rangle = \langle {\mathbb{M}} {\mathbf{v}}, {\mathbf{w}} \rangle$[/latex] for every [latex]${\mathbf{v}}, {\mathbf{w}} \in {\pmb{\mathbb{R}}}^n$[/latex].
Answer: First, assume that [latex]${\mathbb{M}}$[/latex] is symmetric. For an arbitrary [latex]${\mathbf{v}}, {\mathbf{w}} \in {\pmb{\mathbb{R}}}^n$[/latex], we can use a spectral decomposition of [latex]${\mathbb{M}}$[/latex] to see that [latex]\begin{align*} \langle {\mathbf{v}}, {\mathbb{M}} {\mathbf{w}} \rangle &= {\mathbf{v}}^T {\mathbb{M}} {\mathbf{w}}\\ &= {\mathbf{v}}^T (\lambda_1 {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \lambda_n {\mathbf{q}}_n {\mathbf{q}}_n^T) {\mathbf{w}}\\ &= \lambda_1 ({\mathbf{v}}^T {\mathbf{q}}_1) ({\mathbf{q}}_1^T {\mathbf{w}}) + \ldots + \lambda_n ({\mathbf{v}}^T {\mathbf{q}}_n) ({\mathbf{q}}_n^T {\mathbf{w}})\\ &= \lambda_1 ({\mathbf{w}}^T {\mathbf{q}}_1) ({\mathbf{q}}_1^T {\mathbf{v}}) + \ldots + \lambda_n ({\mathbf{w}}^T {\mathbf{q}}_n) ({\mathbf{q}}_n^T {\mathbf{v}})\\ &= {\mathbf{w}}^T (\lambda_1 {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \lambda_n {\mathbf{q}}_n {\mathbf{q}}_n^T) {\mathbf{v}}\\ &= {\mathbf{w}}^T {\mathbb{M}} {\mathbf{v}}\\ &= \langle {\mathbf{w}}, {\mathbb{M}} {\mathbf{v}} \rangle. \end{align*}[/latex] Next, suppose [latex]$\langle {\mathbf{v}}, {\mathbb{M}} {\mathbf{w}} \rangle = \langle {\mathbb{M}} {\mathbf{v}}, {\mathbf{w}} \rangle$[/latex] for every [latex]${\mathbf{v}}, {\mathbf{w}} \in {\pmb{\mathbb{R}}}^n$[/latex]. In particular, apply two canonical basis vectors [latex]${\mathbf{e}}_i$[/latex] and [latex]${\mathbf{e}}_j$[/latex]. The vector [latex]${\mathbb{M}} {\mathbf{e}}_j$[/latex] is the [latex]$j$[/latex]th column of [latex]${\mathbb{M}}$[/latex], so [latex]$\langle {\mathbf{e}}_i, {\mathbb{M}} {\mathbf{e}}_j \rangle$[/latex] is the [latex]$(i,j)$[/latex]-entry of [latex]${\mathbb{M}}$[/latex]. By our assumption, it is equal to [latex]$\langle {\mathbf{e}}_j, {\mathbb{M}} {\mathbf{e}}_i \rangle$[/latex] which is the [latex]$(j, i)$[/latex]-entry of [latex]${\mathbb{M}}$[/latex] which shows that [latex]${\mathbb{M}}$[/latex] must be symmetric.

Number: Exercise 1.65
Question: Let [latex]${\mathbb{H}}$[/latex] be a real matrix. Use Exercise 1.64 to show that if [latex]${\mathbb{H}}$[/latex] is an orthogonal projection matrix then it must be symmetric.
Answer: [latex]\begin{align*} \langle {\mathbb{H}} {\mathbf{v}}, {\mathbf{w}} \rangle &= \langle {\mathbb{H}} {\mathbf{v}}, {\mathbb{H}} {\mathbf{w}} + ({\mathbf{w}} - {\mathbb{H}} {\mathbf{w}}) \rangle\\ &= \langle {\mathbb{H}} {\mathbf{v}}, {\mathbb{H}} {\mathbf{w}} \rangle + \langle {\mathbb{H}} {\mathbf{v}}, {\mathbf{w}} - {\mathbb{H}} {\mathbf{w}} \rangle \end{align*}[/latex] Because [latex]${\mathbb{H}} {\mathbf{w}}$[/latex] is in the space that [latex]${\mathbb{H}}$[/latex] projects onto while [latex]${\mathbf{w}} - {\mathbb{H}} {\mathbf{w}}$[/latex] is orthogonal to it, the second term is zero. Similarly, [latex]\begin{align*} \langle {\mathbf{v}}, {\mathbb{H}} {\mathbf{w}} \rangle &= \langle {\mathbb{H}} {\mathbf{v}} + ({\mathbf{v}} - {\mathbb{H}} {\mathbf{v}}), {\mathbb{H}} {\mathbf{w}} \rangle\\ &= \langle {\mathbb{H}} {\mathbf{v}}, {\mathbb{H}} {\mathbf{w}} \rangle + \underbrace{\langle {\mathbf{v}} - {\mathbb{H}} {\mathbf{v}}, {\mathbb{H}} {\mathbf{w}} \rangle}_0. \end{align*}[/latex] Both [latex]$\langle {\mathbf{v}}, {\mathbb{H}} {\mathbf{w}} \rangle$[/latex] and [latex]$\langle {\mathbb{H}} {\mathbf{v}}, {\mathbf{w}} \rangle$[/latex] simplify to [latex]$\langle {\mathbb{H}} {\mathbf{v}}, {\mathbb{H}} {\mathbf{w}} \rangle$[/latex], so they are equal to each other.
It’s also clear from the formula derived in Exercise 1.71 that orthogonal projection matrices are symmetric. However, I prefer the argument used here because it’s more readily extended to orthogonal projection operators.

Number: Exercise 1.66
Question: Provide a formula for the matrix that maps [latex]${\mathbf{y}}$[/latex] to its orthogonal projection onto the span of the unit vector [latex]${\mathbf{u}} \in {\pmb{\mathbb{R}}}^n$[/latex].
Answer: The orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto the span of [latex]${\mathbf{u}}$[/latex] is [latex]$\langle {\mathbf{y}}, {\mathbf{u}} \rangle {\mathbf{u}}$[/latex]. By rewriting this as [latex]${\mathbf{u}} {\mathbf{u}}^T {\mathbf{y}}$[/latex], we realize that the matrix [latex]${\mathbf{u}} {\mathbf{u}}^T$[/latex] maps any vector to its orthogonal projection onto the span of [latex]${\mathbf{u}}$[/latex].

Number: Exercise 1.67
Question: Show that the trace of an orthogonal projection matrix equals the dimension of the subspace that it projects onto.
Answer: From Exercise 1.60, we know that the trace of [latex]${\mathbb{H}}$[/latex] equals the sum of its eigenvalues [latex]$\lambda_1, \ldots, \lambda_n$[/latex]. Furthermore, because it’s is an orthogonal projection matrix, we know that it yields the spectral decomposition [latex]\begin{align*} {\mathbb{H}} &= (1) {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + (1) {\mathbf{q}}_m {\mathbf{q}}_m^T + (0) {\mathbf{q}}_{m+1} {\mathbf{q}}_{m+1}^T + \ldots + (0) {\mathbf{q}}_n {\mathbf{q}}_n^T \end{align*}[/latex] where [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_m$[/latex] are in the subspace that [latex]${\mathbb{H}}$[/latex] projects onto and the rest are necessarily orthogonal to it. We see [latex]$m$[/latex] terms with the eigenvalue [latex]$1$[/latex] and the remaining terms with the eigenvalue [latex]$0$[/latex], so their sum is [latex]$m$[/latex] which is the dimension of the subspace that [latex]${\mathbb{H}}$[/latex] projects onto.

Number: Exercise 1.68
Question: Let [latex]${\mathbb{M}}$[/latex] be a matrix. Explain why the rank of the orthogonal projection matrix onto [latex]$C({\mathbb{M}})$[/latex] must be exactly the same as the rank of [latex]${\mathbb{M}}$[/latex].
Answer: The equality of ranks follows from the stronger observation that the orthogonal projection matrix must have the exact same column space as [latex]${\mathbb{M}}$[/latex]. Every vector in [latex]$C({\mathbb{M}})$[/latex] gets mapped to itself by the orthogonal projection matrix, so its column space is at least as large as [latex]$C({\mathbb{M}})$[/latex]. However, the orthogonal projection of any vector onto [latex]$C({\mathbb{M}})$[/latex] must by definition be in [latex]$C({\mathbb{M}})$[/latex], so the orthogonal projection matrix cannot map any vector to a result outside of [latex]$C({\mathbb{M}})$[/latex].

Number: Exercise 1.69
Question: Let [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] and [latex]${\mathbf{y}} \in {\pmb{\mathbb{R}}}^n$[/latex]. Explain why the Normal equation [latex]\begin{align*} {\mathbb{M}}^T {\mathbb{M}} {\mathbf{\hat{b}}} &= {\mathbb{M}}^T {\mathbf{y}} \end{align*}[/latex] is satisfied by the coefficient vector [latex]${\mathbf{\hat{b}}} \in {\pmb{\mathbb{R}}}^m$[/latex] if and only if [latex]${\mathbb{M}} {\mathbf{\hat{b}}}$[/latex] is the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]$C({\mathbb{M}})$[/latex].
Answer: The orthogonal projection [latex]${\mathbb{M}} {\mathbf{\hat{b}}}$[/latex] is the unique vector in [latex]$C({\mathbb{M}})$[/latex] with the property that [latex]${\mathbf{y}} - {\mathbb{M}} {\mathbf{\hat{b}}} \perp C({\mathbb{M}})$[/latex]. It is equivalent to check that [latex]${\mathbf{y}} - {\mathbb{M}} {\mathbf{\hat{b}}}$[/latex] is orthogonal to every column [latex]${\mathbf{v}}_1, \ldots, {\mathbf{v}}_m$[/latex] of [latex]${\mathbb{M}}$[/latex]. Equivalently the following quantity should be equal to the zero vector: [latex]\begin{align*} {\mathbb{M}}^T ({\mathbf{y}} - {\mathbb{M}} {\mathbf{\hat{b}}}) &= \begin{bmatrix}- & {\mathbf{v}}_1 & -\\ & \vdots & \\ - & {\mathbf{v}}_m & -\end{bmatrix} ({\mathbf{y}} - {\mathbb{M}} {\mathbf{\hat{b}}})\\ &= \begin{bmatrix} {\mathbf{v}}_1^T ({\mathbf{y}} - {\mathbb{M}} {\mathbf{\hat{b}}}) \\ \vdots \\ {\mathbf{v}}_m^T ({\mathbf{y}} - {\mathbb{M}} {\mathbf{\hat{b}}}) \end{bmatrix}. \end{align*}[/latex] Setting this vector [latex]${\mathbb{M}}^T ({\mathbf{y}} - {\mathbb{M}} {\mathbf{\hat{b}}})$[/latex] equal to the zero vector results in the Normal equation.

Number: Exercise 1.70
Question: Suppose [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] has linearly independent columns. Provide a formula for the coefficient vector [latex]${\mathbf{\hat{b}}}$[/latex] for which [latex]${\mathbb{M}} {\mathbf{\hat{b}}}$[/latex] equals the orthogonal projection of [latex]${\mathbf{y}} \in {\pmb{\mathbb{R}}}^n$[/latex] onto [latex]$C({\mathbb{M}})$[/latex].
Answer: Because the columns are linearly independent, we know that [latex]${\mathbb{M}}^T {\mathbb{M}}$[/latex] is invertible and thus the Normal equation [latex]\begin{align*} {\mathbb{M}}^T {\mathbb{M}} {\mathbf{\hat{b}}} &= {\mathbb{M}}^T {\mathbf{y}} \end{align*}[/latex] is uniquely solved by [latex]${\mathbf{\hat{b}}} = ({\mathbb{M}}^T {\mathbb{M}})^{-1} {\mathbb{M}}^T {\mathbf{y}}$[/latex].

Number: Exercise 1.71
Question: Suppose [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] has linearly independent columns. Provide a formula for the orthogonal projection matrix onto [latex]$C({\mathbb{M}})$[/latex].
Answer: We’ve already derived in Exercise 1.70 a formula for the desired coefficient vector [latex]${\mathbf{\hat{b}}} = ({\mathbb{M}}^T {\mathbb{M}})^{-1} {\mathbb{M}}^T {\mathbf{y}}$[/latex], so we simply plug this into [latex]${\mathbb{M}} {\mathbf{\hat{b}}}$[/latex] to find the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]$C({\mathbb{M}})$[/latex]. [latex]\begin{align*} {\mathbf{\hat{y}}} &= {\mathbb{M}} {\mathbf{\hat{b}}}\\ &= {\mathbb{M}} ({\mathbb{M}}^T {\mathbb{M}})^{-1} {\mathbb{M}}^T {\mathbf{y}} \end{align*}[/latex] Therefore, we see that [latex]${\mathbf{y}}$[/latex] is mapped to its orthogonal projection onto [latex]$C({\mathbb{M}})$[/latex] by the matrix [latex]${\mathbb{M}} ({\mathbb{M}}^T {\mathbb{M}})^{-1} {\mathbb{M}}^T$[/latex].

Number: Exercise 1.72
Question: Show that [latex]${\mathbb{M}}^- {\mathbb{M}}$[/latex] equals the orthogonal projection matrix onto the row space of [latex]${\mathbb{M}}$[/latex].
Answer: Let [latex]${\mathbb{M}} = {\mathbb{U}} {\mathbb{S}} {\mathbb{V}}^T$[/latex] be a singular value decomposition for which [latex]${\mathbb{S}}$[/latex] is square and has only positive values along its diagonal. [latex]\begin{align*} {\mathbb{M}}^- {\mathbb{M}} &= {\mathbb{V}} {\mathbb{S}}^{-1} {\mathbb{U}}^T {\mathbb{U}} {\mathbb{S}} {\mathbb{V}}^T\\ &= {\mathbb{V}} {\mathbb{V}}^T \end{align*}[/latex] Exercise 1.71 indicates that the orthogonal projection matrix onto [latex]$C({\mathbb{V}})$[/latex] is [latex]${\mathbb{V}} ({\mathbb{V}}^T {\mathbb{V}})^{-1} {\mathbb{V}}^T$[/latex] which simplifies to [latex]${\mathbb{V}} {\mathbb{V}}^T$[/latex] because [latex]${\mathbb{V}}^T {\mathbb{V}}$[/latex] is the identity.
It only remains to establish that [latex]$C({\mathbb{V}})$[/latex] is the row space of [latex]${\mathbb{M}}$[/latex]. Letting the entries of [latex]${\mathbf{w}}$[/latex] be the coefficients of the linear combination, [latex]\begin{align*} {\mathbf{w}}^T {\mathbb{M}} &= {\mathbf{w}}^T (\sigma_1 {\mathbf{u}}_1 {\mathbf{v}}_1^T + \ldots \sigma_d {\mathbf{u}}_d {\mathbf{v}}_d^T)\\ &= \sigma_1 \langle {\mathbf{w}}, {\mathbf{u}}_1 \rangle {\mathbf{v}}_1^T + \ldots + \sigma_d \langle {\mathbf{w}}, {\mathbf{u}}_d \rangle {\mathbf{v}}_d^T. \end{align*}[/latex] Any linear combination of [latex]${\mathbf{v}}_1^T, \ldots, {\mathbf{v}}_d^T$[/latex] can be produced by the appropriate choice of [latex]${\mathbf{w}}$[/latex], but no vector outside of their span can be produced.

Number: Exercise 1.73
Question: Explain why the Moore-Penrose inverse of an invertible matrix must be its inverse.
Answer: If [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times n}$[/latex] is invertible, then its row space is [latex]${\pmb{\mathbb{R}}}^n$[/latex]. Exercise 1.72 implies that [latex]${\mathbb{M}}^- {\mathbb{M}} {\mathbf{w}} = {\mathbf{w}}$[/latex] for every [latex]${\mathbf{w}} \in {\pmb{\mathbb{R}}}^n$[/latex]. [latex]${\mathbb{M}}^-$[/latex] must be the inverse according to Exercise 1.22.

Number: Exercise 1.74
Question: Of all coefficient vectors that satisfy the Normal equation, show that [latex]${\mathbf{\hat{b}}} = ({\mathbb{M}}^T {\mathbb{M}})^- {\mathbb{M}}^T {\mathbf{\hat{y}}}$[/latex] has the smallest norm.
Answer: From Section 1.5, we know that every solution can be represented as [latex]${\mathbf{\hat{b}}} + {\mathbf{w}}$[/latex] for some [latex]${\mathbf{w}}$[/latex] in the null space of [latex]${\mathbb{M}}^T {\mathbb{M}}$[/latex]. Let [latex]${\mathbb{M}}^T {\mathbb{M}}$[/latex] have spectral decomposition [latex]\begin{align*} {\mathbb{M}}^T {\mathbb{M}} &= \sigma_1^2 {\mathbf{v}}_1 {\mathbf{v}}_1^T + \ldots + \sigma_d^2 {\mathbf{v}}_d {\mathbf{v}}_d^T + (0) {\mathbf{v}}_{d+1} {\mathbf{v}}_{d+1}^T + \ldots + (0) {\mathbf{v}}_m {\mathbf{v}}_m^T. \end{align*}[/latex] with positive [latex]$\sigma_1^2, \ldots, \sigma_d^2$[/latex]. It’s clear that the null space is exactly the span of [latex]$\{{\mathbf{v}}_{d+1}, \ldots, {\mathbf{v}}_m\}$[/latex]. On the other hand, by definition of generalized inverse, [latex]${\mathbf{\hat{b}}}$[/latex] is in the span of [latex]$\{{\mathbf{v}}_1, \ldots, {\mathbf{v}}_d\}$[/latex]. The squared norm of any solution [latex]${\mathbf{\hat{b}}} + {\mathbf{w}}$[/latex] is [latex]$\| {\mathbf{\hat{b}}} \|^2 + \| {\mathbf{w}} \|^2$[/latex], so the solution of smallest norm is [latex]${\mathbf{\hat{b}}}$[/latex].

Number: Exercise 1.75
Question: Show that [latex]${\mathbb{M}} {\mathbb{M}}^- {\mathbb{M}} = {\mathbb{M}}$[/latex].
Answer: Let [latex]${\mathbb{M}}$[/latex] have singular value decomposition [latex]${\mathbb{U}} {\mathbb{S}} {\mathbb{V}}^T$[/latex] where [latex]${\mathbb{S}}$[/latex] is a square matrix with strictly positive diagonals. Then [latex]\begin{align*} {\mathbb{M}} {\mathbb{M}}^- {\mathbb{M}} &= {\mathbb{U}} {\mathbb{S}} \underbrace{{\mathbb{V}}^T  \mathbb{V}}_{\mathbb{I}} {\mathbb{S}}^{-1} \underbrace{{\mathbb{U}}^T \mathbb{U}}_{\mathbb{I}} {\mathbb{S}} {\mathbb{V}}^T\\ &= {\mathbb{U}} {\mathbb{S}} {\mathbb{V}}^T\\ &= {\mathbb{M}}. \end{align*}[/latex]

Number: Exercise 1.76
Question: For a unit vector [latex]${\mathbf{u}}$[/latex], express the quadratic form [latex]${\mathbf{u}}^T {\mathbb{M}} {\mathbf{u}}$[/latex] as a weighted average of the eigenvalues of [latex]${\mathbb{M}} \in {\pmb{\mathbb{R}}}^{n \times n}$[/latex].
Answer: Let [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex] be an orthonormal basis of eigenvectors for [latex]${\mathbb{M}}$[/latex] with eigenvalues [latex]$\lambda_1, \ldots, \lambda_n$[/latex]. We can represent [latex]${\mathbf{u}}$[/latex] with respect to the eigenvector basis as [latex]$\langle {\mathbf{u}}, {\mathbf{q}}_1 \rangle {\mathbf{q}}_1 + \ldots + \langle {\mathbf{u}}, {\mathbf{q}}_n \rangle {\mathbf{q}}_n$[/latex]. [latex]\begin{align*} {\mathbf{u}}^T {\mathbb{M}} {\mathbf{u}} &= {\mathbf{u}}^T {\mathbb{M}} (\langle {\mathbf{u}}, {\mathbf{q}}_1 \rangle {\mathbf{q}}_1 + \ldots + \langle {\mathbf{u}}, {\mathbf{q}}_n \rangle {\mathbf{q}}_n)\\ &= {\mathbf{u}}^T (\langle {\mathbf{u}}, {\mathbf{q}}_1 \rangle \underbrace{{\mathbb{M}} {\mathbf{q}}_1}_{\lambda_1 {\mathbf{q}}_1} + \ldots + \langle {\mathbf{u}}, {\mathbf{q}}_n \rangle \underbrace{{\mathbb{M}} {\mathbf{q}}_n}_{\lambda_n {\mathbf{q}}_n})\\ &= \langle {\mathbf{u}}, {\mathbf{q}}_1 \rangle^2 \lambda_1 + \ldots + \langle {\mathbf{u}}, {\mathbf{q}}_n \rangle^2 \lambda_n \end{align*}[/latex] [latex]$\langle {\mathbf{u}}, {\mathbf{q}}_1 \rangle, \ldots, \langle {\mathbf{u}}, {\mathbf{q}}_n \rangle$[/latex] provide the coordinates of [latex]${\mathbf{u}}$[/latex] with respect to the basis [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex]. Because [latex]${\mathbf{u}}$[/latex] is a unit vector, the sum of these squared coordinates has to be [latex]$1$[/latex]. Additionally, the squared coordinates are non-negative. Consequently, we’ve expressed [latex]${\mathbf{u}}^T {\mathbb{M}} {\mathbf{u}}$[/latex] as a weighted average of the eigenvalues; the weights are the squared coordinates of [latex]${\mathbf{u}}$[/latex] with respect to the eigenvector basis.

Number: Exercise 1.77
Question: Identify a unit vector [latex]${\mathbf{u}}$[/latex] that maximizes the quadratic form [latex]${\mathbf{u}}^T {\mathbb{M}} {\mathbf{u}}$[/latex].
Answer: From Exercise 1.76, we know that the quadratic form equals a weighted average of the eigenvalues. This weighted average is maximized by placing all of the weight on the largest eigenvalue, that is, by letting [latex]${\mathbf{u}}$[/latex] be a principal eigenvector. Such a choice of [latex]${\mathbf{u}}$[/latex] makes [latex]${\mathbf{u}}^T {\mathbb{M}} {\mathbf{u}}$[/latex] equal to the largest eigenvalue.

Number: Exercise 1.78
Question: Given any real matrix [latex]${\mathbb{M}}$[/latex], show that [latex]${\mathbb{M}}^T {\mathbb{M}}$[/latex] is positive semi-definite.
Answer: Exercise 1.54 established that the matrix in question is symmetric. The quadratic form [latex]\begin{align*} {\mathbf{v}}^T ({\mathbb{M}}^T {\mathbb{M}}) {\mathbf{v}} &= ({\mathbf{v}}^T {\mathbb{M}}^T) ({\mathbb{M}} {\mathbf{v}})\\ &= ({\mathbb{M}} {\mathbf{v}})^T ({\mathbb{M}} {\mathbf{v}}) \end{align*}[/latex] equals the squared norm of the vector [latex]${\mathbb{M}} {\mathbf{v}}$[/latex] which is non-negative.

Number: Exercise 1.79
Question: Let [latex]${\mathbb{M}}$[/latex] be a symmetric real matrix. Show that [latex]${\mathbb{M}}$[/latex] is positive semi-definite if and only if its eigenvalues are all non-negative.
Answer: From our work in Exercise 1.77, we’ve seen how to express the quadratic form as a linear combination of the eigenvalues [latex]\begin{align*} {\mathbf{v}}^T {\mathbb{M}} {\mathbf{v}} &= \langle {\mathbf{v}}, {\mathbf{q}}_1 \rangle^2 \lambda_1 + \ldots + \langle {\mathbf{v}}, {\mathbf{q}}_n \rangle^2 \lambda_n. \end{align*}[/latex] If every eigenvalue is at least zero, then every term in this sum is non-negative so the quadratic form must be non-negative. Conversely, if [latex]$\lambda_j$[/latex] is negative, then the quadratic form arising from [latex]${\mathbf{v}} = {\mathbf{q}}_j$[/latex] is negative, as it equals [latex]$\lambda_j$[/latex].

Number: Exercise 1.80
Question: Let [latex]${\mathbb{H}} \in {\pmb{\mathbb{R}}}^{n \times n}$[/latex] be an orthogonal projection matrix, and let [latex]${\mathbf{v}} \in {\pmb{\mathbb{R}}}^n$[/latex]. Show that the squared length of [latex]${\mathbb{H}} {\mathbf{v}}$[/latex] equals the quadratic form [latex]${\mathbf{v}}^T {\mathbb{H}} {\mathbf{v}}$[/latex].
Answer: Because [latex]${\mathbb{H}}$[/latex] is symmetric and idempotent, [latex]\begin{align*} \|{\mathbb{H}} {\mathbf{v}} \|^2 &= ({\mathbb{H}} {\mathbf{v}})^T({\mathbb{H}} {\mathbf{v}})\\ &= {\mathbf{v}}^T {\mathbb{H}}^T {\mathbb{H}} {\mathbf{v}}\\ &= {\mathbf{v}}^T {\mathbb{H}} {\mathbf{v}}. \end{align*}[/latex]

Number: Exercise 1.81
Question: Let [latex]${\mathbf{x}}_1, \ldots, {\mathbf{x}}_n$[/latex] be the rows of a real matrix [latex]${\mathbb{X}}$[/latex]. Show that the quadratic form [latex]${\mathbf{u}}^T (\frac{1}{n} {\mathbb{X}}^T {\mathbb{X}}) {\mathbf{u}}$[/latex] is equal to the average of the squares of the coefficients of [latex]${\mathbf{x}}_1, \ldots, {\mathbf{x}}_n$[/latex] with respect to [latex]${\mathbf{u}}$[/latex].
Answer: We’ll first express the quadratic form in terms of the squared norm of a vector. [latex]\begin{align*} {\mathbf{u}}^T (\tfrac{1}{n} {\mathbb{X}}^T {\mathbb{X}}) {\mathbf{u}} &= \tfrac{1}{n} ({\mathbb{X}} {\mathbf{u}})^T ({\mathbb{X}} {\mathbf{u}})\\ &= \tfrac{1}{n} \| {\mathbb{X}} {\mathbf{u}} \|^2 \end{align*}[/latex] The entries of the vector [latex]${\mathbb{X}} {\mathbf{u}}$[/latex] are the coefficients of [latex]${\mathbf{x}}_1, \ldots, {\mathbf{x}}_n$[/latex] with respect to [latex]${\mathbf{u}}$[/latex]. Its squared norm is the sum of its squared entries, so [latex]$\frac{1}{n} \| {\mathbb{X}} {\mathbf{u}} \|^2$[/latex] is the average of the squared coefficients.

Number: Exercise 1.82
Question: Let [latex]${\mathbf{x}}_1, \ldots, {\mathbf{x}}_n$[/latex] be the rows of the matrix [latex]${\mathbb{X}}$[/latex]. Show that [latex]$\frac{1}{n} {\mathbb{X}}^T {\mathbb{X}}$[/latex] is the matrix whose [latex]$(j, k)$[/latex]-entry is the average of the product of the [latex]$j$[/latex]th and [latex]$k$[/latex]th coordinates of the vectors [latex]${\mathbf{x}}_1, \ldots, {\mathbf{x}}_n$[/latex].
Answer: The product of the matrices [latex]\begin{align*} {\mathbb{X}}^T {\mathbb{X}} &= \begin{bmatrix}| & & |\\ {\mathbf{x}}_1 & \cdots & {\mathbf{x}}_n\\ | & & | \end{bmatrix} \begin{bmatrix} - & {\mathbf{x}}_1 & - \\ & \vdots & \\ - & {\mathbf{x}}_n & -\end{bmatrix} \end{align*}[/latex] has as its [latex]$(j, k)$[/latex]-entry the inner product of the [latex]$j$[/latex]th row of [latex]${\mathbb{X}}^T$[/latex] and the [latex]$k$[/latex]th column of [latex]${\mathbb{X}}$[/latex]. With [latex]$x_{i,j}$[/latex] denoting the [latex]$j$[/latex]th coordinate of [latex]${\mathbf{x}}_i$[/latex], this inner product equals [latex]$\sum_i x_{i, j} x_{i, k}$[/latex]. When multiplied by [latex]$1/n$[/latex], this entry is indeed the average of the products of the coordinates. By thinking about summing over the observations, [latex]$\frac{1}{n} {\mathbb{X}}^T {\mathbb{X}}$[/latex] can also be understood as an average of rank-[latex]$1$[/latex] matrices [latex]$\frac{1}{n} {\mathbf{x}}_i {\mathbf{x}}_i^T$[/latex].

Number: Exercise 1.83
Question: Let [latex]${\mathbf{x}}_1, \ldots, {\mathbf{x}}_n$[/latex] be the rows of a real matrix [latex]${\mathbb{X}}$[/latex]. Show that the average squared length [latex]$\tfrac{1}{n} \sum_i \| {\mathbf{x}}_i \|^2$[/latex] equals the sum of the eigenvalues of [latex]$\frac{1}{n} {\mathbb{X}}^T {\mathbb{X}}$[/latex].
Answer: By Parseval’s identity, the squared norm equals the sum of the squared coordinates using any basis; let’s consider the orthonormal eigenvectors [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_m$[/latex] of [latex]$\frac{1}{n} {\mathbb{X}}^T {\mathbb{X}}$[/latex], with [latex]$\lambda_1, \ldots, \lambda_m$[/latex] denoting their eigenvalues. Recall that Exercise 1.81 allows us to rewrite the average of squared coefficients as a quadratic form. [latex]\begin{align*} \tfrac{1}{n} \sum_i \| {\mathbf{x}}_i \|^2 &= \tfrac{1}{n} \sum_i (\langle {\mathbf{x}}_i, {\mathbf{q}}_1 \rangle^2 + \ldots + \langle {\mathbf{x}}_i, {\mathbf{q}}_m \rangle^2)\\ &= \tfrac{1}{n} \sum_i \langle {\mathbf{x}}_i, {\mathbf{q}}_1 \rangle^2 + \ldots + \tfrac{1}{n} \sum_i \langle {\mathbf{x}}_i, {\mathbf{q}}_m \rangle^2\\ &= \underbrace{{\mathbf{q}}_1^T (\tfrac{1}{n} {\mathbb{X}}^T {\mathbb{X}}) {\mathbf{q}}_1}_{\lambda_1} + \ldots + \underbrace{{\mathbf{q}}_m^T (\tfrac{1}{n} {\mathbb{X}}^T {\mathbb{X}}) {\mathbf{q}}_m}_{\lambda_m} \end{align*}[/latex] Exercise 1.76 demonstrated that a quadratic form evaluated at a unit eigenvector equals the corresponding eigenvalue.

Number: Exercise 2.01
Question: Show that the entries of [latex]${\mathbf{v}} = (v_1, \ldots, v_n)$[/latex] have mean zero if and only if [latex]${\mathbf{v}}$[/latex] is orthogonal to [latex]${\mathbf{1}} = (1, \ldots, 1)$[/latex].
Answer: The average of the entries is proportional to the inner product of [latex]${\mathbf{v}}$[/latex] with [latex]${\mathbf{1}}$[/latex]. [latex]\begin{align*} \tfrac{1}{n} \sum_i v_i &= \tfrac{1}{n} \langle {\mathbf{v}}, {\mathbf{1}} \rangle \end{align*}[/latex] So the average is zero if and only if the inner product is zero.

Number: Exercise 2.02
Question: Use the Pythagorean identity to decompose the average of the squared differences between the response values and [latex]$a \in {\pmb{\mathbb{R}}}$[/latex], that is [latex]$\tfrac{1}{n} \sum_i (y_i - a)^2$[/latex], into two terms, one of which is the empirical variance of [latex]$y_1, \ldots, y_n$[/latex].
Answer: We can write [latex]$\sum_i (y_1 - a)^2$[/latex] as the squared norm [latex]$\| {\mathbf{y}} - a {\mathbf{1}}\|^2$[/latex]. The vector [latex]${\mathbf{y}}- a {\mathbf{1}}$[/latex] is the hypotenuse of the right triangle whose other two sides are [latex]${\mathbf{y}}- {\bar{y}} {\mathbf{1}}$[/latex] and [latex]${\bar{y}} {\mathbf{1}} - a {\mathbf{1}}$[/latex]. By the Pythagorean identity, [latex]\begin{align*} \tfrac{1}{n} \sum_i (y_i - a)^2 &= \tfrac{1}{n} \| {\mathbf{y}}- a {\mathbf{1}}\|^2\\ &= \tfrac{1}{n} [\| {\mathbf{y}}- {\bar{y}} {\mathbf{1}} \|^2 + \| {\bar{y}} {\mathbf{1}} - a {\mathbf{1}}\|^2]\\ &= \tfrac{1}{n} \left[\sum_i (y_i - {\bar{y}})^2 + n ({\bar{y}} - a)^2\right]\\ &= \tfrac{1}{n} \sum_i (y_i - {\bar{y}})^2 + ({\bar{y}} - a)^2. \end{align*}[/latex]

Number: Exercise 2.03
Question: Is it possible for the least-squares line’s sum of squared residuals to be greater than the least-squares point’s sum of squared residuals?
Answer: The set of possible prediction functions corresponding to lines [latex]$\{f(x) = a + bx : a,b \in {\pmb{\mathbb{R}}}\}$[/latex] is strictly larger than the set of possible prediction functions corresponding to points [latex]$\{f(x) = a : a \in {\pmb{\mathbb{R}}}\}$[/latex]. A line predicts every response value by the same number if its slope is zero. By definition, the least-squares line will use a slope of zero if and only if that leads to the smallest possible sum of squared residuals, in which case its sum of squared residuals would be equal to that of the least-squares point.

Number: Exercise 2.04
Question: The variables picture provides us with a more specific answer to the question posed in Exercise 2.3. Use the Pythagorean identity to quantify the difference between the least-squares point’s sum of squared residuals and the least-squares line’s sum of squared residuals.
Answer: Because [latex]${\bar{y}} {\mathbf{1}}$[/latex] is in the span of [latex]${\mathbf{1}}$[/latex] and [latex]${\mathbf{x}}$[/latex], we see that the least-squares line’s residual vector [latex]${\mathbf{y}} - {\mathbf{\hat{y}}}$[/latex] must be orthogonal to [latex]${\mathbf{\hat{y}}} - {\bar{y}} {\mathbf{1}}$[/latex]. Invoking the Pythagorean identity, [latex]\begin{align*} \| {\mathbf{y}} - {\bar{y}} {\mathbf{1}} \|^2 = \| {\mathbf{\hat{y}}} - {\bar{y}} {\mathbf{1}} \|^2 + \| {\mathbf{y}} - {\mathbf{\hat{y}}} \|^2. \end{align*}[/latex] The least-squares point’s sum of squared residuals is larger than the least-squares line’s sum of squared residuals by [latex]$\| {\mathbf{\hat{y}}} - {\bar{y}} {\mathbf{1}} \|^2$[/latex].

Number: Exercise 2.05
Question: Show that the correlation between two vectors equals the empirical covariance of their standardized versions.
Answer: [latex]\begin{align*} \rho_{{\mathbf{x}}, \mathbf{y}} &:= \frac{\sigma_{{\mathbf{x}}, \mathbf{y}}}{\sigma_{\mathbf{x}} \sigma_{\mathbf{y}}}\\ &= \frac{(1/n) \langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, {\mathbf{y}} - {\bar{y}} {\mathbf{1}} \rangle}{(\sqrt{1/n} \| {\mathbf{x}} - {\bar{x}} {\mathbf{1}} \|) (\sqrt{1/n} \| {\mathbf{y}} - {\bar{y}} {\mathbf{1}} \|) }\\ &= \frac{\langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, {\mathbf{y}} - {\bar{y}} {\mathbf{1}} \rangle}{\| {\mathbf{x}} - {\bar{x}} {\mathbf{1}} \| \| {\mathbf{y}} - {\bar{y}} {\mathbf{1}} \| }. \end{align*}[/latex]

Number: Exercise 2.06
Question: Let [latex]${\mathbf{y}} \in {\pmb{\mathbb{R}}}^n$[/latex] be a response variable and [latex]${\mathbf{x}} \in {\pmb{\mathbb{R}}}^n$[/latex] be an explanatory variable. Consider fitting the response variable using quadratic functions of the explanatory variable: [latex]\begin{align*} \{f_{a, b, c}(x) = a + b x + c x^2 : a, b, c \in {\pmb{\mathbb{R}}}\}. \end{align*}[/latex] Show that the set of possible prediction vectors is a subspace of [latex]${\pmb{\mathbb{R}}}^n$[/latex].
Answer: Let [latex]$f_{a, b, c}({\mathbf{x}})$[/latex] denote the vector of predictions [latex]$(f_{a, b, c}(x_1), \ldots, f_{a, b, c}(x_n))$[/latex]. With [latex]${\mathbf{x}}^2$[/latex] representing the vector of squared explanatory values, an arbitrary linear combination of two arbitrary vectors of predicted values is [latex]\begin{align*} \alpha_1 f_{a_1, b_1, c_1}({\mathbf{x}}) + \alpha_2 f_{a_2, b_2, c_2}({\mathbf{x}}) &= \alpha_1 (a_1 {\mathbf{1}} + b_1 {\mathbf{x}} + c_1 {\mathbf{x}}^2) + \alpha_2 (a_2 {\mathbf{1}} + b_2 {\mathbf{x}} + c_2 {\mathbf{x}}^2)\\ &= (\alpha_1 a_1 + \alpha_2 a_2) {\mathbf{1}} + (\alpha_1 b_1 + \alpha_2 b_2) {\mathbf{x}} + (\alpha_1 c_1 + \alpha_2 c_2) {\mathbf{x}}^2\\ &= f_{\alpha_1 a_1 + \alpha_2 a_2, \alpha_1 b_1 + \alpha_2 b_2, \alpha_1 c_1 + \alpha_2 c_2}({\mathbf{x}}) \end{align*}[/latex] which is another possible vector of predicted values that can be achieved using a quadratic function. In fact, we can see that the set of possible predictions is exactly the span of [latex]${\mathbf{1}}$[/latex], [latex]${\mathbf{x}}$[/latex], and [latex]${\mathbf{x}}^2$[/latex].

Number: Exercise 2.07
Question: Let [latex]${\mathbf{y}} \in {\pmb{\mathbb{R}}}^n$[/latex] be a response variable vector and [latex]${\mathbf{x}} \in {\pmb{\mathbb{R}}}^n$[/latex] be an explanatory variable vector. Consider predicting the response variable by using quadratic functions of the explanatory variable: [latex]\begin{align*} \{f_{a, b, c}(x) = a + b x + c x^2 : a, b, c \in {\pmb{\mathbb{R}}}\}. \end{align*}[/latex] Explain how to find the coefficients [latex]$(\hat{a}, \hat{b}, \hat{c})$[/latex] of the quadratic function that minimizes the sum of squared residuals.
Answer: With [latex]${\mathbf{x}}^2$[/latex] representing the vector of squared explanatory values, we can use the design matrix [latex]\begin{align*} {\mathbb{M}} := \begin{bmatrix} | & | & | \\ {\mathbf{1}} & {\mathbf{x}} & {\mathbf{x}}^2 \\ | & | & | \end{bmatrix}. \end{align*}[/latex] According to Theorem 2.4, the least-squares coefficients are [latex]$({\hat{a}}, {\hat{b}}, {\hat{c}}) = ({\mathbb{M}}^T {\mathbb{M}})^- {\mathbb{M}}^T {\mathbf{y}}$[/latex].

Number: Exercise 2.08
Question: Let [latex]${\mathbf{\hat{y}}}$[/latex] be the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]$C({\mathbb{M}})$[/latex]. Explain why [latex]$({\mathbb{M}}^T {\mathbb{M}})^- {\mathbb{M}}^T {\mathbf{\hat{y}}}$[/latex] must be equal to [latex]$({\mathbb{M}}^T {\mathbb{M}})^- {\mathbb{M}}^T {\mathbf{y}}$[/latex].
Answer: There’s an intuitive explanation for this. You can think of [latex]$({\mathbb{M}}^T {\mathbb{M}})^- {\mathbb{M}}^T$[/latex] as the matrix that maps any vector in [latex]${\pmb{\mathbb{R}}}^n$[/latex] to the (minimum norm) coefficients of the columns of [latex]${\mathbb{M}}$[/latex] that lead to the orthogonal projection of that vector onto [latex]$C({\mathbb{M}})$[/latex]. Because the orthogonal projection of [latex]${\mathbf{\hat{y}}}$[/latex] onto [latex]$C({\mathbb{M}})$[/latex] is exactly the same as the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]$C({\mathbb{M}})$[/latex] (namely, both are [latex]${\mathbf{\hat{y}}}$[/latex]), the coefficients leading to this orthogonal projection must be the same.

Number: Exercise 2.09
Question: Suppose [latex]${\mathbf{\hat{y}}}$[/latex] is the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}$[/latex], [latex]${\mathbf{\check{y}}}$[/latex] is the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}_0 \subseteq {\mathcal{S}}$[/latex], and that [latex]${\mathbf{1}} \in {\mathcal{S}}_0$[/latex]. Explain why [latex]\begin{align*} \| {\mathbf{\hat{y}}} - {\bar{y}} {\mathbf{1}} \|^2 &= \| {\mathbf{\check{y}}} - {\bar{y}} {\mathbf{1}} \|^2 + \| {\mathbf{\hat{y}}} - {\mathbf{\check{y}}} \|^2. \end{align*}[/latex]
Answer: The vector [latex]${\mathbf{\check{y}}}$[/latex] is defined to be the orthogonal projection of [latex]${\mathbf{y}}$[/latex] onto [latex]${\mathcal{S}}_0$[/latex]. However, it’s also the orthogonal projection of [latex]${\mathbf{\hat{y}}}$[/latex] onto [latex]${\mathcal{S}}_0$[/latex] because according to Exercise 1.50, orthogonal projection onto [latex]${\mathcal{S}}$[/latex] followed by orthogonal projection onto [latex]${\mathcal{S}}_0$[/latex] lands you at the exact same vector that a single orthogonal projection onto [latex]${\mathcal{S}}_0$[/latex] does. Likewise, [latex]${\bar{y}} {\mathbf{1}}$[/latex] is the orthogonal projection of [latex]${\mathbf{\check{y}}}$[/latex] onto [latex]${\mathbf{1}}$[/latex]. Invoke the ANOVA decomposition with [latex]${\mathbf{\hat{y}}}$[/latex] playing the role of the response variable.

Number: Exercise 3.01
Question: Explain why the sum of the probabilities of [latex]$E_1$[/latex] and [latex]$E_2$[/latex] is at least as large as the probability of their union.
Answer: The union of [latex]$E_1$[/latex] and [latex]$E_2$[/latex] is the same as the union of the disjoint sets [latex]$E_1$[/latex] and [latex]$E_2/E_1$[/latex] (the part of [latex]$E_2$[/latex] that isn’t in [latex]$E_1$[/latex]). With [latex]${\mathbb{P}}$[/latex] mapping each event to its probability, [latex]\begin{align*} {\mathbb{P}} (E_1 \cup E_2) &= {\mathbb{P}} [E_1 \cup (E_2 / E_1)]\\ &= {\mathbb{P}} E_1 + {\mathbb{P}} (E_2 / E_1)\\ &\leq {\mathbb{P}} E_1 + {\mathbb{P}} E_2 \end{align*}[/latex] To understand the last step, realize that [latex]$E_2$[/latex] can be represented as the disjoint union [latex]$E_2 = (E_2 / E_1) \cup (E_2 \cap E_1)$[/latex].

Number: Exercise 3.02
Question: Let [latex]${\mathbf{X}}$[/latex] be a random vector mapping to the complex plane with the representation [latex]${\mathbf{X}} = Y + iZ$[/latex] where [latex]$Y$[/latex] and [latex]$Z$[/latex] are random variables. Verify that [latex]${\mathbb{E}} Y + i {\mathbb{E}} Z$[/latex] is the expectation of [latex]${\mathbf{X}}$[/latex] by checking property (ii), assuming that property (iii) holds for random variables.
Answer: Let [latex]${\mathbf{X}} = Y + iZ$[/latex] with random variables [latex]$Y$[/latex] and [latex]$Z$[/latex], and let [latex]$a = b + ic$[/latex] be a complex number. [latex]\begin{align*} {\mathbb{E}} \langle {\mathbf{X}}, a \rangle &= {\mathbb{E}} \langle Y + iZ, b + ic \rangle\\ &= {\mathbb{E}} [\langle Y, ic \rangle + \langle Y, b \rangle + \langle iZ, b \rangle + \langle iZ, ic \rangle]\\ &= {\mathbb{E}} [(-i) \langle Y, c \rangle + \langle Y, b \rangle + i \langle Z, b \rangle + \underbrace{(i)(-i)}_{-i^2=1} \langle Z, c \rangle]\\ &= {\mathbb{E}} (\langle Y, b \rangle + \langle Z, c \rangle) + i {\mathbb{E}} (\langle Z, b \rangle - \langle Y, c \rangle)\\ &= \langle {\mathbb{E}} Y, b \rangle + \underbrace{\langle {\mathbb{E}} Z, c \rangle}_{\langle i {\mathbb{E}} Z, i c \rangle} + i \langle {\mathbb{E}} Z, b \rangle - \underbrace{i \langle {\mathbb{E}} Y, c \rangle}_{- \langle {\mathbb{E}} Y, i c \rangle}\\ &= \langle {\mathbb{E}} Y + i {\mathbb{E}} Z, b \rangle + \langle {\mathbb{E}} Y + i {\mathbb{E}} Z, i c \rangle\\ &= \langle {\mathbb{E}} Y + i {\mathbb{E}} Z, a \rangle \end{align*}[/latex]

Number: Exercise 3.03
Question: Suppose [latex]${\mathbf{x}}$[/latex] satisfies [latex]${\mathbb{E}} \langle {\mathbf{X}}, {\mathbf{v}} \rangle = \langle {\mathbf{x}}, {\mathbf{v}} \rangle$[/latex] and [latex]${\mathbf{y}}$[/latex] satisfies [latex]${\mathbb{E}} \langle {\mathbf{Y}}, {\mathbf{v}} \rangle = \langle {\mathbf{y}}, {\mathbf{v}} \rangle$[/latex] for all [latex]${\mathbf{v}}$[/latex]. In order to justify an implicit claim in our definition of expectation for random vectors, verify that [latex]\begin{align*} {\mathbb{E}} \langle {\mathbf{X}} + {\mathbf{Y}}, {\mathbf{v}} \rangle = \langle {\mathbf{x}} + {\mathbf{y}}, {\mathbf{v}} \rangle \end{align*}[/latex] for all [latex]${\mathbf{v}}$[/latex]. In other words, verify that the expectation of a sum is indeed the sum of the expectations when all expectations are defined by property (ii), assuming that property (iii) holds for random variables.
Answer: First, let’s verify the claim in question when [latex]${\mathbf{X}}$[/latex] and [latex]${\mathbf{Y}}$[/latex] map to the complex plane; we’ll represent them by [latex]$X_1 + iX_2$[/latex] and [latex]$Y_1 + iY_2$[/latex] respectively. Based on Exercise 3.2, [latex]\begin{align*} {\mathbb{E}} ({\mathbf{X}} + {\mathbf{Y}}) &= {\mathbb{E}} (X_1 + iX_2 + Y_1 + iY_2)\\ &= {\mathbb{E}} [(X_1 + Y_1) + i(X_2 + Y_2)]\\ &= {\mathbb{E}} (X_1 + Y_1) + i {\mathbb{E}} (X_2 + Y_2)\\ &= {\mathbb{E}} X_1 + {\mathbb{E}} Y_1 + i {\mathbb{E}} X_2 + i {\mathbb{E}} Y_2\\ &= (\underbrace{{\mathbb{E}} X_1 + i {\mathbb{E}} X_2}_{{\mathbb{E}} \mathbf{X}}) + (\underbrace{{\mathbb{E}} Y_1 + i {\mathbb{E}} Y_2}_{{\mathbb{E}} \mathbf{Y}}) \end{align*}[/latex]
Using this result, we can prove the general case. [latex]\begin{align*} {\mathbb{E}} \langle {\mathbf{X}} + {\mathbf{Y}}, {\mathbf{v}} \rangle &= {\mathbb{E}} [\langle {\mathbf{X}}, {\mathbf{v}} \rangle + \langle {\mathbf{Y}}, {\mathbf{v}} \rangle]\\ &= {\mathbb{E}} \langle {\mathbf{X}}, {\mathbf{v}} \rangle + {\mathbb{E}} \langle {\mathbf{Y}}, {\mathbf{v}} \rangle\\ &= \langle {\mathbf{x}}, {\mathbf{v}} \rangle + \langle {\mathbf{y}}, {\mathbf{v}} \rangle\\ &= \langle {\mathbf{x}} + {\mathbf{y}}, {\mathbf{v}} \rangle \end{align*}[/latex]

Number: Exercise 3.04
Question: For a random vector [latex]${\mathbf{X}}$[/latex] and scalar [latex]$a$[/latex], show that [latex]${\mathbb{E}} a {\mathbf{X}} = a {\mathbb{E}} {\mathbf{X}}$[/latex].
Answer: If [latex]${\mathbf{X}}$[/latex] is a random vector taking values in the complex plane, then inner product is the ordinary product, so property (ii) of our definition of expectation says that [latex]${\mathbb{E}} a{\mathbf{X}} = a {\mathbb{E}} {\mathbf{X}}$[/latex].
Now we’ll use that fact to establish the more general result for random vectors. [latex]\begin{align*} {\mathbb{E}} \langle a {\mathbf{X}}, {\mathbf{v}} \rangle &= {\mathbb{E}} a \langle {\mathbf{X}}, {\mathbf{v}} \rangle\\ &= a {\mathbb{E}} \langle {\mathbf{X}}, {\mathbf{v}} \rangle\\ &= a \langle {\mathbb{E}} {\mathbf{X}}, {\mathbf{v}} \rangle\\ &= \langle a {\mathbb{E}} {\mathbf{X}}, {\mathbf{v}} \rangle \end{align*}[/latex]

Number: Exercise 3.05
Question: Suppose the random vector [latex]${\mathbf{X}}$[/latex] maps every point in the sample space to [latex]${\mathbf{w}}$[/latex]. Show that [latex]${\mathbb{E}} {\mathbf{X}} = {\mathbf{w}}$[/latex].
Answer: [latex]\begin{align*} {\mathbb{E}} \langle {\mathbf{X}}, {\mathbf{v}} \rangle &= {\mathbb{E}} \langle {\mathbf{w}}, {\mathbf{v}} \rangle\\ &= \langle {\mathbf{w}}, {\mathbf{v}} \rangle \underbrace{{\mathbb{E}} \boldsymbol{1}_{\Omega}}_1\\ &= \langle {\mathbf{w}}, {\mathbf{v}} \rangle \end{align*}[/latex] where [latex]$\Omega$[/latex] represents the whole sample space and therefore has probability [latex]$1$[/latex].

Number: Exercise 3.06
Question: Let [latex]${\mathbf{X}}$[/latex] be a random vector and [latex]${\mathbf{v}}$[/latex] be a non-random vector. Explain why [latex]${\mathbb{E}} ({\mathbf{X}} + {\mathbf{v}}) = {\mathbb{E}} {\mathbf{X}} + {\mathbf{v}}$[/latex].
Answer: The random vector [latex]${\mathbf{X}} + {\mathbf{v}}$[/latex] maps any [latex]$\omega$[/latex] to [latex]${\mathbf{X}}(\omega) + {\mathbf{v}}$[/latex]; we’re justified in treating [latex]${\mathbf{v}}$[/latex] as if it’s the random vector that maps every element of the sample space to the vector [latex]${\mathbf{v}}$[/latex]. By property (iii), [latex]${\mathbb{E}} ({\mathbf{X}} + {\mathbf{v}}) = {\mathbb{E}} {\mathbf{X}} + {\mathbb{E}} {\mathbf{v}}$[/latex], and by Exercise 3.5, [latex]${\mathbb{E}} {\mathbf{v}} = {\mathbf{v}}$[/latex].

Number: Exercise 3.07
Question: Suppose [latex]${\mathbb{E}} {\mathbf{X}} = {\mathbf{0}}$[/latex]. Show that the coordinate of [latex]${\mathbf{X}}$[/latex] with respect to [latex]${\mathbf{u}}$[/latex] has expectation [latex]$0$[/latex].
Answer: [latex]\begin{align*} {\mathbb{E}} \langle {\mathbf{X}}, {\mathbf{u}} \rangle &= \langle \underbrace{{\mathbb{E}} \mathbf{X}}_{\mathbf{0}}, {\mathbf{u}} \rangle\\ &= 0 \end{align*}[/latex]

Number: Exercise 3.08
Question: Let [latex]${\mathbf{X}}$[/latex] be a random vector that maps to a real vector space with an inner product. Show that the expected squared length of [latex]${\mathbf{X}}$[/latex] equals sum of the expected squares of its coordinates with respect to any orthonormal basis [latex]${\mathbf{u}_1, \ldots, \mathbf{u}_m}$[/latex].
Answer: This is a simple consequence of Parseval’s identity. [latex]\begin{align*} {\mathbb{E}} \| {\mathbf{X}} \|^2 &= {\mathbb{E}} [\langle {\mathbf{X}}, {\mathbf{u}}_1 \rangle^2 + \ldots + \langle {\mathbf{X}}, {\mathbf{u}}_m \rangle^2]\\ &= {\mathbb{E}} \langle {\mathbf{X}}, {\mathbf{u}}_1 \rangle^2 + \ldots + {\mathbb{E}} \langle {\mathbf{X}}, {\mathbf{u}}_m \rangle^2 \end{align*}[/latex]

Number: Exercise 3.09
Question: Let [latex]${\mathbf{X}} = (X_1, \ldots, X_n)$[/latex] be a random vector, [latex]${\mathbf{v}} = (v_1, \ldots, v_n)$[/latex] be a non-random vector, and [latex]${\mathbb{M}}$[/latex] be an [latex]$n \times m$[/latex] matrix. Show that [latex]\begin{align*} {\mathbb{E}} ({\mathbf{v}} + {\mathbb{M}} {\mathbf{X}}) &= {\mathbf{v}} + {\mathbb{M}} {\mathbb{E}} {\mathbf{X}}. \end{align*}[/latex]
Answer: From Exercise 3.6, [latex]${\mathbb{E}} ({\mathbf{v}} + {\mathbb{M}} {\mathbf{X}}) = {\mathbf{v}} + {\mathbb{E}} {\mathbb{M}} {\mathbf{X}}$[/latex]. Let [latex]${\mathbf{m}}_1, \ldots, {\mathbf{m}}_n$[/latex] be the rows of [latex]${\mathbb{M}}$[/latex]. Putting the expectation into each coordinate of the vector, [latex]\begin{align*} {\mathbb{E}} {\mathbb{M}} {\mathbf{X}} &= {\mathbb{E}} \begin{bmatrix} {\mathbf{m}}_1^T {\mathbf{X}} \\ \vdots \\ {\mathbf{m}}_n^T {\mathbf{X}} \end{bmatrix}\\ &= \begin{bmatrix} {\mathbb{E}} {\mathbf{m}}_1^T {\mathbf{X}} \\ \vdots \\ {\mathbb{E}} {\mathbf{m}}_n^T {\mathbf{X}} \end{bmatrix}\\ &= \begin{bmatrix} {\mathbf{m}}_1^T {\mathbb{E}} {\mathbf{X}} \\ \vdots \\ {\mathbf{m}}_n^T {\mathbb{E}} {\mathbf{X}} \end{bmatrix}\\ &= {\mathbb{M}} {\mathbb{E}} {\mathbf{X}}. \end{align*}[/latex]

Number: Exercise 3.10
Question: Suppose [latex]${\mathbf{X}}$[/latex] is a discrete random vector with probability mass function [latex]$p$[/latex] on [latex]$\{{\mathbf{x}}_1, \ldots, {\mathbf{x}}_n\}$[/latex]. Show that [latex]${\mathbb{E}} {\mathbf{X}} = \sum_i {\mathbf{x}}_i p({\mathbf{x}}_i)$[/latex].
Answer: The random vector can be represented by the sum [latex]\begin{align*} {\mathbf{X}}(\omega) = {\mathbf{x}}_1 \boldsymbol{1}_{{\mathbf{X}}(\omega) = {\mathbf{x}}_1} + \ldots + {\mathbf{x}}_n \boldsymbol{1}_{{\mathbf{X}}(\omega) = {\mathbf{x}}_n} \end{align*}[/latex] Taking the expectation, [latex]\begin{align*} {\mathbb{E}} {\mathbf{X}} &= {\mathbb{E}} [{\mathbf{x}}_1 \boldsymbol{1}_{{\mathbf{X}} = {\mathbf{x}}_1} + \ldots + {\mathbf{x}}_n \boldsymbol{1}_{{\mathbf{X}} = {\mathbf{x}}_n}]\\ &= {\mathbf{x}}_1 \underbrace{{\mathbb{E}} \boldsymbol{1}_{{\mathbf{X}} = {\mathbf{x}}_1}}_{p({\mathbf{x}}_1)} + \ldots + {\mathbf{x}}_n \underbrace{{\mathbb{E}} \boldsymbol{1}_{{\mathbf{X}} = {\mathbf{x}}_n}}_{p({\mathbf{x}}_n)} \end{align*}[/latex] by property (i) of the definition of expectation.

Number: Exercise 3.11
Question: Let [latex]$X$[/latex] be a discrete random variable whose possible values are the positive integers. In particular, suppose that [latex]${\mathbb{P}} \{X = k\}$[/latex] is proportional to [latex]$1/k^2$[/latex] for [latex]$k \in \{1, 2, \ldots\}$[/latex]. What’s the expectation of [latex]$X$[/latex]?
Answer: Recall that [latex]$\sum_{k=1}^\infty \frac{1}{k^2} = \pi^2/6$[/latex], so this distribution is well-defined. However, its expectation is [latex]\begin{align*} {\mathbb{E}} X &= \sum_{k=1}^\infty k {\mathbb{P}} \{X = k\}\\ &= \sum_{k=1}^\infty k \tfrac{6}{\pi^2} \tfrac{1}{k^2}\\ &= \tfrac{6}{\pi^2} \sum_{k=1}^\infty \tfrac{1}{k}\\ &= \infty. \end{align*}[/latex]

Number: Exercise 3.12
Question: Suppose [latex]$X$[/latex] is uncorrelated with each of [latex]$Y_1, \ldots, Y_n$[/latex]. Show that [latex]$X$[/latex] is also uncorrelated with [latex]$a_1 Y_1 + \ldots + a_n Y_n$[/latex].
Answer: [latex]\begin{align*} {\mathbb{E}} (X - {\mathbb{E}} X)[a_1 Y_1 + \ldots + a_n Y_n - {\mathbb{E}} (a_1 Y_1 + \ldots + a_n Y_n)] &= {\mathbb{E}} (X - {\mathbb{E}} X)[a_1 Y_1 + \ldots + a_n Y_n - (a_1 {\mathbb{E}} Y_1 + \ldots + a_n {\mathbb{E}} Y_n)]\\ &= a_1 \underbrace{{\mathbb{E}} (X - {\mathbb{E}} X)(Y_1 - {\mathbb{E}} Y_1)}_0 + \ldots + a_n \underbrace{{\mathbb{E}} (X - {\mathbb{E}} X)(Y_n - {\mathbb{E}} Y_n)}_0\\ &= 0 \end{align*}[/latex]

Number: Exercise 3.13
Question: If events [latex]$E_1, \ldots, E_m$[/latex] are independent (meaning that their indicator functions are independent random variables) and each has probability [latex]$q$[/latex], what’s the probability that at least one of them occurs?
Answer: Each event has probability [latex]$1-q$[/latex] of not occurring. The probability of an intersection of independent events equals the product of their probabilities, so the probability that none of the events occur is [latex]$(1-q)^m$[/latex]. The probability that at least one occurs is [latex]$1 - (1-q)^m$[/latex] since it’s the complement of the event that none of them occur.

Number: Exercise 3.14
Question: Let [latex]${\mathbf{X}}$[/latex] be a random vector and [latex]${\mathbf{v}}$[/latex] be a non-random vector. Show that if [latex]${\mathbb{E}} \langle {\mathbf{X}}, {\mathbf{v}} \rangle$[/latex] is real, then it’s equal to [latex]${\mathbb{E}} \langle {\mathbf{v}}, {\mathbf{X}} \rangle$[/latex].
Answer: We can express the first inner product as [latex]$\langle {\mathbf{X}}, {\mathbf{v}} \rangle = Y + iZ$[/latex] for some random variables [latex]$Y$[/latex] and [latex]$Z$[/latex]. If its expectation [latex]${\mathbb{E}} Y + i {\mathbb{E}} Z$[/latex] is real, then [latex]${\mathbb{E}} Z$[/latex] must be [latex]$0$[/latex]. The other inner product is the complex conjugate [latex]$\langle {\mathbf{v}}, {\mathbf{X}} \rangle = Y - iZ$[/latex]. Its expectation [latex]${\mathbb{E}} Y - i {\mathbb{E}} Z$[/latex] simplifies to [latex]${\mathbb{E}} Y$[/latex] as well.

Number: Exercise 3.15
Question: Let [latex]${\mathbf{Y}}$[/latex] be a random vector with expectation [latex]${\boldsymbol{\mu}}$[/latex]. Find the non-random vector [latex]${\mathbf{v}}$[/latex] that minimizes [latex]${\mathbb{E}} \| {\mathbf{Y}} - {\mathbf{v}} \|^2$[/latex].
Answer: By the bias-variance decomposition, the objective function equals [latex]$\| {\mathbf{v}} - {\boldsymbol{\mu}} \|^2 + {\mathbb{E}} \| {\mathbf{Y}} - {\boldsymbol{\mu}} \|^2$[/latex]. The second term doesn’t depend on [latex]${\mathbf{v}}$[/latex], so we can minimize the sum by taking [latex]${\mathbf{v}}$[/latex] to be [latex]${\boldsymbol{\mu}}$[/latex] which makes the first term zero.

Number: Exercise 3.16
Question: Explain how Exercise 2.2 is an instance of the bias-variance decomposition.
Answer: If the distribution of the random variable [latex]$Y$[/latex] is the empirical distribution defined by [latex]${\mathbf{y}} = (y_1, \ldots, y_n)$[/latex], then its expectation is [latex]$\bar{y}$[/latex]. By the bias-variance decomposition, [latex]\begin{align*} {\mathbb{E}} (Y - a)^2 &= (a - {\mathbb{E}} Y)^2 + {\mathbb{E}} (Y - {\mathbb{E}} Y)^2\\ &\Updownarrow\\ \tfrac{1}{n} \sum_i (y_i - a)^2 &= (a - \bar{y})^2 + \tfrac{1}{n} \sum_i (y_i - \bar{y})^2. \end{align*}[/latex]

Number: Exercise 3.17
Question: Let [latex]${\mathbf{Y}}$[/latex] be a random vector that is an unbiased estimator for [latex]${\boldsymbol{\theta}}$[/latex], that is [latex]${\mathbb{E}} {\mathbf{Y}} = {\boldsymbol{\theta}}$[/latex]. If [latex]$\lambda \in {\pmb{\mathbb{R}}}$[/latex], express [latex]$\| {\mathbb{E}} (\lambda {\mathbf{Y}}) - {\boldsymbol{\theta}} \|^2$[/latex] (which can be thought of as the squared bias of the estimator [latex]$\lambda {\mathbf{Y}}$[/latex]) in terms of [latex]$\lambda$[/latex] and [latex]$\| {\boldsymbol{\theta}} \|^2$[/latex].
Answer: [latex]\begin{align*} \| {\mathbb{E}} (\lambda {\mathbf{Y}}) - {\boldsymbol{\theta}} \|^2 &= \| \lambda \underbrace{{\mathbb{E}} \mathbf{Y}}_{\boldsymbol{\theta}} - {\boldsymbol{\theta}} \|^2\\ &= \| (\lambda - 1) {\boldsymbol{\theta}} \|^2\\ &= (1 - \lambda)^2 \| {\boldsymbol{\theta}} \|^2 \end{align*}[/latex] Note that the factor [latex]$(\lambda - 1)^2$[/latex] is equal to [latex]$(1 - \lambda)^2$[/latex] which is a bit more intuitive when [latex]$\lambda \in [0, 1]$[/latex].

Number: Exercise 3.18
Question: Let [latex]${\mathbf{Y}}$[/latex] be a random vector, and let [latex]$\lambda \in {\pmb{\mathbb{R}}}$[/latex]. Express [latex]${\mathbb{E}} \| \lambda {\mathbf{Y}} - {\mathbb{E}} (\lambda {\mathbf{Y}}) \|^2$[/latex] in terms of [latex]$\lambda$[/latex] and [latex]${\mathbb{E}} \| {\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}} \|^2$[/latex].
Answer: Factoring out [latex]$\lambda$[/latex], [latex]\begin{align*} {\mathbb{E}} \| \lambda {\mathbf{Y}} - {\mathbb{E}} (\lambda {\mathbf{Y}}) \|^2 &= {\mathbb{E}} \| \lambda ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) \|^2\\ &= \lambda^2 {\mathbb{E}} \| {\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}} \|^2. \end{align*}[/latex]

Number: Exercise 3.19
Question: Let [latex]${\mathbf{Y}}$[/latex] be a random vector that is an unbiased estimator for [latex]${\boldsymbol{\theta}} \in {\pmb{\mathbb{R}}}^n$[/latex]. Use the bias-variance decomposition along with your results from Exercises 3.17 and 3.18 to find an expression for [latex]$\lambda \in {\pmb{\mathbb{R}}}$[/latex] (in terms of [latex]$\| {\boldsymbol{\theta}} \|^2$[/latex] and [latex]${\mathbb{E}} \| {\mathbf{Y}} - {\boldsymbol{\theta}} \|^2$[/latex]) for which [latex]${\mathbb{E}} \| {\boldsymbol{\theta}} - \lambda {\mathbf{Y}} \|^2$[/latex] is as small as possible.
Answer: By the bias-variance decomposition and our previous results, [latex]\begin{align*} {\mathbb{E}} \| {\boldsymbol{\theta}} - \lambda {\mathbf{Y}} \|^2 &= \| {\mathbb{E}} (\lambda {\mathbf{Y}}) - {\boldsymbol{\theta}} \|^2 + {\mathbb{E}} \| \lambda {\mathbf{Y}} - {\mathbb{E}} (\lambda {\mathbf{Y}}) \|^2\\ &= (1 - \lambda)^2 \| {\boldsymbol{\theta}} \|^2 + \lambda^2 {\mathbb{E}} \| {\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}} \|^2. \end{align*}[/latex] Taking the derivative with respect to [latex]$\lambda$[/latex], and setting it to zero, we get the critical [latex]$\lambda^*$[/latex]: [latex]\begin{align*} (1 - \lambda^*) \| {\boldsymbol{\theta}} \|^2 &= \lambda^* {\mathbb{E}} \| {\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}} \|^2 \end{align*}[/latex] is solved by [latex]$\lambda^* = \frac{\| {\boldsymbol{\theta}} \|^2}{\| {\boldsymbol{\theta}} \|^2 + {\mathbb{E}} \| {\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}} \|^2}$[/latex]. Realize of course that when estimating an unknown parameter [latex]${\boldsymbol{\theta}}$[/latex], we can’t actually calculate this optimal value.

Number: Exercise 3.20
Question: Let [latex]${\pmb{\mathbb{X}}}$[/latex] be a random matrix whose entries have finite expectations, and let [latex]${\mathbb{M}}$[/latex] be a non-random matrix. Assuming [latex]${\mathbb{M}} {\pmb{\mathbb{X}}}$[/latex] is well-defined, show that [latex]${\mathbb{E}} {\mathbb{M}} {\pmb{\mathbb{X}}} = {\mathbb{M}} {\mathbb{E}} {\pmb{\mathbb{X}}}$[/latex]. Alternatively, assuming [latex]${\pmb{\mathbb{X}}} {\mathbb{M}}$[/latex] is well-defined, show that [latex]${\mathbb{E}} {\pmb{\mathbb{X}}} {\mathbb{M}} = ({\mathbb{E}} {\pmb{\mathbb{X}}}) {\mathbb{M}}$[/latex].
Answer: The [latex]$(i,j)$[/latex] entry of [latex]${\mathbb{E}} {\mathbb{M}} {\pmb{\mathbb{X}}}$[/latex] is [latex]${\mathbb{E}} {\mathbf{m}}_i^T {\mathbf{X}}_j = {\mathbf{m}}_i^T {\mathbb{E}} {\mathbf{X}}_j$[/latex] where [latex]${\mathbf{m}}_i$[/latex] represents the [latex]$i$[/latex]th row of [latex]${\mathbb{M}}$[/latex] and [latex]${\mathbf{X}}_j$[/latex] represents the [latex]$j$[/latex]th column of [latex]${\pmb{\mathbb{X}}}$[/latex]. This is also the [latex]$(i,j)$[/latex] entry of [latex]${\mathbb{M}} {\mathbb{E}} {\pmb{\mathbb{X}}}$[/latex]. Similarly, the [latex]$(i,j)$[/latex] entry of [latex]${\mathbb{E}} {\pmb{\mathbb{X}}} {\mathbb{M}}$[/latex] is [latex]${\mathbb{E}} {\mathbf{X}}_i^T {\mathbf{m}}_j = ({\mathbb{E}} {\mathbf{X}}_i)^T {\mathbf{m}}_j$[/latex] where [latex]${\mathbf{X}}_i$[/latex] represents the [latex]$i$[/latex]th row of [latex]${\pmb{\mathbb{X}}}$[/latex] and [latex]${\mathbf{m}}_j$[/latex] represents the [latex]$j$[/latex]th column of [latex]${\mathbb{M}}$[/latex]

Number: Exercise 3.21
Question: Show that an alternative expression for the covariance matrix of [latex]${\mathbf{Y}}$[/latex] is [latex]${\mathbb{E}} [({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}})^T]$[/latex].
Answer: We’ll work out the matrix resulting from the multiplication then move the expectation into the matrix entries. [latex]\begin{align*} {\mathbb{E}} [({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}})^T] &= {\mathbb{E}} \begin{bmatrix} Y_1 - {\mathbb{E}} Y_1 \\ \vdots \\ Y_n - {\mathbb{E}} Y_n \end{bmatrix} \begin{bmatrix} Y_1 - {\mathbb{E}} Y_1 & \cdots & Y_n - {\mathbb{E}} Y_n \end{bmatrix}\\ &= {\mathbb{E}} \begin{bmatrix} (Y_1 - {\mathbb{E}} Y_1)(Y_1 - {\mathbb{E}} Y_1) & \cdots & (Y_1 - {\mathbb{E}} Y_1)(Y_n - {\mathbb{E}} Y_n) \\ \vdots & \ddots & \vdots \\ (Y_n - {\mathbb{E}} Y_n)(Y_1 - {\mathbb{E}} Y_1) & \cdots & (Y_n - {\mathbb{E}} Y_n)(Y_n - {\mathbb{E}} Y_n) \end{bmatrix}\\ &= \begin{bmatrix} {\mathbb{E}} [(Y_1 - {\mathbb{E}} Y_1)(Y_1 - {\mathbb{E}} Y_1)] & \cdots & {\mathbb{E}} [(Y_1 - {\mathbb{E}} Y_1)(Y_n - {\mathbb{E}} Y_n)] \\ \vdots & \ddots & \vdots \\ {\mathbb{E}} [(Y_n - {\mathbb{E}} Y_n)(Y_1 - {\mathbb{E}} Y_1)] & \cdots & {\mathbb{E}} [(Y_n - {\mathbb{E}} Y_n)(Y_n - {\mathbb{E}} Y_n)] \end{bmatrix} \end{align*}[/latex] The empirical covariance matrix expression in Equation 1.7 can be understood as an empirical version of this.

Number: Exercise 3.22
Question: Let [latex]${\mathbf{Y}}$[/latex] be an [latex]${\pmb{\mathbb{R}}}^n$[/latex]-valued random vector, and let [latex]${\mathbf{v}} \in {\pmb{\mathbb{R}}}^n$[/latex]. Use Exercise 3.21 to show that the covariance of [latex]${\mathbf{v}} + {\mathbf{Y}}$[/latex] has the same covariance matrix as [latex]${\mathbf{Y}}$[/latex].
Answer: [latex]\begin{align*} \text{cov}\, ({\mathbf{v}} + {\mathbf{Y}}) &= {\mathbb{E}} [({\mathbf{v}} + {\mathbf{Y}} - {\mathbb{E}} ({\mathbf{v}} + {\mathbf{Y}}) ) ({\mathbf{v}} + {\mathbf{Y}} - {\mathbb{E}} ({\mathbf{v}} + {\mathbf{Y}}))^T]\\ &= {\mathbb{E}} [({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}})^T]\\ &= \text{cov}\, {\mathbf{Y}} \end{align*}[/latex]

Number: Exercise 3.23
Question: Let [latex]${\mathbf{Y}}$[/latex] be a random vector with covariance matrix [latex]${\mathbb{C}}$[/latex]. Let [latex]${\mathbf{v}}$[/latex] be a non-random vector, and let [latex]${\mathbb{M}}$[/latex] be a real matrix. Show that the covariance of [latex]${\mathbf{v}} + {\mathbb{M}} {\mathbf{Y}}$[/latex] is [latex]${\mathbb{M}} {\mathbb{C}} {\mathbb{M}}^T$[/latex].
Answer: By Exercise 3.22, [latex]$\text{cov}\, ({\mathbf{v}} + {\mathbb{M}} {\mathbf{Y}}) = \text{cov}\, {\mathbb{M}} {\mathbf{Y}}$[/latex]. [latex]\begin{align*} \text{cov}\, {\mathbb{M}} {\mathbf{Y}} &= {\mathbb{E}} [({\mathbb{M}} {\mathbf{Y}} - {\mathbb{E}} {\mathbb{M}} {\mathbf{Y}}) ({\mathbb{M}} {\mathbf{Y}} - {\mathbb{E}} {\mathbb{M}} {\mathbf{Y}})^T]\\ &= {\mathbb{E}} [({\mathbb{M}} {\mathbf{Y}} - {\mathbb{M}} {\mathbb{E}} {\mathbf{Y}}) ({\mathbb{M}} {\mathbf{Y}} - {\mathbb{M}} {\mathbb{E}} {\mathbf{Y}})^T]\\ &= {\mathbb{E}} [({\mathbb{M}} ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}})) ({\mathbb{M}} ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}))^T]\\ &= {\mathbb{M}} ({\mathbb{E}} [({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}})^T]) {\mathbb{M}}^T\\ &= {\mathbb{M}} {\mathbb{C}} {\mathbb{M}}^T \end{align*}[/latex]

Number: Exercise 3.24
Question: Show that every covariance matrix is positive semi-definite.
Answer: To satisfy the definition, we need to show that every quadratic form is non-negative. We’ll use the covariance expression from Exercise 3.21 and consider its quadratic form for an arbitrary vector [latex]${\mathbf{v}}$[/latex], [latex]\begin{align*} {\mathbf{v}}^T {\mathbb{E}} [({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}})^T] {\mathbf{v}} &= {\mathbb{E}} [{\mathbf{v}}^T ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}})^T] {\mathbf{v}}\\ &= {\mathbb{E}} [{\mathbf{v}}^T ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}})^T{\mathbf{v}}]\\ &= {\mathbb{E}} \langle {\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}, {\mathbf{v}} \rangle^2. \end{align*}[/latex] The expectation of a non-negative random variable has to be non-negative.

Number: Exercise 3.25
Question: Show that [latex]${\mathbb{E}} \| {\mathbf{X}} - {\mathbb{E}} {\mathbf{X}} \|^2 = \text{tr} \, (\text{cov}\, {\mathbf{X}})$[/latex].
Answer: [latex]\begin{align*} {\mathbb{E}} \| {\mathbf{X}} - {\mathbb{E}} {\mathbf{X}} \|^2 &= {\mathbb{E}} [(X_1 - {\mathbb{E}} X_1)^2 + \ldots + (X_n - {\mathbb{E}} X_n)^2]\\ &= {\mathbb{E}} (X_1 - {\mathbb{E}} X_1)^2 + \ldots + {\mathbb{E}} (X_n - {\mathbb{E}} X_n)^2 \end{align*}[/latex] These variances are the diagonals of the covariance matrix, so its trace is their sum.

Number: Exercise 3.26
Question: Suppose [latex]$X_1, \ldots, X_n$[/latex] are uncorrelated random variables. Show that the variance of their sum equals the sum of their variances.
Answer: Let [latex]${\mathbf{X}} := (X_1, \ldots, X_n)$[/latex], and let [latex]$\sigma_1^2, \ldots, \sigma_n^2$[/latex] represent the variances. Using Exercise 3.23, [latex]\begin{align*} \text{var}\, \sum_i X_i &= \text{var}\, {\mathbf{1}}^T {\mathbf{X}}\\ &= {\mathbf{1}}^T \begin{bmatrix} \sigma_1^2 & & \\ & \ddots & \\ & & \sigma_n^2 \end{bmatrix} {\mathbf{1}}\\ &= {\mathbf{1}}^T \begin{bmatrix} \sigma_1^2 \\ \vdots \\ \sigma_n^2 \end{bmatrix}\\ &= \sum_i \sigma_i^2. \end{align*}[/latex]

Number: Exercise 3.27
Question: Let [latex]${\boldsymbol{\epsilon}}$[/latex] be a random vector with expectation [latex]${\mathbf{0}}$[/latex] and covariance matrix [latex]$\sigma^2 {\mathbb{I}}$[/latex]. Let [latex]${\mathbf{v}}$[/latex] be a non-random vector, and let [latex]${\mathbb{H}}$[/latex] be an orthogonal projection matrix. Find the covariance matrix of [latex]${\mathbb{H}} ({\mathbf{v}} + {\boldsymbol{\epsilon}})$[/latex].
Answer: Distribute the matrix multiplication to get [latex]${\mathbb{H}} {\mathbf{v}} + {\mathbb{H}} {\boldsymbol{\epsilon}}$[/latex]. According to Exercise 3.23, the covariance is [latex]\begin{align*} {\mathbb{H}} (\sigma^2 {\mathbb{I}}) {\mathbb{H}}^T &= \sigma^2 {\mathbb{H}} {\mathbb{H}}^T\\ &= \sigma^2 {\mathbb{H}} \end{align*}[/latex] by symmetry and idempotence of orthogonal projection matrices.

Number: Exercise 3.28
Question: Let [latex]${\mathbf{X}}$[/latex] have expectation [latex]${\boldsymbol{\mu}}_{\mathbf{X}}$[/latex] and [latex]${\mathbf{Y}}$[/latex] have expectation [latex]${\boldsymbol{\mu}}_{\mathbf{Y}}$[/latex]. Show that the expected inner product between the centered vectors [latex]${\mathbf{X}} - {\boldsymbol{\mu}}_{\mathbf{X}}$[/latex] and [latex]${\mathbf{Y}} - {\boldsymbol{\mu}}_{\mathbf{Y}}$[/latex] is the same as the expected inner product when only one of them is centered.
Answer: [latex]\begin{align*} {\mathbb{E}} \langle {\mathbf{X}} - {\boldsymbol{\mu}}_{\mathbf{X}}, {\mathbf{Y}} - {\boldsymbol{\mu}}_{\mathbf{Y}} \rangle &= {\mathbb{E}} \langle {\mathbf{X}} - {\boldsymbol{\mu}}_{\mathbf{X}}, {\mathbf{Y}} \rangle - {\mathbb{E}} \langle {\mathbf{X}} - {\boldsymbol{\mu}}_{\mathbf{X}}, {\boldsymbol{\mu}}_{\mathbf{Y}} \rangle\\ &= {\mathbb{E}} \langle {\mathbf{X}} - {\boldsymbol{\mu}}_{\mathbf{X}}, {\mathbf{Y}} \rangle - \langle \underbrace{{\mathbb{E}} {\mathbf{X}} - {\boldsymbol{\mu}}_{\mathbf{X}}}_{\mathbf{0}}, {\boldsymbol{\mu}}_{\mathbf{Y}} \rangle\\ &= {\mathbb{E}} \langle {\mathbf{X}} - {\boldsymbol{\mu}}_{\mathbf{X}}, {\mathbf{Y}} \rangle \end{align*}[/latex] The same argument works for [latex]${\mathbf{Y}} - {\boldsymbol{\mu}}_{\mathbf{Y}}$[/latex] if you keep Exercise 3.14 in mind.

Number: Exercise 3.29
Question: Use Exercise 3.28 to observe that [latex]\begin{align*} \langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, {\mathbf{y}} \rangle = \langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, {\mathbf{y}} - {\bar{y}} {\mathbf{1}} \rangle. \end{align*}[/latex]
Answer: Let the joint distribution of [latex]$(X, Y)$[/latex] be the empirical distribution of [latex]$(x_1, y_1), \ldots, (x_n, y_n)$[/latex]. [latex]\begin{align*} \langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, {\mathbf{y}} \rangle &= n \tfrac{1}{n} \sum_i [(x_i - {\bar{x}}) y_i]\\ &= n {\mathbb{E}} [(X - {\mathbb{E}} X) Y]\\ &= n {\mathbb{E}} [(X - {\mathbb{E}} X) (Y - {\mathbb{E}} Y)]\\ &= n \tfrac{1}{n} \sum_i [(x_i - {\bar{x}}) (y_i - {\bar{y}})]\\ &= \langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, {\mathbf{y}} - {\bar{y}} {\mathbf{1}} \rangle \end{align*}[/latex]

Number: Exercise 3.30
Question: Let [latex]${\mathbf{X}}$[/latex] be a random vector mapping to a real vector space, and let [latex]${\mathbf{v}}$[/latex] be a non-random vector. Show that the variance of the coordinate of [latex]${\mathbf{X}}$[/latex] with respect to [latex]${\mathbf{u}}$[/latex] is the same as the variance of the coordinate of [latex]${\mathbf{X}} + {\mathbf{v}}$[/latex] with respect to [latex]${\mathbf{u}}$[/latex].
Answer: The difference between [latex]$\langle {\mathbf{X}} + {\mathbf{v}}, {\mathbf{u}} \rangle$[/latex] and [latex]$\langle {\mathbf{X}}, {\mathbf{u}} \rangle$[/latex] is [latex]$\langle {\mathbf{v}}, {\mathbf{u}} \rangle$[/latex] which is non-random. By Exercise 3.23, we can conclude that they must therefore have the same variance.

Number: Exercise 3.31
Question: If [latex]${\mathbf{X}}$[/latex] has expectation [latex]${\boldsymbol{\mu}}$[/latex], find the expectation of the centered version [latex]${\mathbf{X}} - {\boldsymbol{\mu}}$[/latex].
Answer: [latex]\begin{align*} {\mathbb{E}} ({\mathbf{X}} - {\boldsymbol{\mu}}) &= \underbrace{{\mathbb{E}} \mathbf{X}}_{\boldsymbol{\mu}} - {\boldsymbol{\mu}}\\ &= {\mathbf{0}} \end{align*}[/latex]

Number: Exercise 3.32
Question: Let [latex]${\mathbb{M}}$[/latex] be a positive definite matrix. Based on Exercises 1.24 and 1.58, explain why the inverse of the square root of [latex]${\mathbb{M}}$[/latex] is the same as the square root of the inverse of [latex]${\mathbb{M}}$[/latex].
Answer: To find the square root of a positive semi-definite matrix, you replace the eigenvalues by their square roots. To find the inverse of an invertible symmetric matrix, you replace the eigenvalues by their reciprocals. No matter which order you do these two operations in, you end up with the same matrix: [latex]\begin{align*} \frac{1}{\sqrt{\lambda_1}} {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \frac{1}{\sqrt{\lambda_n}} {\mathbf{q}}_n {\mathbf{q}}_n^T \end{align*}[/latex] where [latex]${\mathbf{q}}_1, \ldots, {\mathbf{q}}_n$[/latex] are eigenvectors of [latex]${\mathbb{M}}$[/latex] with eigenvalues [latex]$\lambda_1, \ldots, \lambda_n$[/latex].

Number: Exercise 3.33
Question: Let [latex]${\mathbf{Y}}$[/latex] have expectation [latex]${\boldsymbol{\mu}}$[/latex] and covariance matrix [latex]${\mathbb{C}}$[/latex]. Find the expectation and covariance of [latex]${\mathbb{C}}^{-1/2} ({\mathbf{Y}} - {\boldsymbol{\mu}})$[/latex].
Answer: The random vector [latex]${\mathbf{Y}} - {\boldsymbol{\mu}}$[/latex] has expectation zero, so based on Exercise 3.9, [latex]${\mathbb{C}}^{-1/2} ({\mathbf{Y}} - {\boldsymbol{\mu}})$[/latex] has expectation [latex]${\mathbb{C}}^{-1/2} {\mathbf{0}} = {\mathbf{0}}$[/latex]. For the covariance, we apply the formula from Exercise 3.23 to get [latex]\begin{align*} \text{cov}\, [{\mathbb{C}}^{-1/2} ({\mathbf{Y}} - {\boldsymbol{\mu}})] &= {\mathbb{C}}^{-1/2} {\mathbb{C}} ({\mathbb{C}}^{-1/2})^T\\ &= \underbrace{{\mathbb{C}}^{-1/2} {\mathbb{C}}^{1/2}}_{\mathbb{I}} \underbrace{{\mathbb{C}}^{1/2} {\mathbb{C}}^{-1/2}}_{\mathbb{I}}\\ &= {\mathbb{I}}. \end{align*}[/latex]

Number: Exercise 3.34
Question: If [latex]${\mathbf{Y}}$[/latex] has expectation [latex]${\boldsymbol{\mu}}$[/latex] and a positive definite covariance matrix [latex]${\mathbb{C}}$[/latex], find the expected squared Mahalanobis distance from [latex]${\mathbf{Y}}$[/latex] to its own distribution.
Answer: Let [latex]${\mathbf{Z}} := {\mathbb{C}}^{-1/2}({\mathbf{Y}} - {\boldsymbol{\mu}})$[/latex] represent the standardized version of [latex]${\mathbf{Y}}$[/latex], and let [latex]$(Z_1, \ldots, Z_n)$[/latex] represent its coordinates. Notice that the squared Mahalanobis distance from [latex]${\mathbf{Y}}$[/latex] to its distribution is exactly the squared norm of the standardized version. [latex]\begin{align*} {\mathbb{E}} \| {\mathbb{C}}^{-1/2} [{\mathbf{Y}} - {\boldsymbol{\mu}}] \|^2 &= {\mathbb{E}} \| {\mathbf{Z}} \|^2\\ &= {\mathbb{E}} Z_1^2 + \ldots + {\mathbb{E}} Z_n^2\\ &= \underbrace{\text{var}\, Z_1}_1 + \ldots + \underbrace{\text{var}\, Z_n}_1\\ &= n \end{align*}[/latex] The expected squared Mahalanobis distance is the dimension of the vector space that [latex]${\mathbf{Y}}$[/latex] inhabits.

Number: Exercise 3.35
Question: Let [latex]${\mathbb{H}}$[/latex] be the orthogonal projection matrix onto a [latex]$d$[/latex]-dimensional subspace [latex]${\mathcal{S}} \subseteq {\pmb{\mathbb{R}}}^n$[/latex], and let [latex]${\mathbf{Y}}$[/latex] be a random vector with covariance matrix [latex]$\sigma^2 {\mathbb{I}}$[/latex]. Show that [latex]\begin{align*} {\mathbb{E}} \| {\mathbb{H}} {\mathbf{Y}} \|^2 &= d \sigma^2 + \| {\mathbb{H}} {\boldsymbol{\mu}} \|^2. \end{align*}[/latex]
Answer: By comparison to Equation 3.1, all that remains is to verify that the trace of [latex]${\mathbb{H}} \sigma^2 {\mathbb{I}}$[/latex] is [latex]$d \sigma^2$[/latex]. [latex]\begin{align*} \text{tr} \, [{\mathbb{H}} \sigma^2 {\mathbb{I}}] &= \sigma^2 \text{tr} \, {\mathbb{H}}\\ &= d \sigma^2 \end{align*}[/latex] because according to Exercise 1.67 the trace of an orthogonal projection matrix equals the dimension of the subspace that it projects onto.

Number: Exercise 3.36
Question: Let [latex]$X_1, \ldots, X_n$[/latex] all have expectation [latex]${\boldsymbol{\mu}}_X$[/latex], and let [latex]$Y_1, \ldots, Y_n$[/latex] all have expectation [latex]${\boldsymbol{\mu}}_Y$[/latex]. Suppose cov[latex]$(X_i, Y_j)$[/latex] equals [latex]$\sigma_{X,Y}$[/latex] if [latex]$i=j$[/latex] and zero otherwise. Find the expectation of [latex]\begin{align*} \sum_i (X_i - {\bar{X}}) (Y_i - {\bar{Y}}). \end{align*}[/latex]
Answer: Let [latex]${\mathbf{Y}} := (Y_1, \ldots, Y_n)$[/latex] and [latex]${\mathbf{X}} := (X_1, \ldots, X_n)$[/latex]. The matrix [latex]\begin{align*} ({\mathbf{X}} - {\boldsymbol{\mu}}_X {\mathbf{1}})({\mathbf{Y}} - {\boldsymbol{\mu}}_Y {\mathbf{1}})^T &= \begin{bmatrix} (X_1 - {\boldsymbol{\mu}}_X)(Y_1 - {\boldsymbol{\mu}}_Y) & \cdots & (X_1 - {\boldsymbol{\mu}}_X)(Y_n - {\boldsymbol{\mu}}_Y)\\ \vdots & \ddots & \vdots \\ (X_n - {\boldsymbol{\mu}}_X)(Y_1 - {\boldsymbol{\mu}}_Y) & \cdots & (X_n - {\boldsymbol{\mu}}_X)(Y_n - {\boldsymbol{\mu}}_Y)\end{bmatrix} \end{align*}[/latex] has expectation [latex]$\sigma_{X, Y} {\mathbb{I}}$[/latex].
Let [latex]${\mathbb{J}}$[/latex] be the orthogonal projection matrix onto the span of [latex]$\{{\mathbf{1}}\}$[/latex]. We’ll use the same trace cyclic permutation trick that was advantageous for evaluating expected quadratic forms. [latex]\begin{align*} {\mathbb{E}} \sum_i (X_i - {\bar{X}}) (Y_i - {\bar{Y}}) &= {\mathbb{E}} ({\mathbf{Y}} - {\bar{Y}} {\mathbf{1}})^T ({\mathbf{X}} - {\bar{X}} {\mathbf{1}})\\ &= {\mathbb{E}} [({\mathbb{I}} - {\mathbb{J}}) {\mathbf{Y}}]^T [({\mathbb{I}} - {\mathbb{J}}) {\mathbf{X}}]\\ &= {\mathbb{E}} [({\mathbb{I}} - {\mathbb{J}}) ({\mathbf{Y}} - {\boldsymbol{\mu}}_Y {\mathbf{1}})]^T [({\mathbb{I}} - {\mathbb{J}}) ({\mathbf{X}} - {\boldsymbol{\mu}}_X {\mathbf{1}})]\\ &= {\mathbb{E}} ({\mathbf{Y}} - {\boldsymbol{\mu}}_Y {\mathbf{1}})^T ({\mathbb{I}} - {\mathbb{J}}) ({\mathbf{X}} - {\boldsymbol{\mu}}_X {\mathbf{1}})\\ &= {\mathbb{E}} \text{tr} \, [({\mathbf{Y}} - {\boldsymbol{\mu}}_Y {\mathbf{1}})^T ({\mathbb{I}} - {\mathbb{J}}) ({\mathbf{X}} - {\boldsymbol{\mu}}_X {\mathbf{1}})]\\ &= \text{tr} \, [({\mathbb{I}} - {\mathbb{J}}) \underbrace{{\mathbb{E}} ({\mathbf{X}} - {\boldsymbol{\mu}}_X {\mathbf{1}})({\mathbf{Y}} - {\boldsymbol{\mu}}_Y {\mathbf{1}})^T}_{\sigma_{X,Y} \mathbb{I}}]\\ &= (n-1) \sigma_{X,Y} \end{align*}[/latex]

Number: Exercise 4.01
Question: Suppose that [latex]$Y_1, \ldots, Y_n$[/latex] satisfy a location model [latex]\begin{align*} Y_i = \alpha + \epsilon_i. \end{align*}[/latex] Show that the least-squares point (Theorem 2.1) is an unbiased estimator for [latex]$\alpha$[/latex].
Answer: Remember that the least-squares point is simply the average of the response values. The expectation is [latex]\begin{align*} {\mathbb{E}} {\bar{Y}} &= {\mathbb{E}} (\tfrac{1}{n} \sum_i Y_i)\\ &= \tfrac{1}{n} \sum_i \underbrace{{\mathbb{E}} Y_i}_\alpha\\ &= \alpha. \end{align*}[/latex]

Number: Exercise 4.02
Question: Suppose that [latex]$Y_1, \ldots, Y_n$[/latex] are uncorrelated and all have the same variance [latex]$\sigma^2$[/latex]. What’s the variance of the least-squares point?
Answer: The variance of a constant times a random variable equals the square of that constant times the variance of the random variable (Exercise 3.23). Furthermore, the variance of a sum of uncorrelated random variables equals the sum of the variances (Exercise 3.26). [latex]\begin{align*} \text{var}\, {\bar{Y}} &= \text{var}\, (\tfrac{1}{n} \sum_i Y_i)\\ &= \tfrac{1}{n^2} \sum_i \underbrace{\text{var}\, Y_i}_{\sigma^2}\\ &= \tfrac{1}{n^2} (n\sigma^2)\\ &= \frac{\sigma^2}{n}. \end{align*}[/latex]

Number: Exercise 4.03
Question: Let [latex]$x_1, \ldots, x_n \in {\pmb{\mathbb{R}}}$[/latex] be values of an explanatory variable, and suppose that the response variable [latex]$Y_1, \ldots, Y_n$[/latex] satisfies a simple linear model [latex]\begin{align*} Y_i = \alpha + \beta (x_i - {\bar{x}}) + \epsilon_i. \end{align*}[/latex] Assuming [latex]$x_1, \ldots, x_n$[/latex] are all the same number, show that the coefficients [latex]${\hat{\alpha}}$[/latex] and [latex]${\hat{\beta}}$[/latex] in the least-squares line [latex]$y = {\hat{\alpha}} + {\hat{\beta}} (x - {\bar{x}})$[/latex] are unbiased estimators for [latex]$\alpha$[/latex] and [latex]$\beta$[/latex].
Answer: The least-squares line has [latex]${\hat{\alpha}} = {\bar{Y}}$[/latex], and its expectation is [latex]\begin{align*} {\mathbb{E}} {\bar{Y}} &= {\mathbb{E}} (\tfrac{1}{n} \sum_i Y_i)\\ &= \tfrac{1}{n} \sum_i \underbrace{{\mathbb{E}} Y_i}_{\alpha + \beta (x_i - {\bar{x}})}\\ &= \alpha + \beta \tfrac{1}{n} \underbrace{\sum_i (x_i - {\bar{x}})}_0\\ &= \alpha. \end{align*}[/latex] The other coefficient’s expectation is [latex]\begin{align*} {\mathbb{E}} {\hat{\beta}} &= {\mathbb{E}} \frac{\langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, {\mathbf{Y}} - {\bar{Y}} {\mathbf{1}} \rangle}{\| {\mathbf{x}} - {\bar{x}} {\mathbf{1}} \|^2}\\ &= \frac{\langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, {\mathbb{E}} {\mathbf{Y}} - {\mathbb{E}} {\bar{Y}} {\mathbf{1}} \rangle}{\| {\mathbf{x}} - {\bar{x}} {\mathbf{1}} \|^2}\\ &= \frac{\langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, [\alpha {\mathbf{1}} + \beta ({\mathbf{x}} - {\bar{x}} {\mathbf{1}})] - \alpha {\mathbf{1}} \rangle}{\| {\mathbf{x}} - {\bar{x}} {\mathbf{1}} \|^2}\\ &= \beta \frac{\langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}}, {\mathbf{x}} - {\bar{x}} {\mathbf{1}} \rangle}{\| {\mathbf{x}} - {\bar{x}} {\mathbf{1}} \|^2}\\ &= \beta. \end{align*}[/latex]

Number: Exercise 4.04
Question: Suppose [latex]$Y_1, \ldots, Y_n$[/latex] are uncorrelated and all have the same variance [latex]$\sigma^2$[/latex]. If [latex]$x_1, \ldots, x_n$[/latex] are values of an explanatory variable, what’s the covariance matrix of the coefficients [latex]${\hat{\alpha}}$[/latex] and [latex]${\hat{\beta}}$[/latex] in the least-squares line [latex]$y = {\hat{\alpha}} + {\hat{\beta}} (x - {\bar{x}})$[/latex]?
Answer: The variance of [latex]${\hat{a}}$[/latex] works out to be [latex]$\frac{\sigma^2}{n}$[/latex], exactly as in Exercise 4.2. The variance of [latex]${\hat{b}}$[/latex] is [latex]\begin{align*} \text{var}\, {\hat{b}} &= \text{var}\, \frac{\tfrac{1}{n}\langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}} , {\mathbf{Y}} \rangle}{\sigma_{\mathbf{x}}^2}\\ &= \tfrac{1}{n^2 \sigma_{\mathbf{x}}^4} \text{cov}\, ({\mathbf{x}} - {\bar{x}} {\mathbf{1}})^T {\mathbf{Y}}\\ &= \tfrac{1}{n^2 \sigma_{\mathbf{x}}^4} ({\mathbf{x}} - {\bar{x}} {\mathbf{1}})^T \sigma^2 {\mathbb{I}} ({\mathbf{x}} - {\bar{x}} {\mathbf{1}})\\ &= \frac{\sigma^2}{n^2 \sigma_{\mathbf{x}}^4} \underbrace{\| {\mathbf{x}} - {\bar{x}} {\mathbf{1}} \|^2}_{n \sigma_{\mathbf{x}}^2}\\ &= \frac{\sigma^2}{n \sigma_{\mathbf{x}}^2} \end{align*}[/latex] unless [latex]$\sigma_{\mathbf{x}}^2=0$[/latex] in which case [latex]${\hat{\beta}} \equiv 0$[/latex] has variance [latex]$0$[/latex]. These two variances are the diagonals of the covariance matrix. The off-diagonals are equal to the covariance between [latex]${\hat{a}}$[/latex] and [latex]${\hat{b}}$[/latex]. It will be important to realize that the average of the entries of [latex]${\mathbb{E}} {\mathbf{Y}}$[/latex] is [latex]\begin{align*} \tfrac{1}{n} {\mathbf{1}}^T {\mathbb{E}} {\mathbf{Y}} &= \tfrac{1}{n} {\mathbf{1}}^T [\alpha {\mathbf{1}} - \beta ({\mathbf{x}} - {\bar{x}} {\mathbf{1}})]\\ &= \alpha - \beta ({\bar{x}} - {\bar{x}})\\ &= \alpha. \end{align*}[/latex] Thus [latex]${\bar{Y}} - \alpha$[/latex] can be rewritten as [latex]$\tfrac{1}{n} {\mathbf{1}}^T ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}})$[/latex]. [latex]\begin{align*} {\mathbb{E}} ({\bar{Y}} - \alpha) \left(\tfrac{\tfrac{1}{n}\langle {\mathbf{x}} - {\bar{x}} {\mathbf{1}} , {\mathbf{Y}} \rangle}{\sigma_{\mathbf{x}}^2}\right) &= {\mathbb{E}} \tfrac{1}{n^2 \sigma_{\mathbf{x}}^2} {\mathbf{1}}^T ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) {\mathbf{Y}}^T ({\mathbf{x}} - {\bar{x}} {\mathbf{1}})\\ &= \tfrac{1}{n^2 \sigma_{\mathbf{x}}^2} {\mathbf{1}}^T [\underbrace{{\mathbb{E}} ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) {\mathbf{Y}}^T}_{\sigma^2 \mathbb{I}}] ({\mathbf{x}} - {\bar{x}} {\mathbf{1}})\\ &= \tfrac{\sigma^2}{n \sigma_{\mathbf{x}}^2} \underbrace{\tfrac{1}{n} {\mathbf{1}}^T ({\mathbf{x}} - {\bar{x}} {\mathbf{1}})}_{{\bar{x}} - \bar{x}}\\ &= 0 \end{align*}[/latex]

Number: Exercise 4.05
Question: Suppose that a response variable satisfies a simple linear model of an explanatory variable and that it is predicted by the least-squares line. Which is larger: the sum of squared errors or the sum of squared residuals? Base your answer on the definition of the least-squares line, and explain.
Answer: The sum of squared errors is the sum of squared differences between the response values and the true line, while the sum of squared residuals is the sum of squared differences between the points and the least-squares line. The least-squares line is, by definition, the one with the smallest possible sum of squared differences from the points, so the sum of squared residuals can’t possibly be larger than the sum of squared errors.

Number: Exercise 4.06
Question: The variables picture provides us with a more specific answer to the question posed in Exercise 4.5. Use the Pythagorean identity to quantify the difference between the sum of squared errors and the sum of squared residuals.
Answer: The error vector forms the hypotenuse of a right triangle whose other sides are [latex]${\mathbf{\widehat{Y}}} - {\mathbb{E}} {\mathbf{Y}}$[/latex] and the residual vector [latex]${\mathbf{Y}} - {\mathbf{\widehat{Y}}}$[/latex]. Invoking the Pythagorean identity, [latex]\begin{align*} \| {\boldsymbol{\epsilon}} \|^2 = \| {\mathbf{Y}} - {\mathbf{\widehat{Y}}} \|^2 + \| {\mathbf{\widehat{Y}}} - {\mathbb{E}} {\mathbf{Y}} \|^2. \end{align*}[/latex] The sum of squared errors is larger than the sum of squared residuals by [latex]$\| {\mathbf{\widehat{Y}}} - {\mathbb{E}} {\mathbf{Y}} \|^2$[/latex].

Number: Exercise 4.07
Question: Let [latex]$(x^{(1)}_1, \ldots, x^{(m)}_1), \ldots, (x^{(1)}_n, \ldots, x^{(m)}_n) \in {\pmb{\mathbb{R}}}^m$[/latex] be [latex]$n$[/latex] observations of [latex]$m$[/latex] explanatory variables, and suppose that the response variable [latex]$Y_1, \ldots, Y_n$[/latex] satisfies a multiple linear model [latex]\begin{align*} Y_i = \alpha + \beta_1 (x^{(1)} - {\bar{x}}^{(1)}) + \ldots + \beta_m (x^{(m)} - {\bar{x}}^{(m)}) + \epsilon_i. \end{align*}[/latex] Assuming the explanatory variables’ empirical covariance matrix [latex]$\Sigma$[/latex] is full rank, show that the coefficients [latex]${\hat{\alpha}}, {\hat{\beta}}_1, \ldots, {\hat{\beta}}_m$[/latex] in the least-squares hyperplane [latex]$y = {\hat{\alpha}} + {\hat{\beta}}_1 (x^{(1)} - {\bar{x}}^{(1)}) + \ldots + {\hat{\beta}}_m (x^{(m)} - {\bar{x}}^{(m)})$[/latex] are unbiased estimators for [latex]$\alpha, \beta_1, \ldots, \beta_m$[/latex].
Answer: The least-squares hyperplane has [latex]${\hat{\alpha}} = {\bar{Y}}$[/latex], which can be expressed as [latex]\begin{align*} {\bar{Y}} &= \tfrac{1}{n} \sum_i Y_i\\ &= \tfrac{1}{n} \sum_i [\alpha + \beta_1 (x^{(1)}_i - {\bar{x}}^{(1)}) + \ldots + \beta_m (x^{(m)}_i - {\bar{x}}^{(m)}) + \epsilon_i]\\ &= \alpha + \beta_1 \underbrace{\tfrac{1}{n} \sum_i (x^{(1)}_i - {\bar{x}}^{(1)})}_0 + \ldots + \beta_m \underbrace{\tfrac{1}{n} \sum_i (x^{(m)}_i - {\bar{x}}^{(m)})}_0 + \tfrac{1}{n} \sum_i \epsilon_i\\ &= \alpha + \tfrac{1}{n} \sum_i \epsilon_i. \end{align*}[/latex] Its expectation is [latex]\begin{align*} {\mathbb{E}} {\bar{Y}} &= \alpha + \tfrac{1}{n} \sum_i \underbrace{{\mathbb{E}} \epsilon_i}_0\\ &= \alpha. \end{align*}[/latex] The vector of empirical covariances of [latex]${\mathbf{Y}}$[/latex] with [latex]${\mathbf{x}}^{(1)}, \ldots, {\mathbf{x}}^{(m)}$[/latex] can be expressed as [latex]$\tfrac{1}{n} {\widetilde{\mathbb{X}}}^T {\mathbf{Y}}$[/latex] where [latex]${\widetilde{\mathbb{X}}}$[/latex] is the centered version of the explanatory data matrix. Substituting this representation into the formula from Theorem 2.3, [latex]\begin{align*} {\mathbb{E}} {\boldsymbol{\hat{\beta}}} &= {\mathbb{E}} \Sigma^{-1} \tfrac{1}{n} {\widetilde{\mathbb{X}}}^T {\mathbf{Y}}\\ &= \Sigma^{-1} \tfrac{1}{n} {\widetilde{\mathbb{X}}}^T {\mathbb{E}} {\mathbf{Y}}\\ &= \Sigma^{-1} \tfrac{1}{n} {\widetilde{\mathbb{X}}}^T (\alpha {\mathbf{1}} + {\widetilde{\mathbb{X}}} {\boldsymbol{\beta}})\\ &= \Sigma^{-1} (\tfrac{\alpha}{n} \underbrace{{\widetilde{\mathbb{X}}}^T \mathbf{1}}_0 + \underbrace{\tfrac{1}{n} {\widetilde{\mathbb{X}}}^T \widetilde{\mathbb{X}}}_{\Sigma} {\boldsymbol{\beta}})\\ &= \Sigma^{-1} \Sigma {\boldsymbol{\beta}}\\ &= {\boldsymbol{\beta}}. \end{align*}[/latex]

Number: Exercise 4.08
Question: Suppose [latex]$Y_1, \ldots, Y_n$[/latex] are uncorrelated and all have the same variance [latex]$\sigma^2$[/latex]. With [latex]$(x^{(1)}_1, \ldots, x^{(m)}_1), \ldots, (x^{(1)}_n, \ldots, x^{(m)}_n) \in {\pmb{\mathbb{R}}}^m$[/latex] as [latex]$n$[/latex] observations of [latex]$m$[/latex] explanatory variables, what’s the variance of [latex]${\hat{\alpha}}$[/latex] and the covariance matrix of [latex]${\boldsymbol{\hat{\beta}}} = ({\hat{\beta}}_1, \ldots, {\hat{\beta}}_m)$[/latex] in the least-squares hyperplane [latex]$y = {\hat{\alpha}} + {\hat{\beta}}_1 (x^{(1)} - {\bar{x}}^{(1)}) + \ldots + {\hat{\beta}}_m (x^{(m)} - {\bar{x}}^{(m)})$[/latex]?
Answer: Remember that [latex]${\hat{\alpha}} = {\bar{Y}}$[/latex] has the representation [latex]$\alpha + \tfrac{1}{n}\sum_i \epsilon_i$[/latex]. Its variance once again works out to be [latex]$\frac{\sigma^2}{n}$[/latex]. The covariance matrix of [latex]${\boldsymbol{\hat{\beta}}}$[/latex] is [latex]\begin{align*} \text{cov}\, {\boldsymbol{\hat{\beta}}} &= \text{cov}\, \Sigma^- \tfrac{1}{n} {\widetilde{\mathbb{X}}}^T {\mathbf{Y}}\\ &= \Sigma^- \tfrac{1}{n} {\widetilde{\mathbb{X}}}^T (\sigma^2 {\mathbb{I}}) [\Sigma^- \tfrac{1}{n} {\widetilde{\mathbb{X}}}^T]^T\\ &= \tfrac{\sigma^2}{n} \Sigma^- (\underbrace{\tfrac{1}{n} {\widetilde{\mathbb{X}}}^T \widetilde{\mathbb{X}}}_\Sigma) \Sigma^-\\ &= \tfrac{\sigma^2}{n} \Sigma^- \end{align*}[/latex] by Exercise 1.75.

Number: Exercise 4.09
Question: Suppose [latex]$Y_1, \ldots, Y_n$[/latex] are uncorrelated and all have the same variance [latex]$\sigma^2$[/latex]. With [latex]$(x^{(1)}_1, \ldots, x^{(m)}_1), \ldots, (x^{(1)}_n, \ldots, x^{(m)}_n) \in {\pmb{\mathbb{R}}}^m$[/latex] as [latex]$n$[/latex] observations of [latex]$m$[/latex] explanatory variables, show that [latex]${\hat{\alpha}}$[/latex] is uncorrelated with every [latex]${\hat{\beta}}_1, \ldots, {\hat{\beta}}_m$[/latex] in the least-squares hyperplane.
Answer: Let [latex]${\widetilde{\mathbb{X}}}$[/latex] be the centered version of the explanatory data matrix, and let [latex]$\Sigma_j^-$[/latex] be the [latex]$j$[/latex]th row of the generalized inverse of its empirical covariance matrix (as a column vector). Borrowing tricks from Exercise 4.4, the covariance between [latex]${\hat{\alpha}}$[/latex] and [latex]${\hat{\beta}}_j$[/latex] is [latex]\begin{align*} {\mathbb{E}} ({\bar{Y}} - \alpha) ((\Sigma_j^-)^T \tfrac{1}{n} {\widetilde{\mathbb{X}}}^T {\mathbf{Y}}) &= {\mathbb{E}} \tfrac{1}{n^2} {\mathbf{1}}^T ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) {\mathbf{Y}}^T {\widetilde{\mathbb{X}}} \Sigma_j^-\\ &= \tfrac{1}{n^2} {\mathbf{1}}^T [\underbrace{{\mathbb{E}} ({\mathbf{Y}} - {\mathbb{E}} {\mathbf{Y}}) {\mathbf{Y}}^T}_{\text{cov}\, {\mathbf{Y}} = \sigma^2 \mathbb{I}}] {\widetilde{\mathbb{X}}} \Sigma_j^-\\ &= \tfrac{\sigma^2}{n} \underbrace{\tfrac{1}{n} {\mathbf{1}}^T \widetilde{\mathbb{X}}}_{{\mathbf{0}}^T} \Sigma_j^-\\ &= 0. \end{align*}[/latex]

Number: Exercise 4.10
Question: Let [latex]${\mathbf{x}}_1, \ldots, {\mathbf{x}}_n \in {\pmb{\mathbb{R}}}^m$[/latex] be [latex]$n$[/latex] observations of [latex]$m$[/latex] explanatory variables, and suppose that the response variable [latex]$Y_1, \ldots, Y_n$[/latex] satisfies a linear model [latex]\begin{align*} Y_i = \gamma_1 g_1({\mathbf{x}}_i) + \ldots + \gamma_d g_d({\mathbf{x}}_i) + \epsilon_i. \end{align*}[/latex] Assuming the columns of the design matrix are linearly independent, show that the coefficients [latex]${\hat{\gamma}}_1, \ldots, {\hat{\gamma}}_d$[/latex] in the least-squares linear fit [latex]$y = {\hat{\gamma}}_1 g_1({\mathbf{x}}) + \ldots + {\hat{\gamma}}_d g_d({\mathbf{x}})$[/latex] are unbiased estimators for [latex]$\gamma_1, \ldots, \gamma_d$[/latex].
Answer: Let [latex]${\mathbb{M}}$[/latex] represent the design matrix [latex]\begin{align*} {\mathbb{M}} := \begin{bmatrix} g_1({\mathbf{x}}_1) & \cdots & g_d({\mathbf{x}}_1) \\ \vdots & \ddots & \vdots \\ g_1({\mathbf{x}}_n) & \cdots & g_d({\mathbf{x}}_n) \end{bmatrix}. \end{align*}[/latex] The expectation of [latex]${\mathbf{Y}} = {\mathbb{M}} {\boldsymbol{\gamma}} + {\boldsymbol{\epsilon}}$[/latex] is [latex]${\mathbb{M}} {\boldsymbol{\gamma}}$[/latex]. Using the formula for the least-squares coefficients provided in Theorem 2.4, [latex]\begin{align*} {\mathbb{E}} {\boldsymbol{\hat{\gamma}}} &= {\mathbb{E}} ({\mathbb{M}}^T {\mathbb{M}})^{-1} {\mathbb{M}}^T {\mathbf{Y}}\\ &= ({\mathbb{M}}^T {\mathbb{M}})^{-1} {\mathbb{M}}^T \underbrace{{\mathbb{E}} \mathbf{Y}}_{{\mathbb{M}} \boldsymbol{\gamma}}\\ &= ({\mathbb{M}}^T {\mathbb{M}})^{-1} ({\mathbb{M}}^T {\mathbb{M}}) {\boldsymbol{\gamma}}\\ &= {\boldsymbol{\gamma}}. \end{align*}[/latex] (We know that [latex]${\mathbb{M}}^T {\mathbb{M}}$[/latex] is invertible because the columns of [latex]${\mathbb{M}}$[/latex] are assumed to be linearly independent – see Exercise 1.63.)

Number: Exercise 4.11
Question: Suppose [latex]$Y_1, \ldots, Y_n$[/latex] are uncorrelated and all have the same variance [latex]$\sigma^2$[/latex]. With [latex]${\mathbf{x}}_1, \ldots, {\mathbf{x}}_n \in {\pmb{\mathbb{R}}}^m$[/latex] as [latex]$n$[/latex] observations of [latex]$m$[/latex] explanatory variables, what’s the covariance matrix of [latex]${\boldsymbol{\hat{\gamma}}} = ({\hat{\gamma}}_1, \ldots, {\hat{\gamma}}_d)$[/latex] in the least-squares linear fit [latex]$y = {\hat{\gamma}}_1 g_1({\mathbf{x}}) + \ldots + {\hat{\gamma}}_d g_d({\mathbf{x}})$[/latex]?
Answer: The covariance matrix of [latex]${\boldsymbol{\hat{\gamma}}}$[/latex] is [latex]\begin{align*} \text{cov}\, {\boldsymbol{\hat{\gamma}}} &= \text{cov}\, ({\mathbb{M}}^T {\mathbb{M}})^- {\mathbb{M}}^T {\mathbf{Y}}\\ &= ({\mathbb{M}}^T {\mathbb{M}})^- {\mathbb{M}}^T (\sigma^2 {\mathbb{I}}) [({\mathbb{M}}^T {\mathbb{M}})^- {\mathbb{M}}^T]^T\\ &= \sigma^2 ({\mathbb{M}}^T {\mathbb{M}})^- {\mathbb{M}}^T {\mathbb{M}} ({\mathbb{M}}^T {\mathbb{M}})^-\\ &= \sigma^2 ({\mathbb{M}}^T {\mathbb{M}})^- \end{align*}[/latex] by Exercise 1.75.

Number: Exercise 4.12
Question: Assume the columns of [latex]${\mathbb{M}}$[/latex] are linearly independent. Suppose [latex]${\mathbf{Y}} = {\mathbb{M}} {\boldsymbol{\gamma}} + {\boldsymbol{\epsilon}}$[/latex] where [latex]${\boldsymbol{\epsilon}}$[/latex] is a random vector with mean [latex]${\mathbf{0}}$[/latex] and covariance [latex]$\sigma^2 {\mathbb{I}}$[/latex]. Let [latex]${\boldsymbol{\hat{\gamma}}} := ({\mathbb{M}}^T{\mathbb{M}})^{-1} {\mathbb{M}}^T {\mathbf{Y}}$[/latex] denote the least-squares estimator for the coefficients. Suppose an alternative estimator [latex]${\boldsymbol{\check{\gamma}}} := {\mathbb{L}} {\mathbf{Y}}$[/latex] is also unbiased for [latex]${\boldsymbol{\gamma}}$[/latex]. Use the Gauss-Markov theorem to show that [latex]${\mathbb{E}} ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbb{L}} ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}}) \leq {\mathbb{E}} ({\boldsymbol{\check{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbb{L}} ({\boldsymbol{\check{\gamma}}} - {\boldsymbol{\gamma}})$[/latex] for every positive semi-definite matrix [latex]${\mathbb{L}}$[/latex].
Answer: First, consider the claim of the Gauss-Markov theorem: the variance of [latex]${\boldsymbol{\hat{\gamma}}}^T {\mathbf{v}}$[/latex] is no greater than the variance of [latex]${\boldsymbol{\check{\gamma}}}^T {\mathbf{v}}$[/latex]. An alternative expression for the squared deviation of [latex]${\boldsymbol{\hat{\gamma}}}^T {\mathbf{v}}$[/latex] from its mean is [latex]\begin{align*} ({\boldsymbol{\hat{\gamma}}}^T {\mathbf{v}} - {\mathbb{E}} {\boldsymbol{\hat{\gamma}}}^T {\mathbf{v}})^2 &= ({\boldsymbol{\hat{\gamma}}}^T {\mathbf{v}} - {\boldsymbol{\gamma}}^T {\mathbf{v}})^2\\ &= [({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbf{v}}] [({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbf{v}}]\\ &= ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbf{v}} {\mathbf{v}}^T ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}}), \end{align*}[/latex] and likewise for [latex]${\boldsymbol{\check{\gamma}}}$[/latex]. The expected squared deviation is the variance, so Gauss-Markov tells us that [latex]${\mathbb{E}} ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbf{v}} {\mathbf{v}}^T ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})$[/latex] is no greater than [latex]$({\boldsymbol{\check{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbf{v}} {\mathbf{v}}^T ({\boldsymbol{\check{\gamma}}} - {\boldsymbol{\gamma}})$[/latex] for every [latex]${\mathbf{v}}$[/latex].
With a spectral decomposition for [latex]${\mathbb{L}}$[/latex], [latex]\begin{align*} ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbb{L}} ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}}) &= ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})^T (\lambda_1 {\mathbf{q}}_1 {\mathbf{q}}_1^T + \ldots + \lambda_d {\mathbf{q}}_d {\mathbf{q}}_d^T) ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})\\ &= \lambda_1 ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbf{q}}_1 {\mathbf{q}}_1^T ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}}) + \ldots + \lambda_d ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}})^T {\mathbf{q}}_d {\mathbf{q}}_d^T) ({\boldsymbol{\hat{\gamma}}} - {\boldsymbol{\gamma}}). \end{align*}[/latex] Each eigenvalue is non-negative, so each term is no greater than the corresponding expression with [latex]${\boldsymbol{\check{\gamma}}}$[/latex] in place of [latex]${\boldsymbol{\hat{\gamma}}}$[/latex].

Number: Exercise 4.13
Question: Suppose [latex]${\mathbf{Y}} = {\mathbb{M}} {\boldsymbol{\gamma}} + {\boldsymbol{\epsilon}}$[/latex] with [latex]${\mathbb{E}} {\boldsymbol{\epsilon}} = 0$[/latex], and let [latex]${\boldsymbol{\hat{\gamma}}}$[/latex] be coefficients of least-squares linear regression estimators for the correctly specified model. If a new explanatory observation [latex]${\mathbf{v}}_{n+1}$[/latex] is in the row space of [latex]${\mathbb{M}}$[/latex], and [latex]$Y_{n+1} = {\mathbf{v}}_{n+1}^T {\boldsymbol{\gamma}} + \epsilon_{n+1}$[/latex] with [latex]${\mathbb{E}} \epsilon_{n+1} = 0$[/latex], show that the expectation of the predictor [latex]${\widehat{Y}}_{n+1} = {\mathbf{v}}_{n+1}^T {\boldsymbol{\hat{\gamma}}}$[/latex] equals the expectation of [latex]$Y_{n+1}$[/latex] regardless of whether or not [latex]${\mathbb{M}}$[/latex] has full rank.
Answer: Let [latex]${\mathbf{v}}_{n+1}^T = {\mathbf{w}}^T {\mathbb{M}}$[/latex]. The expectation of the new response value can be represented as the [latex]${\mathbf{w}}$[/latex] linear combination of the expectations of previous response values: [latex]\begin{align*} {\mathbb{E}} Y_{n+1} &= {\mathbf{v}}_{n+1}^T {\boldsymbol{\gamma}}\\ &= {\mathbf{w}}^T {\mathbb{M}} {\boldsymbol{\gamma}}\\ &= {\mathbf{w}}^T {\mathbb{E}} {\mathbf{Y}}. \end{align*}[/latex] The predictor can be written as the same linear combination of the previous predicted values: [latex]\begin{align*} {\widehat{Y}}_{n+1} &= {\mathbf{v}}_{n+1}^T {\boldsymbol{\hat{\gamma}}}\\ &= {\mathbf{w}}^T {\mathbb{M}} {\boldsymbol{\hat{\gamma}}}\\ &= {\mathbf{w}}^T {\mathbf{\widehat{Y}}}. \end{align*}[/latex] Because [latex]${\mathbf{\widehat{Y}}}$[/latex] is unbiased for [latex]${\mathbb{E}} {\mathbf{Y}}$[/latex], we have [latex]\begin{align*} {\mathbb{E}} {\widehat{Y}}_{n+1} &= {\mathbf{w}}^T {\mathbb{E}} {\mathbf{\widehat{Y}}}\\ &= {\mathbf{w}}^T {\mathbb{E}} {\mathbf{Y}} \end{align*}[/latex] which is the same expectation we found for [latex]${\mathbb{E}} Y_{n+1}$[/latex].

Number: Exercise 4.14
Question: Let [latex]${\mathbf{x}} = (x_1, \ldots, x_n)$[/latex] and [latex]${\mathbf{z}} = (z_1, \ldots, z_n)$[/latex] be explanatory variables such that [latex]${\mathbf{x}} = a {\mathbf{z}}$[/latex] for some [latex]$c \in {\pmb{\mathbb{R}}}$[/latex]. Assume [latex]${\mathbf{Y}} = (Y_1, \ldots, Y_n)$[/latex] satisfy [latex]${\mathbf{Y}} = b_1 {\mathbf{x}} + b_2 {\mathbf{z}} + {\boldsymbol{\epsilon}}$[/latex] for some [latex]$b_1, b_2 \in {\pmb{\mathbb{R}}}$[/latex] and [latex]${\mathbb{E}} {\boldsymbol{\epsilon}} = {\mathbf{0}}$[/latex]. Argue that the derived parameter [latex]$c := a b_1 + b_2$[/latex] can be estimated.
Answer: The expectation of the response variable is [latex]\begin{align*} {\mathbb{E}} {\mathbf{Y}} &= b_1 {\mathbf{x}} + b_2 {\mathbf{z}}\\ &= b_1 a {\mathbf{z}} + b_2 {\mathbf{z}}\\ &= (a b_1 + b_2) {\mathbf{z}}\\ &= c {\mathbf{z}}. \end{align*}[/latex] Every possible value of [latex]$a$[/latex] implies a different expectation for the response variable, so it can be estimated. In fact the orthogonal projection’s coefficient [latex]$\frac{\langle {\mathbf{Y}}, {\mathbf{z}} \rangle}{\| {\mathbf{z}} \|^2}$[/latex] is unbiased based on Exercise 4.10.

Number: Exercise 4.15
Question: Suppose [latex]${\mathbf{Y}} = {\mathbb{M}} {\boldsymbol{\beta}} + {\boldsymbol{\epsilon}}$[/latex] with [latex]$\text{cov}\, {\boldsymbol{\epsilon}} = \sigma^2 {\mathbb{I}}_n$[/latex], and let [latex]${\mathbf{\widehat{Y}}}$[/latex] be the orthogonal projection of [latex]${\mathbf{Y}}$[/latex] onto [latex]$C({\mathbb{M}})$[/latex]. Find [latex]${\mathbb{E}} \| {\mathbf{Y}} - {\mathbf{\widehat{Y}}} \|^2$[/latex], the expected sum of squared residuals.
Answer: We’ll let [latex]${\mathbb{H}}$[/latex] be the orthogonal projection matrix onto [latex]${\mathbb{M}}$[/latex], and use Exercise 3.35 along with Exercises 1.67 and 1.68. [latex]\begin{align*} {\mathbb{E}} \| {\mathbf{Y}} - {\mathbf{\widehat{Y}}} \|^2 &= \| ({\mathbb{I}} - {\mathbb{H}}) {\boldsymbol{\epsilon}} \|^2\\ &= \text{tr} \, [({\mathbb{I}} - {\mathbb{H}}) \sigma^2 {\mathbb{I}}]\\ &= \sigma^2 (n - \text{rank} \, {\mathbb{M}}) \end{align*}[/latex]

Number: Exercise 4.16
Question: Suppose the predictor [latex]${\widehat{Y}}_{n+1}$[/latex] is a function of [latex]$Y_1, \ldots, Y_n$[/latex] which are independent of [latex]$Y_{n+1}$[/latex]. Show that [latex]\begin{align*} {\mathbb{E}} (Y_{n+1} - {\widehat{Y}}_{n+1})^2 &= \text{var}\, \, Y_{n+1} + {\mathbb{E}} ({\widehat{Y}}_{n+1} - {\mathbb{E}} Y_{n+1})^2. \end{align*}[/latex]
Answer: [latex]\begin{align*} {\mathbb{E}} (Y_{n+1} - {\widehat{Y}}_{n+1})^2 &= {\mathbb{E}} [(Y_{n+1} - {\mathbb{E}} Y_{n+1}) - ({\widehat{Y}}_{n+1} - {\mathbb{E}} Y_{n+1})]^2\\ &= {\mathbb{E}} (Y_{n+1} - {\mathbb{E}} Y_{n+1})^2 - 2 {\mathbb{E}} (Y_{n+1} - {\mathbb{E}} Y_{n+1})({\widehat{Y}}_{n+1} - {\mathbb{E}} Y_{n+1}) + {\mathbb{E}} ({\widehat{Y}}_{n+1} - {\mathbb{E}} Y_{n+1})^2\\ &= \text{var}\, \, Y_{n+1} - 2 \underbrace{{\mathbb{E}} (Y_{n+1} - {\mathbb{E}} Y_{n+1})}_0 {\mathbb{E}} ({\widehat{Y}}_{n+1} - {\mathbb{E}} Y_{n+1}) + {\mathbb{E}} ({\widehat{Y}}_{n+1} - {\mathbb{E}} Y_{n+1})^2\\ &= \text{var}\, \, Y_{n+1} + {\mathbb{E}} ({\widehat{Y}}_{n+1} - {\mathbb{E}} Y_{n+1})^2 \end{align*}[/latex]

Number: Exercise 4.17
Question: Suppose [latex]$Y_1, \ldots, Y_n$[/latex] are uncorrelated and all have the same variance [latex]$\sigma^2$[/latex]. Let [latex]${\mathbf{x}}_1, \ldots, {\mathbf{x}}_n \in {\pmb{\mathbb{R}}}^m$[/latex] be [latex]$n$[/latex] observations of [latex]$m$[/latex] explanatory variables, and assume their empirical covariance matrix [latex]$\Sigma$[/latex] has full rank. Let [latex]${\boldsymbol{\hat{\beta}}} = ({\hat{\beta}}_1, \ldots, {\hat{\beta}}_m)$[/latex] be the coefficients of the explanatory variables in the least-squares hyperplane [latex]$y = {\hat{\alpha}} + {\hat{\beta}}_1 (x^{(1)} - {\bar{x}}^{(1)}) + \ldots + {\hat{\beta}}_m (x^{(m)} - {\bar{x}}^{(m)})$[/latex]. Find [latex]${\mathbb{E}} \| {\boldsymbol{\hat{\beta}}} - {\mathbb{E}} {\boldsymbol{\hat{\beta}}} \|^2$[/latex] in terms of [latex]$\sigma^2$[/latex], [latex]$n$[/latex], and the eigenvalues of [latex]$\Sigma$[/latex].
Answer: The “variance” of any random vector is the trace of its covariance matrix (Exercise 3.25). [latex]\begin{align*} {\mathbb{E}} \| {\boldsymbol{\hat{\beta}}} - {\mathbb{E}} {\boldsymbol{\hat{\beta}}} \|^2 &= \text{tr} \, \text{cov}\, {\boldsymbol{\hat{\beta}}}\\ &= \tfrac{\sigma^2}{n} \text{tr} \, \Sigma^{-1}\\ &= \tfrac{\sigma^2}{n} (\lambda_1^{-1} + \ldots + \lambda_m^{-1}) \end{align*}[/latex] where [latex]$\lambda_1, \ldots, \lambda_m$[/latex] are the eigenvalues of [latex]$\Sigma$[/latex].

Number: Exercise 4.18
Question: Based on Exercise 4.12, the Gauss-Markov theorem implies that the least-squares coefficient vector has the smallest possible expected squared estimation error among all random vectors that are both linear functions of the response and unbiased for its expectation. However, Equation 4.1 identified [latex]$a < 1$[/latex] such that [latex]$a$[/latex] times the least-squares coefficients of the explanatory variables has smaller expected squared estimation error than the least-squares coefficient vector do; explain why this doesn’t contradict the Gauss-Markov theorem.
Answer: Let’s check the conditions of the Gauss-Markov theorem. It applies to linear functions of the response [latex]${\mathbf{Y}}$[/latex] that are unbiased for [latex]${\mathbb{E}} {\mathbf{Y}}$[/latex]. Because [latex]${\boldsymbol{\hat{\beta}}}$[/latex] is linear in [latex]${\mathbf{Y}}$[/latex], so is [latex]$a {\boldsymbol{\hat{\beta}}}$[/latex]. However, it’s biased; its expectation is [latex]$a {\boldsymbol{\beta}} \neq {\boldsymbol{\beta}}$[/latex], so Gauss-Markov doesn’t apply.

Number: Exercise 5.01
Question: Find the probability density function for a standard Normal random vector on [latex]${\pmb{\mathbb{R}}}^n$[/latex].
Answer: Let [latex]${\mathbf{Z}}$[/latex] be an [latex]${\pmb{\mathbb{R}}}^n$[/latex]-valued standard Normal random vector. By independence, its pdf equals the product of the individual pdfs of its coordinates [latex]$(Z_1, \ldots, Z_n)$[/latex]. [latex]\begin{align*} f({\mathbf{z}}) &= \prod_i \tfrac{1}{\sqrt{2\pi}} e^{-z_i^2/2}\\ &= \tfrac{1}{(2\pi)^{n/2}} e^{-(z_1^2 + \ldots + z_n^2)/2}\\ &= \tfrac{1}{(2\pi)^{n/2}} e^{-\| {\mathbf{z}} \|^2/2} \end{align*}[/latex]

Number: Exercise 5.02
Question: For standard Normal random vectors [latex]${\mathbf{Z}}_1$[/latex] and [latex]${\mathbf{Z}}_2$[/latex], suppose [latex]${\mathbb{M}}_1 {\mathbf{Z}}_1$[/latex] and [latex]${\mathbb{M}}_2 {\mathbf{Z}}_2$[/latex] have the same covariance. Show that they have the same distribution.
Answer: From Exercise 3.23, we calculate the covariances to be [latex]${\mathbb{M}}_1 {\mathbb{M}}_1^T$[/latex] and [latex]${\mathbb{M}}_2 {\mathbb{M}}_2^T$[/latex].
Suppose [latex]${\mathbb{M}}_1$[/latex] has the singular value decomposition [latex]${\mathbb{U}} {\mathbb{S}} {\mathbb{V}}_1^T$[/latex]. Then the spectral decomposition of [latex]${\mathbb{M}}_1 {\mathbb{M}}_1^T$[/latex] is [latex]${\mathbb{U}} {\mathbb{S}} {\mathbb{U}}^T$[/latex]. By the assumption that the covariances are equal, we see that [latex]${\mathbb{M}}_2 {\mathbb{M}}_2^T$[/latex] must also be equal to [latex]${\mathbb{U}} {\mathbb{S}} {\mathbb{U}}^T$[/latex]. Thus, a singular value decomposition of [latex]${\mathbb{M}}_2$[/latex] has the same matrix [latex]${\mathbb{U}}$[/latex] on the left and the same matrix of singular values; we’ll write [latex]${\mathbb{M}}_2 = {\mathbb{U}} {\mathbb{S}} {\mathbb{V}}_2^T$[/latex].
We need to compare the distributions of [latex]${\mathbb{U}} {\mathbb{S}} {\mathbb{V}}_1^T {\mathbf{Z}}_1$[/latex] and [latex]${\mathbb{U}} {\mathbb{S}} {\mathbb{V}}_2^T {\mathbf{Z}}_2$[/latex]. The entries of [latex]${\mathbb{V}}_1^T {\mathbf{Z}}_1$[/latex] are the coordinates of [latex]${\mathbf{Z}}_1$[/latex] with respect to the orthonormal columns of [latex]${\mathbb{V}}$[/latex], so they’re iid standard Normal according to our discussion in Section 5.1. Likewise, the entries of [latex]${\mathbb{V}}_2^T {\mathbf{Z}}_2$[/latex] are standard Normal, so we can conclude that the two random vectors in question have the same distribution.

Number: Exercise 5.03
Question: Show that if [latex]${\mathbf{X}}$[/latex] is a Normal random vector, then so is [latex]${\mathbb{M}} {\mathbf{X}} + {\mathbf{v}}$[/latex] where [latex]${\mathbb{M}}$[/latex] is a real matrix and [latex]${\mathbf{v}}$[/latex] is a vector.
Answer: With [latex]${\boldsymbol{\mu}}$[/latex] and [latex]${\mathbb{C}}$[/latex] representing the expectation and covariance of [latex]${\mathbf{X}}$[/latex], the transformed random vector is [latex]\begin{align*} {\mathbb{M}} {\mathbf{X}} + {\mathbf{v}} &= {\mathbb{M}} ({\mathbb{C}}^{1/2} {\mathbf{Z}} + {\boldsymbol{\mu}}) + {\mathbf{v}}\\ &= [{\mathbb{M}} {\mathbb{C}}^{1/2}] {\mathbf{Z}} + [{\mathbb{M}} {\boldsymbol{\mu}} + {\mathbf{v}}] \end{align*}[/latex] with [latex]${\mathbf{Z}}$[/latex] standard Normal. This fits the definition of a Normal random vector.

Number: Exercise 5.04
Question: Show that if [latex]${\mathbf{X}}_1$[/latex] and [latex]${\mathbf{X}}_2$[/latex] are Normal random vectors, then so is [latex]${\mathbf{X}}_1 + {\mathbf{X}}_2$[/latex].
Answer: Let [latex]${\mathbf{X}}_1 \sim N({\boldsymbol{\mu}}_1, {\mathbb{C}}_1)$[/latex] and [latex]${\mathbf{X}}_2 \sim N({\boldsymbol{\mu}}_2, {\mathbb{C}}_2)$[/latex]. With standard Normal [latex]${\mathbf{Z}}_1$[/latex] and [latex]${\mathbf{Z}}_2$[/latex], we can represent the sum as [latex]\begin{align*} {\mathbf{X}}_1 + {\mathbf{X}}_2 &= ({\mathbb{C}}_1^{1/2} {\mathbf{Z}}_1 + {\boldsymbol{\mu}}_1) + ({\mathbb{C}}_2^{1/2} {\mathbf{Z}}_2 + {\boldsymbol{\mu}}_2)\\ &= \begin{bmatrix} {\mathbb{C}}_1^{1/2} & {\mathbb{C}}_2^{1/2} \end{bmatrix} \begin{bmatrix} {\mathbf{Z}}_1 \\ {\mathbf{Z}}_2 \end{bmatrix} + [{\boldsymbol{\mu}}_1 + {\boldsymbol{\mu}}_2]. \end{align*}[/latex] We’re almost finished, but consider carefully the vector [latex]$\begin{bmatrix} {\mathbf{Z}}_1 \\ {\mathbf{Z}}_2 \end{bmatrix}$[/latex] that has the entries of [latex]${\mathbf{Z}}_1$[/latex] stacked on top of the entries of [latex]${\mathbf{Z}}_2$[/latex]. We can’t assume that the entries of [latex]${\mathbf{Z}}_1$[/latex] are independent of the entries of [latex]${\mathbf{Z}}_2$[/latex], so the stacked vector isn’t necessarily standard Normal. However, with [latex]${\mathbb{C}}$[/latex] representing the covariance matrix of [latex]$\begin{bmatrix} {\mathbf{Z}}_1 \\ {\mathbf{Z}}_2 \end{bmatrix}$[/latex] and with [latex]${\mathbf{Z}}$[/latex] a standard Normal random vector of the same size as [latex]$\begin{bmatrix} {\mathbf{Z}}_1 \\ {\mathbf{Z}}_2 \end{bmatrix}$[/latex], then we can rewrite the expression as [latex]\begin{align*} \begin{bmatrix} {\mathbb{C}}_1^{1/2} & {\mathbb{C}}_2^{1/2} \end{bmatrix} {\mathbb{C}}^{1/2} {\mathbf{Z}} + [{\boldsymbol{\mu}}_1 + {\boldsymbol{\mu}}_2] \end{align*}[/latex] which fits the definition of a Normal random vector.

Number: Exercise 5.05
Question: If two random variables are multivariate Normal and are uncorrelated with each other, then they are independent; one can verify that their joint density factors into a product of their marginal densities. However, without multivariate Normality, uncorrelated doesn’t necessarily imply independent. Construct a pair of Normal random variables that are uncorrelated but not independent.
Answer: Let [latex]$Z \sim N(0, 1)$[/latex]. Independently of [latex]$Z$[/latex], let [latex]$B$[/latex] take values [latex]$-1$[/latex] and [latex]$1$[/latex] each with probability. Finally, define [latex]$Y := BZ$[/latex]. By inspecting the cdf of [latex]$Y$[/latex], [latex]\begin{align*} {\mathbb{P}}(Y \leq t) &= {\mathbb{P}}(B=1 \cap Z \leq t) + {\mathbb{P}}(B=-1 \cap Z \geq t)\\ &= {\mathbb{P}}(B=1) {\mathbb{P}}(Z \leq t) + {\mathbb{P}}(B=-1) {\mathbb{P}}(Z \geq -t)\\ &= {\mathbb{P}}(B=1) {\mathbb{P}}(Z \leq t) + {\mathbb{P}}(B=-1) \underbrace{{\mathbb{P}}(Z \geq -t)}_{{\mathbb{P}}(Z \leq t)}\\ &= \underbrace{[{\mathbb{P}}(B=1) + {\mathbb{P}}(B=-1)]}_1 {\mathbb{P}}(Z \leq t) \end{align*}[/latex] we see that it is also standard Normal as it has the same cdf as [latex]$Z$[/latex]. If you learn that [latex]$Z=z$[/latex], you know that [latex]$Y$[/latex] is either [latex]$z$[/latex] or [latex]$-z$[/latex], so [latex]$Z$[/latex] and [latex]$Y$[/latex] clearly aren’t independent. However, their correlation is [latex]\begin{align*} {\mathbb{E}} ZY &= {\mathbb{E}} Z(BZ)\\ &= (\underbrace{{\mathbb{E}} B}_0) ({\mathbb{E}} Z^2)\\ &= 0. \end{align*}[/latex]

Number: Exercise 5.06
Question: Find the expectation of [latex]$W \sim \chi^2_k$[/latex].
Answer: [latex]$W$[/latex] can be represented as the squared norm of a standard Normal random vector. Its expectation is the same as the expected squared norm of any standardized random vector [latex]${\mathbf{Z}}$[/latex] on [latex]${\pmb{\mathbb{R}}}^k$[/latex]: [latex]\begin{align*} {\mathbb{E}} \| {\mathbf{Z}} \|^2 &= {\mathbb{E}} (Z_1^2 + \ldots + Z_k^2)\\ &= {\mathbb{E}} Z_1^2 + \ldots + {\mathbb{E}} Z_k^2\\ &= \underbrace{\text{var}\, Z_1}_1 + \ldots + \underbrace{\text{var}\, Z_k}_1\\ &= k. \end{align*}[/latex]

Number: Exercise 5.07
Question: If [latex]${\mathbf{Y}} \sim N({\boldsymbol{\mu}}, {\mathbb{C}})$[/latex] is an [latex]${\pmb{\mathbb{R}}}^n$[/latex]-valued random vector, what’s the distribution of the squared Mahalanobis distance of [latex]${\mathbf{Y}}$[/latex] from its own distribution?
Answer: Allow for degenerate distributions by using the approach described at the end of Section 3.5. Let [latex]${\mathbf{Z}} := {\mathbb{C}}^{-1/2}({\mathbf{Y}} - {\boldsymbol{\mu}}) \sim N({\mathbf{0}}, {\mathbb{I}})$[/latex] represent the standardized version in [latex]${\pmb{\mathbb{R}}}^{\text{rank} \, \mathbb{C}}$[/latex]. The squared Mahalanobis distance from [latex]${\mathbf{Y}}$[/latex] to [latex]$N({\boldsymbol{\mu}}, {\mathbb{C}})$[/latex] is [latex]\begin{align*} \| {\mathbb{C}}^{-1/2} [{\mathbf{Y}} - {\boldsymbol{\mu}}] \|^2 &= \| {\mathbb{C}}^{-1/2}[({\mathbb{C}}^{1/2} {\mathbf{Z}} + {\boldsymbol{\mu}}) - {\boldsymbol{\mu}}] \|^2\\ &= \| {\mathbf{Z}} \|^2\\ &\sim \chi^2_{\text{rank} \, \mathbb{C}}. \end{align*}[/latex]

Number: Exercise 5.08
Question: Let [latex]${\mathbf{Z}}$[/latex] be an [latex]${\pmb{\mathbb{R}}}^n$[/latex]-valued random vector with the standard Normal distribution, and let [latex]${\mathbb{H}}$[/latex] be an orthogonal projection matrix. Find the distribution of [latex]$\| {\mathbb{H}} {\mathbf{Z}} \|^2$[/latex].
Answer: Let [latex]${\mathbf{u}_1, \ldots, \mathbf{u}_n}$[/latex] be an orthonormal basis with [latex]${\mathbf{u}}_1, \ldots, {\mathbf{u}}_{\text{rank} \, \mathbb{H}}$[/latex] spanning the space that [latex]${\mathbb{H}}$[/latex] projects onto. Because the orthogonal projection is [latex]\begin{align*} {\mathbb{H}} {\mathbf{Z}} = \langle {\mathbf{Z}}, {\mathbf{u}}_1 \rangle {\mathbf{u}}_1 + \ldots + \langle {\mathbf{Z}}, {\mathbf{u}}_{\text{rank} \, \mathbb{H}} \rangle {\mathbf{u}}_{\text{rank} \, \mathbb{H}} \end{align*}[/latex] its squared length is the sum of its squared coordinates [latex]\begin{align*} \| {\mathbb{H}} {\mathbf{Z}} \|^2 = \langle {\mathbf{Z}}, {\mathbf{u}}_1 \rangle^2 + \ldots + \langle {\mathbf{Z}}, {\mathbf{u}}_{\text{rank} \, \mathbb{H}} \rangle^2. \end{align*}[/latex] These coordinates are independent standard Normal random variables, according to the discussion in Section 5.1, so their sum of squares has distribution [latex]$\chi^2_{\text{rank} \, \mathbb{H}}$[/latex].

Number: Exercise 5.09
Question: Let [latex]${\boldsymbol{\epsilon}} \sim N({\mathbf{0}}, \sigma^2 {\mathbb{I}})$[/latex]. If [latex]${\mathbb{H}}$[/latex] is an orthogonal projection matrix and [latex]${\mathbf{u}}$[/latex] is a unit vector orthogonal to [latex]$C({\mathbb{H}})$[/latex], find the distribution of [latex]\begin{align*} \frac{\langle {\boldsymbol{\epsilon}}, {\mathbf{u}} \rangle}{\| {\mathbb{H}} {\boldsymbol{\epsilon}} \| / \sqrt{\text{rank} \, \mathbb{H}}}. \end{align*}[/latex]
Answer: First, we’ll divide the numerator and the denominator by [latex]$\sigma$[/latex] to connect this ratio to the standard Normal random vector [latex]${\boldsymbol{\epsilon}}/\sigma$[/latex]. [latex]\begin{align*} \frac{\langle {\boldsymbol{\epsilon}}, {\mathbf{u}} \rangle}{\| {\mathbb{H}} {\boldsymbol{\epsilon}} \| / \sqrt{\text{rank} \, \mathbb{H}}} = \frac{\langle ({\boldsymbol{\epsilon}}/\sigma), {\mathbf{u}} \rangle}{\| {\mathbb{H}} ({\boldsymbol{\epsilon}}/\sigma) \| / \sqrt{\text{rank} \, \mathbb{H}}} \end{align*}[/latex] Let [latex]${\mathbf{u}_1, \ldots, \mathbf{u}_n}$[/latex] be an orthonormal basis with [latex]${\mathbf{u}}_1, \ldots, {\mathbf{u}}_{\text{rank} \, \mathbb{H}}$[/latex] spanning [latex]$C({\mathbb{H}})$[/latex] and [latex]${\mathbf{u}}_{\text{rank} \, {\mathbb{H}} + 1}$[/latex] equal to [latex]${\mathbf{u}}$[/latex]. The numerator is simply the coordinate of [latex]${\boldsymbol{\epsilon}}/\sigma$[/latex] with respect to [latex]${\mathbf{u}}$[/latex], so it’s a standard Normal random variable. From Exercise 5.8, [latex]$\| {\mathbb{H}} ({\boldsymbol{\epsilon}}/\sigma) \|^2 \sim \chi^2_{\text{rank} \, \mathbb{H}}$[/latex]. Because the numerator and the denominator are functions of distinct coordinates, they’re independent of each other, so the random variable has the [latex]$t_{\text{rank} \, \mathbb{H}}$[/latex] distribution.

Number: Exercise 5.10
Question: Let [latex]${\boldsymbol{\epsilon}} \sim N({\mathbf{0}}, \sigma^2 {\mathbb{I}})$[/latex]. If [latex]${\mathbb{H}}$[/latex] is an orthogonal projection matrix and [latex]${\mathbf{u}}$[/latex] is a unit vector orthogonal to [latex]$C({\mathbb{H}})$[/latex], and [latex]$a \in {\pmb{\mathbb{R}}}$[/latex], find the distribution of [latex]\begin{align*} \frac{a + \langle {\boldsymbol{\epsilon}}, {\mathbf{u}} \rangle}{\| {\mathbb{H}} {\boldsymbol{\epsilon}} \| / \sqrt{\text{rank} \, \mathbb{H}}}. \end{align*}[/latex]
Answer: First, we’ll divide the numerator and the denominator by [latex]$\sigma$[/latex]. [latex]\begin{align*} \frac{a + \langle {\boldsymbol{\epsilon}}, {\mathbf{u}} \rangle}{\| {\mathbb{H}} {\boldsymbol{\epsilon}} \| / \sqrt{\text{rank} \, \mathbb{H}}} = \frac{ a/\sigma + \langle ({\boldsymbol{\epsilon}}/\sigma), {\mathbf{u}} \rangle}{\| {\mathbb{H}} ({\boldsymbol{\epsilon}}/\sigma) \| / \sqrt{\text{rank} \, \mathbb{H}}} \end{align*}[/latex] As in Exercise 5.9, the second term in the numerator is standard Normal, the denominator is [latex]$\chi^2_{\text{rank} \, \mathbb{H}}$[/latex] divided by its degrees of freedom, and the numerator and denominator are independent. By the definition of non-central [latex]$t$[/latex]-distributions, the ratio’s distribution is [latex]$t_{\text{rank} \, {\mathbb{H}}, a/\sigma}$[/latex].

Number: Exercise 5.11
Question: Let [latex]$T \sim t_k$[/latex]. What’s the distribution of [latex]$T^2$[/latex]?
Answer: From the definition of [latex]$t_k$[/latex], we can represent [latex]$T$[/latex] using independent [latex]$Z \sim N(0, 1)$[/latex] and [latex]$V \sim \chi^2_k$[/latex]. [latex]\begin{align*} T^2 &= \left( \frac{Z}{\sqrt{V/k}} \right)^2\\ &= \frac{Z^2/1}{V/k} \end{align*}[/latex] Because [latex]$Z^2 \sim \chi^2_1$[/latex], this expression matches the definition of the [latex]$f_{1, k}$[/latex] distribution.

Number: Exercise 5.12
Question: Let [latex]${\boldsymbol{\epsilon}} \sim N({\mathbf{0}}, \sigma^2 {\mathbb{I}})$[/latex], and let [latex]${\mathbb{H}}_1$[/latex] and [latex]${\mathbb{H}}_2$[/latex] be orthogonal projection matrices onto two subspaces that are orthogonal to each other. Find the distribution of [latex]$\frac{\| {\mathbb{H}}_1 {\boldsymbol{\epsilon}} \|^2/ \text{rank} \, {\mathbb{H}}_1}{\| {\mathbb{H}}_2 {\boldsymbol{\epsilon}} \|^2/\text{rank} \, {\mathbb{H}}_2}$[/latex].
Answer: We can divide both the numerator and the denominator by [latex]$\sigma^2$[/latex] to produce random variables whose distributions we know from Exercise 5.8. [latex]\begin{align*} \frac{\| {\mathbb{H}}_1 {\boldsymbol{\epsilon}} \|^2/ \text{rank} \, {\mathbb{H}}_1}{\| {\mathbb{H}}_2 {\boldsymbol{\epsilon}} \|^2/\text{rank} \, {\mathbb{H}}_2} &= \frac{\| {\mathbb{H}}_1 ({\boldsymbol{\epsilon}}/\sigma) \|^2/ \text{rank} \, {\mathbb{H}}_1}{\| {\mathbb{H}}_2 ({\boldsymbol{\epsilon}}/\sigma) \|^2/\text{rank} \, {\mathbb{H}}_2} \end{align*}[/latex] The numerator is a [latex]$\chi^2_{\text{rank} \, {\mathbb{H}}_1}$[/latex]-distributed random variable divided by its degrees of freedom, while the denominator is a [latex]$\chi^2_{\text{rank} \, {\mathbb{H}}_2}$[/latex]-distributed random variable divided by its degrees of freedom. Because the subspaces are orthogonal, we know that the two orthogonal projections are independent of each other, allowing us to conclude that the ratio matches the definition of [latex]$f_{\text{rank} \, {\mathbb{H}}_1, \text{rank} \, {\mathbb{H}}_2}$[/latex].

Number: Exercise 5.13
Question: Let [latex]${\boldsymbol{\epsilon}} \sim N({\mathbf{0}}, \sigma^2 {\mathbb{I}})$[/latex], and let [latex]${\mathbb{H}}_1$[/latex] and [latex]${\mathbb{H}}_2$[/latex] be orthogonal projection matrices onto two subspaces that are orthogonal to each other. Find the distribution of [latex]$\frac{\| {\mathbf{v}} + {\mathbb{H}}_1 {\boldsymbol{\epsilon}} \|^2/ \text{rank} \, {\mathbb{H}}_1}{\| {\mathbb{H}}_2 {\boldsymbol{\epsilon}} \|^2/\text{rank} \, {\mathbb{H}}_2}$[/latex], where [latex]${\mathbf{v}}$[/latex] is a non-random vector.
Answer: Divide both the numerator and the denominator by [latex]$\sigma^2$[/latex]. [latex]\begin{align*} \frac{\| {\mathbf{v}} + {\mathbb{H}}_1 {\boldsymbol{\epsilon}} \|^2/ \text{rank} \, {\mathbb{H}}_1}{\| {\mathbb{H}}_2 {\boldsymbol{\epsilon}} \|^2/\text{rank} \, {\mathbb{H}}_2} &= \frac{\| \tfrac{1}{\sigma}{\mathbf{v}} + {\mathbb{H}}_1 ({\boldsymbol{\epsilon}}/\sigma) \|^2/ \text{rank} \, {\mathbb{H}}_1}{\| {\mathbb{H}}_2 ({\boldsymbol{\epsilon}}/\sigma) \|^2/\text{rank} \, {\mathbb{H}}_2} \end{align*}[/latex] As in Exercise 5.12, the denominator is [latex]$\chi^2_{\text{rank} \, {\mathbb{H}}_2}$[/latex]-distributed and is independent of the numerator. This time the numerator is a non-central [latex]$\chi^2$[/latex] random variable divided by its degrees of freedom with non-centrality parameter [latex]$\| \tfrac{1}{\sigma}{\mathbf{v}} \|^2 = \|{\mathbf{v}}\|^2/\sigma^2$[/latex]. Thus the ratio’s distribution matches the definition of [latex]$f_{\text{rank} \, {\mathbb{H}}_1, \text{rank} \, {\mathbb{H}}_2, \|{\mathbf{v}}\|^2/\sigma^2}$[/latex].

Number: Exercise 6.01
Question: Let [latex]${\mathbf{x}}_i$[/latex] represent the explanatory value(s) of the [latex]$i$[/latex]th observation. Consider modeling the response variable by [latex]\begin{align*} Y_i = f_\theta({\mathbf{x}}_i) + \epsilon_i \end{align*}[/latex] with [latex]$\epsilon_1, \ldots, \epsilon_n \overset{iid}{\sim} N(0, \sigma^2)$[/latex] and [latex]$\theta \in \Theta$[/latex] indexing a set of possible functions. (Notice that this form is far more general than the linear model with iid Normal errors.) Show that the maximum likelihood estimator for [latex]$\theta$[/latex] is precisely the parameter value that minimizes the sum of squared residuals.
Answer: The response values have distribution [latex]$Y_i \sim N(f_\theta({\mathbf{x}}_i), \sigma^2)$[/latex] and are independent of each other. Because of independence, the overall likelihood [latex]$L(\theta; {\mathbf{Y}})$[/latex] is the product of the individual observations’ likelihoods. [latex]\begin{align*} L(\theta; {\mathbf{Y}}) &= \prod_{i=1}^n \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{1}{2\sigma^2} (Y_i - f_\theta({\mathbf{x}}_i))^2 }\\ &= \left( \frac{1}{2 \pi \sigma^2} \right)^{n/2} e^{-\frac{1}{2\sigma^2} \sum_{i=1}^n (Y_i - f_\theta({\mathbf{x}}_i))^2 } \end{align*}[/latex] The parameter [latex]$\theta$[/latex] only appears in the sum of squared residuals [latex]$\sum_{i=1}^n (Y_i - f_\theta({\mathbf{x}}_i))^2$[/latex]. The smaller the sum of squared residuals is, the larger the likelihood is, so the “least-squares parameter” is exactly the maximum likelihood estimator. Notice that this equivalence doesn’t depend on the value of [latex]$\sigma$[/latex] and that it holds even if [latex]$\sigma$[/latex] is unknown.

Number: Exercise 6.02
Question: Suppose [latex]${\mathbf{Y}} = {\mathbb{M}} {\boldsymbol{\gamma}} + {\boldsymbol{\epsilon}}$[/latex] with error vector [latex]${\boldsymbol{\epsilon}} \sim N({\mathbf{0}}, \sigma^2 {\mathbb{I}})$[/latex]. If [latex]${\mathbf{\widehat{Y}}}$[/latex] is the least-squares linear regression’s prediction vector for design matrix [latex]${\mathbb{M}}$[/latex], what’s the distribution of [latex]$\| {\mathbf{Y}} - {\mathbf{\widehat{Y}}} \|^2/\sigma^2$[/latex]?
Answer: We saw in Chapter 4 that the least-squares residual vector [latex]${\mathbf{Y}} - {\mathbf{\widehat{Y}}}$[/latex] is the orthogonal projection of [latex]${\boldsymbol{\epsilon}}$[/latex] onto [latex]$C({\mathbb{M}})^\perp$[/latex] which has dimension [latex]$n-\text{rank} \, {\mathbb{M}}$[/latex]. The standardized version [latex]${\boldsymbol{\epsilon}}/\sigma$[/latex] is standard Normal, so according to Exercise 5.8, [latex]\begin{align*} \frac{\| {\mathbf{Y}} - {\mathbf{\widehat{Y}}} \|^2}{\sigma^2} &= \| ({\mathbb{I}} - {\mathbb{H}}) ({\boldsymbol{\epsilon}}/\sigma) \|^2\\ &\sim \chi^2_{n-\text{rank} \, \mathbb{M}} \end{align*}[/latex] where [latex]${\mathbb{H}}$[/latex] represents the orthogonal projection matrix onto [latex]$C({\mathbb{M}})$[/latex].

Number: Exercise 6.03
Question: Let [latex]${\widetilde{\mathbb{X}}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] be a centered explanatory data matrix with full rank. Assume [latex]\begin{align*} {\mathbf{Y}} = \alpha + {\widetilde{\mathbb{X}}} {\boldsymbol{\beta}} + {\boldsymbol{\epsilon}} \end{align*}[/latex] with [latex]$\epsilon_1, \ldots, \epsilon_n \overset{iid}{\sim} N(0, \sigma^2)$[/latex]. Write the standardized version of the least-squares slope [latex]${\hat{\beta}}_j$[/latex]. What is its distribution?
Answer: The distribution of [latex]${\hat{\beta}}_j$[/latex] is Normal because its a linear transformation of [latex]${\mathbf{Y}}$[/latex] which is Normal. Its expectation equals the [latex]$j$[/latex]th entry of [latex]${\mathbb{E}} {\boldsymbol{\hat{\beta}}} = {\boldsymbol{\beta}}$[/latex], and its variance equals the [latex]$j$[/latex]th diagonal of [latex]$\text{cov}\, {\boldsymbol{\hat{\beta}}} = \tfrac{\sigma^2}{n} \Sigma^{-1}$[/latex]: [latex]\begin{align*} {\hat{\beta}}_j \sim N(\beta_j, \tfrac{\sigma^2}{n} \Sigma^{-1}_{jj}). \end{align*}[/latex] The standardized version is [latex]\begin{align*} \frac{{\hat{\beta}}_j - \beta_j}{\tfrac{\sigma}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}}} \sim N(0, 1). \end{align*}[/latex]

Number: Exercise 6.04
Question: Let [latex]${\widetilde{\mathbb{X}}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] be a centered explanatory data matrix with full rank. Assume [latex]\begin{align*} {\mathbf{Y}} = \alpha + {\widetilde{\mathbb{X}}} {\boldsymbol{\beta}} + {\boldsymbol{\epsilon}} \end{align*}[/latex] with [latex]$\epsilon_1, \ldots, \epsilon_n \overset{iid}{\sim} N(0, \sigma^2)$[/latex]. Devise a t-distributed random variable involving the least-squares slope [latex]${\hat{\beta}}_j$[/latex].
Answer: From Exercise 6.3, the standardized version is [latex]$\frac{{\hat{\beta}}_j - \beta_j}{\tfrac{\sigma}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}}} \sim N(0, 1)$[/latex]. Exercise 2.8 implies that [latex]${\boldsymbol{\hat{\beta}}}$[/latex] is a function of [latex]${\mathbb{H}} {\boldsymbol{\epsilon}}$[/latex], so the ratio trick allows us to substitute [latex]${\hat{\sigma}}$[/latex] for [latex]$\sigma$[/latex] to derive [latex]\begin{align*} \frac{{\hat{\beta}}_j - \beta_j}{\tfrac{\hat{\sigma}}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}}} &\sim t_{n-m-1}. \end{align*}[/latex]

Number: Exercise 6.05
Question: Let [latex]${\widetilde{\mathbb{X}}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] be a centered explanatory data matrix with full rank. Assume [latex]\begin{align*} {\mathbf{Y}} = \alpha + {\widetilde{\mathbb{X}}} {\boldsymbol{\beta}} + {\boldsymbol{\epsilon}} \end{align*}[/latex] with [latex]$\epsilon_1, \ldots, \epsilon_n \overset{iid}{\sim} N(0, \sigma^2)$[/latex]. Devise a [latex]$95\%$[/latex] confidence interval for [latex]$\beta_j$[/latex].
Answer: From Exercise 6.4, [latex]$\frac{{\hat{\beta}}_j - \beta_j}{\tfrac{\hat{\sigma}}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}}} \sim t_{n-m-1}$[/latex], so [latex]\begin{align*} {\mathbb{P}} \left\{ -\tau_{n-m-1}^{-1}(.975) \leq \frac{\beta_j - {\hat{\beta}}_j}{\tfrac{\hat{\sigma}}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}}} \leq \tau_{n-m-1}^{-1}(.975) \right\} = .95. \end{align*}[/latex] The event can be rewritten as [latex]\begin{align*} {\hat{\beta}}_j - \tau_{n-m-1}^{-1}(.975) \tfrac{\hat{\sigma}}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}} \leq \beta_j \leq {\hat{\beta}}_j + \tau_{n-m-1}^{-1}(.975) \tfrac{\hat{\sigma}}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}} \end{align*}[/latex] which means that [latex]${\hat{\beta}}_j \pm \tau_{n-m-1}^{-1}(.975) \tfrac{\hat{\sigma}}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}}$[/latex] is a [latex]$95\%$[/latex] confidence interval for [latex]$\beta_j$[/latex].

Number: Exercise 6.06
Question: Let [latex]${\widetilde{\mathbb{X}}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] be a centered explanatory data matrix with full rank. Assume [latex]\begin{align*} {\mathbf{Y}} = \alpha + {\widetilde{\mathbb{X}}} {\boldsymbol{\beta}} + {\boldsymbol{\epsilon}} \end{align*}[/latex] with [latex]$\epsilon_1, \ldots, \epsilon_n \overset{iid}{\sim} N(0, \sigma^2)$[/latex]. Devise a test statistic [latex]$T_j$[/latex] for the null hypothesis that [latex]$\beta_j = 0$[/latex].
Answer: From Exercise 6.4, [latex]$\frac{{\hat{\beta}}_j - \beta_j}{\tfrac{\hat{\sigma}}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}}} \sim t_{n-m-1}$[/latex]. The assumption that [latex]$\beta_j=0$[/latex] leads to the test statistic [latex]\begin{align*} T_j &:= \frac{{\hat{\beta}}_j}{\tfrac{\hat{\sigma}}{\sqrt{n}} \sqrt{\Sigma^{-1}_{jj}}}\\ &\sim t_{n-m-1}. \end{align*}[/latex] The significance probability is [latex]$2 \tau_{n-m-1}(-|T_j|)$[/latex].

Number: Exercise 6.07
Question: Section 6.3.7.2 described a test statistic (Equation 6.1) for the null hypothesis that all of the slopes in a multiple linear model are [latex]$0$[/latex]. Is it the same as the test statistic prescribed by Equation 6.3?
Answer: The null hypothesis is that [latex]${\mathbb{E}} {\mathbf{Y}}$[/latex] is in the span of [latex]${\mathbf{1}}$[/latex], so the general approach (Equation 6.3) uses the test statistic [latex]\begin{align*} \frac{\| {\mathbf{\widehat{Y}}} - {\bar{Y}} {\mathbf{1}} \|^2/m}{\hat{\sigma}^2} \sim f_{m, n-m-1}, \end{align*}[/latex] while Section 6.3.7.2 derived the test statistic [latex]\begin{align*} \frac{n \| \Sigma^{1/2} {\boldsymbol{\hat{\beta}}} \|^2 / m}{{\hat{\sigma}}^2} \sim f_{m, n-m-1}. \end{align*}[/latex] Let’s analyze the factor in which they appear to differ. Recall that for multiple linear regression the least-squares prediction vector can be expressed as [latex]\begin{align*} {\mathbf{\widehat{Y}}} &= {\bar{Y}} {\mathbf{1}} + {\widetilde{\mathbb{X}}} {\boldsymbol{\hat{\beta}}} \end{align*}[/latex] where [latex]${\widetilde{\mathbb{X}}}$[/latex] is the centered explanatory data matrix. Therefore, [latex]\begin{align*} \| {\mathbf{\widehat{Y}}} - {\bar{Y}} {\mathbf{1}} \|^2 &= \| {\widetilde{\mathbb{X}}} {\boldsymbol{\hat{\beta}}} \|^2\\ &= {\boldsymbol{\hat{\beta}}}^T {\widetilde{\mathbb{X}}}^T {\widetilde{\mathbb{X}}} {\boldsymbol{\hat{\beta}}}. \end{align*}[/latex] And in the other test statistic, [latex]\begin{align*} n \| \Sigma^{1/2} {\boldsymbol{\hat{\beta}}} \|^2 &= n {\boldsymbol{\hat{\beta}}}^T \underbrace{\Sigma}_{\tfrac{1}{n}{\widetilde{\mathbb{X}}}^T\widetilde{\mathbb{X}}} {\boldsymbol{\hat{\beta}}}\\ &= {\boldsymbol{\hat{\beta}}}^T {\widetilde{\mathbb{X}}}^T {\widetilde{\mathbb{X}}} {\boldsymbol{\hat{\beta}}} \end{align*}[/latex] so they turn out to be exactly the same.

Number: Exercise 6.08
Question: Let [latex]${\mathbf{Y}} = \alpha {\mathbf{1}} + {\widetilde{\mathbb{X}}} {\boldsymbol{\beta}} + {\boldsymbol{\epsilon}}$[/latex] with [latex]${\widetilde{\mathbb{X}}} \in {\pmb{\mathbb{R}}}^{n \times m}$[/latex] representing a centered data matrix. If [latex]${\mathbf{\widehat{Y}}}$[/latex] is the least-squares prediction vector that comes from multiple linear regression, find the distribution of [latex]\begin{align*} \frac{\|{\mathbf{\widehat{Y}}} - {\bar{Y}} {\mathbf{1}}\|^2}{{\hat{\sigma}}^2}. \end{align*}[/latex]
Answer: Based on the preceding discussion, the statistic in question has non-central [latex]$f$[/latex]-distribution. [latex]${\mathbf{\widehat{Y}}}$[/latex] is the projection onto an [latex]$(m+1)$[/latex]-dimensional subspace, while [latex]${\bar{Y}} {\mathbf{1}}$[/latex] is the projection onto a [latex]$1$[/latex]-dimensional subspace. Thus the numerator has [latex]$m$[/latex] degrees of freedom, and the denominator has [latex]$n-m-1$[/latex] degrees of freedom. The non-centrality parameter is [latex]\begin{align*} \| (\alpha {\mathbf{1}} + {\widetilde{\mathbb{X}}} {\boldsymbol{\beta}}) - (\alpha {\mathbf{1}}) \|^2/\sigma^2 &= \| {\widetilde{\mathbb{X}}} {\boldsymbol{\beta}} \|^2/\sigma^2. \end{align*}[/latex]