What is a Regression Model? Example: Honolulu tide Statistical model for: I depending on Y (random variable) ←− x (non-random variable) Aim: understanding the effect of x on the random quantity Y General formulation: Y |x ∼ Distribution(g(x)) Statistical Problem: Estimate (learn) g(·) from data {(xi , yi )}ni=1 . Use for: I Inference I Description I Prediction I Data compression (parsimonious representations) I ... Example: Honolulu tide Great Variety of Models Remember general model: Y |x ∼ Distribution(g(x)) x can be: I continuous, discrete, categorical, vector . . . Distribution can be: I Gaussian (Normal), Laplace, General exponential family . . . Function g(·) can be: I g(x) = β0 + β1 x, g(x) = PK k=−K βk e−ikx , Cubic spline . . . Fundamental Case: Normal Linear Regression I Example: Professor’s Van Y, x ∈ R, g(x) = β0 + β1 x, Distribution= Gaussian Y |x ∼ N (β0 + β1 x, σ 2 ) m Y = β0 + β1 x + , I ∼ N (0, σ 2 ) Also, x could be vector (Y, β0 ∈ R, x ∈ Rp , β ∈ Rp ): Y |x ∼ N (β0 + β > x, σ 2 ) m Y = β0 + β > x + , Example: Professor’s Van ∼ N (0, σ 2 ) The tools of the trade... Start from Normal linear model −→ gradually generalise... Important features of Normal linear model: I Gaussian distribution I Linearity These two mix well together and give geometric insights to solve the estimation problem. Thus we need to revise some linear algebra and probability... Will mostly concentrate on the Gaussian assumption, but gradually consider: I linear Gaussian regression I nonlinear Gaussian regression I nonparametric Gaussian regression Reminder: Subspaces, Spectra, Projections. Recall that is Q is an n × p real matrix, we define its column space (or range) of Q to be the set spanned by its columns, M(Q) = {y ∈ Rn : ∃β ∈ Rp , y = Qβ}. I Recall that M(Q) is a subspace of Rp . I The columns of Q provide a coordinate system for the subspace M(Q) I If Q is of full column rank (p), then the coordinates β corresponding to a y ∈ M(Q) are unique. I Allows interpretation of system of linear equations Qβ = y. [existence of solution ↔ is y an element of M(Q)?] [uniqueness of solution ↔ is there a unique coordinate vector β?] Reminder: Subspaces, Spectra, Projections. (The Spectral Theorem) A p × p matrix Q is symmetric if and only if there exists a p × p orthogonal matrix U and a diagonal matrix Λ such that > Q = U ΛU . In particular, 1. The columns of U = (u1 . . . up ) are eigenvectors of Q, i.e. there exist λj such that Quj = λj uj , j = 1, ..., p. 2. The entries of Λ are the corresponding eigenvalues of Q, Λ = diag{λ1 , ..., λp } which are real. 3. The rank of Q is the number of non-zero eigenvalues. Note: If the eigenvalues are distinct, the eigenvectors are unique (up to re-ordering). Reminder: Subspaces, Spectra, Projections. Recall two further important subspaces associated with a real n × p matrix Q: I I The nullspace (or kernel), ker(Q), of Q is the subspace defined as ker(Q) = {y ∈ Rp : Qy = 0}. The orthogonal complement of M(Q), M⊥ (Q), is the subspace defined as M⊥ (Q) = {y ∈ Rn : y > Qx = 0, ∀x ∈ Rp } = {y ∈ Rn : y > v = 0, ∀v ∈ M(Q)}. Obviously, the orthogonal complement is defined for arbitrary subspaces (see second equality). Reminder: Subspaces, Spectra, Projections. (Singular Value Decomposition) Any n × p real matrix can be factorised as Q = U n×p Σ V n×n n×p p×p > , where U and V > are orthogonal with columns called left singular vectors and right singular vectors, respectively, and Σ is diagonal with real entries called singular values. 1. The left singular vectors are eigenvectors of QQ> . 2. The right singular vectors are eigenvectors of Q> Q. 3. The squares of the singular values are eigenvalues of both QQ> and Q> Q. 4. The left singular vectors corresponding to non-zero singular values form an orthonormal basis for M(Q). 5. The left singular vectors corresponding to zero singular values form an orthonormal basis for M⊥ (Q). Reminder: Subspaces, Spectra, Projections. Recall that a matrix Q is called idempotent if Q2 Reminder: Subspaces, Spectra, Projections. = Q. An orthogonal projection (henceforth projection) onto a subspace V is a symmetric idempotent matrix H such that M(H) = V. Proposition Proposition Proof. The only possible eigenvalues of a projection matrix are 0 and 1. (I − H)> = I − H > = I − H since H is symmetric. Furthemore, (I − H)2 = I 2 − 2H + H 2 = I − H since H is idempotent. Thus I − H is a projection matrix. Proposition If P and Q are projection matrices onto a subspace V, then P = Q. Proposition If x1 , ..., xp are linearly independent and are such that span(x1 , ..., xp ) = V, then the projection onto V can be represented as H = X(X > X)−1 X > Let V be a subspace and H be a projection onto V. Then I − H is the projection matrix onto V⊥ . It remains to identify the column space of I − H. Let H = U ΛU > be the spectral decomposition of H. Then I − H = U U > − U ΛU > = U (I − Λ)U > Hence the column space of I − H is spanned by the eigenvectors of H corresponding to zero eigenvalues of H, and this coincides with M⊥ (H) = V⊥ . where X is a matrix with columns x1 , ..., xp . Reminder: Subspaces, Spectra, Projections. Reminder: Subspaces, Spectra, Projections. (proof continued). Notice that for v ∈ V, Proposition Let V be a subspace of Rn and H be a projection onto V. Then kx − Hxk ≤ kx − vk, ∀v ∈ V. Proof. Let H = U ΛU > be the spectral decomposition of H, U = (u1 ...un ) and Λ = diag{λ1 , ..., λn } Letting p = dim(V), kx − Hxk2 = = i=1 n X i=1 = n X i=1 1. λ1 = . . . = λp = 1 and λp+1 = . . . = λn = 0 2. u1 , ..., un is an orthonormal basis of Rn n X (x> ui − (Hx)> ui )2 (x> ui − x> Hui )2 [H is symmetric] (x> ui − λi x> ui )2 [u’s are eigenvectors of H] = 0+ 3. u1 , ..., up is an an orthonormal basis of V. ≤ [orthonormal basis] n X (x> ui )2 i=p+1 n X > [eigenvalues 0 or 1] p X (x ui ) + (x> ui − v > ui )2 i=p+1 2 = kx − vk . 2 i=1 Positive-Definite Matrices A p × p real symmetric matrix Ω is called non-negative definite (denoted as Ω 0) if and only if x> Ωx ≥ 0 for all x ∈ Rp . If x> Ωx > 0 for all x ∈ Rp \ {0}, then we call Ω positive definite (denoted as Ω 0). An equivalent definition is as follows: A p × p real symmetric matrix Ω is called non-negative definite (denoted as Ω 0) if and only the eigenvalues of Ω are non-negative. If the eigenvalues of Ω are strictly positive, then Ω is called positive definite (denoted as Ω 0). Lemma (Exercise) Covariance Matrices Let Y = (Y1 , ..., Yn )> be a random n × 1 vector such that EkY k2 < ∞. The covariance matrix of Y , say Ω, is the n × n symmetric matrix with entries Ωij = Cov(Yi , Yj ) = E[(Yi − E[Yi ])(Yj − E[Yj ])], 1 ≤ i ≤ j ≤ n. That is, the covariance matrix encodes the variances of the coordinates of Y (on the diagonal) and the covariances between the coordinates of Y (off the diagonal). If we write µ = E[Y ] = (E[Y1 ], ..., E[Yn ])> for the mean vector of Y , then the covariance matrix of Y can be written as E[(Y − µ)(Y − µ)> ] = E[Y Y > ] − µµ> . Prove that the two definitions are equivalent. Whenever Y is a random vector, we will write Cov(Y ) for the covariance matrix of Y . Covariance Matrices Non-negative Matrices ≡ Covariance Matrices Lemma Let Y be a random d × 1 vector such that EkY k2 < ∞. Let µ be the mean vector and Ω be the covariance matrix of Y . If A is a p × d real matrix, the mean vector and covariance matrix of AY are given by Aµ and AΩA> , respectively. Proposition (Non-Negative and Covariance Matrices) Exercise. Let Ω be a real symmetric matrix. Then Ω is non-negative definite if and only if Ω is the covariance matrix of some random variable Y . Corollary (Covariance of Projections) Proof. Let Y be a random d × 1 vector such that EkY k2 < ∞. Let β, γ ∈ Rd be fixed vectors. If Ω denotes the covariance matrix of Y, Exercise. Proof. I The variance of β > Y is given by β > Ωβ; I The covariance of β > Y with γ > Y is given by γ > Ωβ. Gaussian Vectors and Affine Transformations Definition: Multivariate Gaussian Distribution A random vector Y in Rd has the multivariate normal distribution if and only if β > Y has the univariate normal distribution ∀ β ∈ Rd . (recall the Cram´er-Wold device which says that the distribution of a random vector is completely determined by the distribution of all its one-dimensional projections). How can we used this definition to determine basic properties? Recall that the moment generating function (MGF) of a random vector W in Rd is defined as MW (θ) = E[eθ > W θ ∈ Rd , ], provided the expectation exists. When the MGF exists it characterises the distribution of the random vector. Furthermore, two random vectors are Gaussian Vectors and Affine Transformations Useful facts: 1. Moment generating function of Y ∼ N (µ, Ω): MY (u) = exp µ> u + 12 u> Ωu 2. Y ∼ N (µp×1 , Ωp×p ) and given Bn×p and θn×1 , then BY + θ ∼ N (θ + Bµ, BΩB > ) 3. N (µ, Ω) density, assuming Ω nonsingular: 1 exp − 12 (y − µ)> Ω−1 (y − µ) fY (y) = p/2 1/2 (2π) |Ω| 4. Constant density isosurfaces are ellipsoidal 5. Marginals of Gaussian are Gaussian (converse NOT true) 6. Ω diagonal ⇔ Independent coordinates 7. If Y ∼ N (µp×1 , Ωp×p ), AY independent of BY ⇐⇒ AΩB > = 0. independent if and only if their joint MGF is the product of their marginal MGF’s. Proposition (Property 1: Moment Generating Function) Proposition (Property 2: Affine Transformation) The moment generating function of Y ∼ N (µ, Ω) is MY (u) = exp µ> u + 21 u> Ωu For Y ∼ N (µp×1 , Ωp×p ) and given Bn×p and θn×1 , we have Proof. Proof. Let θ ∈ Rd be an arbitrary vector. Then θ> Y is Gaussian with mean θ> µ and variance θ> Ωθ. Hence it has moment generating function: h > i t2 > tθ Y > = exp t(θ µ) + (θ Ωθ) . Mθ> Y (t) = E e 2 Now take t = 1 and observe that h Mθ> Y (1) = E eθ > Y i BY + θ ∼ N (θ + Bµ, BΩB > ) MBY +θ (u) = = = = MY (θ) Combining the two, we conclude that 1 > > MY (θ) = exp µ θ + θ Ωθ 2 = = h i n o h i E exp{u> (BY + θ)} = exp u> θ E exp{(B > u)> Y } n o exp u> θ MY (B > u) n o 1 exp u> θ exp (B > u)> µ + u> BΩB > u 2 1 exp u> θ + u> (Bµ) + u> BΩB > u 2 1 exp u> (θ + Bµ) + u> BΩB > u 2 And this last expression is the MGF of a N (θ + Bµ, BΩB > ) distribution. Proposition (Property 3: Density Function) Let Ωp×p be nonsingular. The density of N (µp×1 , Ωp×p ) is 1 fY (y) = exp − 12 (y − µ)> Ω−1 (y − µ) p/2 1/2 (2π) |Ω| Proof. Let Z = (Z1 , ..., Zp )> be a vector of iid N (0, 1) random variables. Then, because of independence, proof continued. (a)+(b) =⇒ the N (0, I) density is fZ (z) = Now observe that from our Property 2, we have Y = Ω1/2 Z + µ ∼ N (µ, Ω). By the change of variables formula, fY (y) Y Y 1 1 1 1 > √ exp − zi2 = fZ (z) = fZi (zi ) = exp − z z 2 2 2π (2π)p/2 i=1 i=1 p (b) The MGF of Z is given by " !# p p X Y MZ (u) = E exp ui Zi = E[exp(ui Zi )] = exp{u> u/2} i=1 i=1 exp − 12 z > z . By the spectral theorem, Ω admits a square root, Ω1/2 . Furthermore, since Ω is non-singular, so is Ω1/2 . (a) The density of Z is given by p 1 (2π)p/2 = fΩ1/2 Z+µ (y) = |Ω−1/2 |fZ (Ω−1/2 (y − µ)) 1 1 > −1 (y − µ) Ω (y − µ) exp − 2 (2π)p/2 |Ω|1/2 = [recall that to obtain the density of W = g(X) at w, we need to evaluate fX at g −1 (w) but also multiply by the Jacobian determinant of g −1 at w] which is the MGF of a p-variate N (0, I) distribution. Proposition (Property 4: Isosurfaces) The isosurfaces of a N (µp×1 , Ωp×p ) are (p − 1)-dimensional ellipsoids centred at µ, with principle axes given by the eigenvectors of Ω and with anisotropies given by the ratios of the square roots of the corresponding eigenvalues of Ω . Proposition (Property 6: Diagonal Ω ⇐⇒ Independence) Let Y = (Y1 , ...Yp )> ∼ N (µp×1 , Ωp×p ). Then the Yi are mutually independent if and only if Ω is diagonal. Proof. Suppose that the Yj are independent. Property 5 yields Yj ∼ N (µj , σj2 ) for some σj > 0. Thus the density of Y is Proof. Exercise: Use Property 3, and the spectral theorem. fY (y) = p Y fYj (yj ) = j=1 Proposition (Property 5: Coordinate Distributions) Let Y = (Y1 , ...Yp )> ∼ N (µp×1 , Ωp×p ). Then Yj ∼ N (µj , Ωjj ) . Proof. Observe that Yj = (0 , 0 , ..., jth 1 , ..., 0 , 0)Y and use Property 2. |{z} position = 1 (2π)p/2 |diag{σ12 , ..., σp2 }|1/2 p Y i=1 1 1 (yj − µj )2 √ exp − 2 σj2 σj 2π 1 exp − (y − µ)> diag{σ1−2 , ..., σp−2 }(y − µ) . 2 Hence Y ∼ N (µ, diag{σ12 , ..., σp2 }), i.e. the covariance Ω is diagonal. Conversely, assume Ω is diagonal, say Ω = diag{σ12 , ..., σp2 }. Then we can reverse the steps of the first part to see that the joint density fY (y) can be written as a product of the marginal densities fYj (yj ), thus proving independence. Proposition (Property 7: AY, BY indep ⇐⇒ AΩB > = 0) If Y ∼ N (µp×1 , Ωp×p ), and Am×p , Bd×p be real matrices. Then, AY independent of BY ⇐⇒ AΩB > = 0. proof continued. For the converse, assume that AY and BY are independent. Then, ∀u, v, MW (θ) = MBY (u)MAY (v) Proof. It suffices to prove the result assuming µ = 0. First assume AΩB > = 0. Let W(m+d)×1 = BY and θ(m+d)×1 = AY MW (θ) = = = = . h n oi E[exp{W > θ}] = E exp Y > B > u + Y > A> v h n oi E exp Y > (B > u + A> v) = MY (B > u + A> v) 1 > exp (B u + A> v)> Ω(B > u + A> v) 2      1 > > > >  u BΩB > u + v > AΩA> v + u> BΩA v + v AΩB u exp | {z } | {z }  2 =0 = um×1 vd×1 =⇒ exp 1 u> BΩB > u + v > AΩA> v + u> BΩA> v + v > AΩB > u = 2 1 > 1 > = exp u BΩB > u exp v AΩA> v 2 2 1 =⇒ exp × 2v > AΩB > u = 1 2 =⇒ v > AΩB > u = 0 =⇒ AΩB > = 0. ∀ u, v =0 MBY (u)MAY (v) i.e. the joint MGF is the product of the marginal MGFs, proving independence. Gaussian Quadratic Forms and the χ2 Distribution Definition (χ2 distribution) Let Z ∼ N (0, Ip×p ). Then kZk2 is said to have the χ2 distribution with p degrees of freedom. [i.e. χ2p is the distribution of the sum of squares of p real independent standard Gaussian r.v.’s] Definition (F distribution) χ2p χ2q Let V ∼ and W ∼ be independent random variables. Then (V /p)/(W/q) is said to have the F distribution with p and q degrees of freedom. Gaussian Quadratic Forms and the χ2 Distribution Proposition (Gaussian Quadratic Forms) 1. If Z ∼ N (0p×1 , Ip×p ) and H is a projection of rank r ≤ p, Z > HZ ∼ χ2r . 2. Y ∼ N (µp×1 , Ωp×p ) with Ω nonsingular =⇒ (Y − µ)> Ω−1 (Y − µ) ∼ χ2p . Exercise: Prove (1) and (2).