Answer: D \( \theta = 90^\circ \)
We can think as Linear Algebra Perspective or as Variance Maximization Perspective.
Properties of a Symmetric Matrix:
- All eigenvectors of a symmetric matrix are linearly independent.
- Eigenvectors corresponding to distinct eigenvalues are orthogonal. If \( \lambda_i \ne \lambda_j \), then \( v_j^\top v_i = 0 \).
Principal components are eigenvectors of the data covariance matrix \( S = \frac{1}{n} X^\top X \).
Since the covariance matrix is symmetric, all principal component vectors can be chosen to be orthogonal to each other. Therefore, \( \theta = 90^\circ \).
Each principal component captures the maximum remaining variance.
Since we do not want any redundancy while capturing variance, we require the covariance between any two principal components to be zero, i.e., \( \text{Cov}(z_i, z_j) = 0 \quad (i \ne j) \).
Uncorrelated linear projections in Euclidean space imply orthogonal directions. \( \Rightarrow \theta = 90^\circ \)
