Consider the least squares regression model, $\hat{\mathbb{Y}}=\mathbb{X} \theta$. Assume that $\mathbb{X}$ and $\mathbb{Y}$ refer to the design matrix and true response vector for our training data.
Let $\hat{\gamma}$ be the parameter vector that minimizes mean squared error without regularization. Specifically:
$$
\hat{\gamma}=\arg \min _\gamma \frac{1}{n}\|\mathbb{Y}-\mathbb{X} \gamma\|_2^2
$$
Let $\hat{\beta}$ be the parameter vector that minimizes mean squared error with $L_2$ regularization, using a non-negative regularization hyperparameter $\lambda$ (i.e. ridge regression). Specifically:
$$
\hat{\beta}=\arg \min _\beta \frac{1}{n}\|\mathbb{Y}-\mathbb{X} \beta\|_2^2+\lambda \sum_{j=1}^p \beta_j^2
$$
If we set $\lambda=0$, then $\|\hat{\gamma}\|_2^2$ is_______ $\|\hat{\beta}\|_2^2$.
- less than
- equal to
- greater than
- impossible to tell
