The test Root Mean Squared Error (RMSE) of the unregularized least squares model ($\hat{\gamma}$) compared to the Ridge regression model ($\hat{\beta}$) depends entirely on the chosen value of the regularization hyperparameter $\lambda$ and the characteristics of the data.
When $\lambda = 0$: The Ridge regression penalty term evaluates to zero, making the two models identical mathematically. In this case, their test RMSEs would be strictly equal.
When $\lambda$ is optimally tuned ($\lambda > 0$): Ridge regression introduces a small amount of bias to achieve a significant reduction in model variance by shrinking the parameter estimates. If the unregularized OLS model is overfitting the training data, an optimally tuned Ridge model will generalize better and yield a lower test RMSE.
When $\lambda$ is excessively large: The $L_2$ penalty dominates the loss function, forcing the parameters too close to zero. This causes the model to underfit the data. The resulting high bias would make the test RMSE of the $\hat{\beta}$ model greater than that of the unregularized $\hat{\gamma}$ model.
Because $\lambda$ is only defined as a non-negative hyperparameter and we do not have information about the unseen testing data or whether the unregularized model is overfitting, there is no guaranteed relationship between their test errors.
The correct option is D. impossible to tell.
