Here, the determinant of the coefficient matrix $\begin{vmatrix} 1 & 1 &1 \\ 1 & 2 & 4\\ 1 & 4 & 10 \end{vmatrix}=1(20-16)-1(10-4)+1(4-2)=0$.
So, the system of the equations must have either no solution or infinitely many solutions.
Now using Gaussian elimination method on the systems of equations:
$ \begin{pmatrix} \left.\begin{matrix} 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 4 & 10 \end{matrix} \right| \begin{matrix} 1 \\ \eta \\ \eta^2 \end{matrix} \end{pmatrix} \\ \sim \begin{pmatrix} \left.\begin{matrix} 1 & 1 & 1 \\ 0 & 1 & 3 \\ 0 & 3 & 9 \end{matrix} \right| \begin{matrix} 1 \\ \eta -1 \\ \eta^2 -1 \end{matrix} \end{pmatrix} ;[ r_2 \leftarrow r_2-r_1, r_3\leftarrow r_3-r_1] \\ \sim \begin{pmatrix} \left.\begin{matrix} 1 & 1 & 1 \\ 0 & 1 & 3 \\ 0 & 0 & 0 \end{matrix} \right| \begin{matrix} 1 \\ \eta -1 \\ \eta^2 -1-3(\eta-1) \end{matrix} \end{pmatrix} ;[ r_3 \leftarrow r_3-3 \times r_2]$
To have a solution (in fact infinitely many solutions), as the LHS of the final row are all zeros, the RHS of this final row must be zero.
$$ \begin{align}\therefore \eta^2-1-3(\eta-1) &=0 \\ \Rightarrow (\eta -1)(\eta +1-3)&=0 \\ \Rightarrow (\eta-1)(\eta-2)&=0 \\ \Rightarrow \eta &= 1,2\end{align}$$
So the correct answer is D.
