$M$ is a square matrix, say of size $n \times n$.

Let $M$ has column vectors $m_1,m_2,...,m_n$ in $\mathbb{R}^n$.

$|M| = 0 \implies$ column vectors of $M$ are linearly dependent.

$\therefore c_1*m_1 + c_2*m_2 + ... + c_n*m_n = 0$, such that atleast one $c_i \neq 0$.

$\implies Mx = 0$ has non-trivial solution.

(Non-trivial means $x \neq 0$)