Option 1 should be correct. Since A$\neq$B and AB = BA hence A can have an inverse. Since if A had an inverse then
multiplying both sides of the equation AB = BA by A$^{-1}$ we get
$A^{-1}*AB = A^{-1} * BA$
$(A^{-1}*A)B = A^{-1} * AB$ // Since AB = BA
$I*B = (A^{-1} * A)B$
$B = I*B$
Which confirms the equality. Hence for sustenance of the equation A will have an inverse and hence
|A| $\neq$ 0
Now since A$^{2}$ = B$^{2}$
Hence
A$^{2}$ – B$^{2}$ = 0
A$^{2} + BA – BA - B^{2}$ = 0
A$^{2} + BA – AB - B^{2}$ = 0 // Since AB = BA
A$^{2} – AB + BA - B^{2}$ = 0
A*(A – B) +B*(A – B) = 0
(A + B)*(A – B) = 0
Since A$\neq$B
A + B = 0
Hence |A+B| = 0