Find the eigen values using det(A-$\Lambda$I)=0 it'll be 5,-5
Then solve for value of a and b using (A-$\Lambda$I)x=0 where x are two given eigen vectors, you will get value of a,b as a=2 b=-2
It's very easy maybe you are doing some calculation mistake.