A mixed pair in a tennis match consists of 1 man and 1 woman. To form the team you have to select 4 pairs of 1 man and 1 woman.So there'll always be 4 men and 4 women in team. Hence your approach is incorrect.
So suppose you have 5 men: M1, M2, M3, M4, M5 and 5 women: W1, W2, W3, W4, W5
Now, for first pair if you select a man, you have 5 options for women to form a team.
For second pair, if you select a man, you have 4 options for women (one women already formed the team, so she can't be included in any other team)
Similarly, for 3rd pair, you have 3 options and 4th pair you have 2 options.
Now from 5 men, you have to select 4 [e.g.you can select M1, M2, M3, M4 or M2, M3, M4, M5 or ... ] i.e., $5 \choose 4$
So total possible number of selections: $5 \choose 4$ * 5 * 4 * 3 * 2 = 5 * 120 = 600