$R = a^*b^*(ba)^*a^*$
for finding shortest string that is not in language it is better to look strings of length $0$, then of length $1$ and so on
$\text{length}0 \{ \epsilon \}$ is in $L$
$\text{length}1 \{a, b\}$ all belong to $L$
$\text{length}2 \{aa, ab, ba, bb\}$ all belong to $L$
$\text{length}3 \{aaa, aab, aba, abb, baa, bab, bba, bbb\}$ bab does not belong to L