int len = ((strlen(s) - strlen(t)) > c) ? strlen(s) : strlen(t);
Means is strlen(s) - strlen(t) > 0?
If yes, choose stren(s). If no, choose strlen(t).
Mathematically, $3-5 = -2$
So, we choose strlen(t) under normal circumstances.
But, as given in the question, strlen is an unsigned int. Unsigned int $-$ unsigned int can't be signed. Depending on the implementation (out of the scope of this question) the value of $3-5$ would be decided. It is however absolutely sure that the value won't be negative.
Hence, strlen(s) - strlen(t) > 0? Yes.
So, choose strlen(s) which is 3.