Let us assume L is RE. So, we have a TM N which will say "yes" if given an encoding of a TM whose language is infinite. Now using N we try to solve the non-halting problem as follows:

A non-halting problem is given as follows: Given an encoding of TM <H> and a word w, whether H does not halt on w. That is, we have to decide if H does not halt on w. This problem is proved to be not RE and so no TM can not even say "yes" for "yes" cases of the problem.

Now, given a non-halting problem (<H>, w), we proceed as follows: We make a new TM say A, which on input x, just simulates H on w for |x| steps. If H halts on w, A goes to reject state. Otherwise, it accepts. So, L(A) = $\Sigma^*$ if H does not halt on w and L(A) = a finite set if H halts on w. (If H halts in |x| steps for w, any string with length greater than |w|, would certainly be not in L, making L a finite set).

Now, we just give the encoding of A to our assumed TM N. If N says "accept", we have that L(A) is infinite => H does not halt on w => we solved "yes" case of the non-halting problem, which is not possible. Hence, our initial assumption of L is RE is false. L is not RE.

Proving L' is not RE is easy.

L' = {<M> | L(M) is finite}

L(M) is finite is a non-monotonic property of TM. Because we can take TM $T_{yes}$ with $L(T_{yes} ) = \phi$ and $T_{no}$ with $L(T_{no}) = \Sigma^*$. Here, $T_{yes}$ satisfies the property (of being finite) and $T_{no}$ does not satisfy the property and $L(T_{yes}) \subset L(T_{no})$, making this a non-monotonic property of TM. And hence this becomes not RE as per Rice's theorem second part.

So, both L and L' are not RE making (D) the correct answer.

http://gatecse.in/wiki/Rice%27s_Theorem_with_Examples