You can also use some intuition after looking at the answers as to what the final answer should look like, and see if you can get one of the answers.
Since the maximum number of edges in a simple graph is $\frac{n^{2} - n}{2}$, and we want atleast $\frac{n^{2} - 3n}{2}$ edges, this will be a sum of terms. We select $\frac{n^{2} - 3n}{2}$ edges, OR $\frac{n^{2} - 3n}{2} + 1$ edges, OR... uptill the maximum of $\frac{n^{2} - n}{2}$ edges.
So the answer should look like $\sum_{T=\frac{n^{2} - 3n}{2}}^{\frac{n^{2} - n}{2}} (_{T}^{upper}\textrm{C})$ where upper is ${\frac{n^{2} - n}{2}}$, as defined above. But we see that none of the answers looks exactly like this. The closest answers are B and D.
If you've done P & C from Rosen, you'll find this technique of shifting the summation in the generating functions section.
So let's try to shift the summation in our answer to see if it can match any of the possible answers.
Note that B cannot be the answer because it says the upper limit of summation is what the lower limit should be, and the combinatorial term is also wrong (we want to select from a max of $\frac{n^{2} - n}{2}$ edges). So let's try out D.
Set $k$ such that when $T = \frac{n^{2} - 3n}{2}$, then $k = 0$. So clearly $k = T - \frac{n^{2} - 3n}{2}$.
Now when $T = \frac{n^{2} - n}{2}$, then $k$ must be $n$. Do we get this?
$k = T - \frac{n^{2} - 3n}{2}$
$k = \frac{n^{2} - n}{2} - \frac{n^{2} - 3n}{2}$
$k = \frac{n^{2} - n - n^{2} + 3n}{2}$
$k = n$
So we have successfully shifted the summation on our answer to match one of the answers given. Answer is D.