$[x^{17}](x+x^2+x^3+x^4+x^5+x^6)^4$
takeout x from the expression
$x^4(1+x+x^2+x^3+x^4+x^5)^4$
we got $x^4$, we need to find $x^{13}$ in the given expression so that we can get $x^{17}$
using generating function
$1/(1-x) = 1 + x + x^2 + x^3 + ..........................................\mathit{eq.1}$
$x^6/(1-x) = x^6 + x^7 + x^8 + x^9 + ..........................................\mathit{eq.2}$
$\mathit{eq.1} - \mathit{eq.2}$
$1-x^6 / 1-x = 1 + x + x^2 + x^3 + x^4 + x^5$
$(1-x^6 / 1-x)^4$
$(1-x^6 )^4 *(1-x)^{-4}$
$(1-x^6 )^4 = \sum_{r \geq 0} (-1)^r \binom{4}{r} x^{6r}$ $[ {x^{6}}^r = x^{6r}]$
using the negative binomial theorem,
$(1-x)^{-4} = \sum_{r \geq 0} \binom{4 + r -1}{r} x^r$
Thus, when we convolve the above two generating functions, the x^13 coefficient is
(0 , 13) , (1 , 7 ) and (2 , 1)
$\binom{4}{0}\binom{4 + 13 - 1}{13} - \binom{4}{1}\binom{4 + 7-1}{5} + \binom{4}{2}\binom{4+1-1}{1}$
$\binom{4}{0}\binom{16}{13} - \binom{4}{1}\binom{10}{7} + \binom{4}{2}\binom{5}{1}$
$560 - 4*120 + 6*4$
$560 - 480 + 24$
$104$
