$x_1 + x_2 + x_3 = 17$, $\ \ \ \ \ x_1<6, \ x_3 >5 $
$[x^{17}] \left ( x^0 + x^1+ x^2 + x^3 + x^4 + x^5 \right )\left ( x^0 + x^1 + x^2 + x^3 ...... \right )\left ( x^6 + x^7 + x^8 +.......... \right )$
$[x^{17}] \left ( x^0 + x^1+ x^2 + x^3 + x^4 + x^5 \right )\left ( x^0 + x^1 + x^2 + x^3 ...... \right ) x^6 \left ( x^0 + x^1 + x^2 + x^3 ...... \right )$
$[x^{11}] \left ( x^0 + x^1+ x^2 + x^3 + x^4 + x^5 \right )\left ( x^0 + x^1 + x^2 + x^3 ...... \right ) \left ( x^0 + x^1 + x^2 + x^3 .......\right )$
So now equation became,
$x_1 + x_2 + x_3 = 11, \ \ \ x_1<6 , x_2 \geq 0, x_3 \geq 0 $
First, we'll find out the number of invalid integral solutions,
let $x_1 \geq 6$
$x_1 + 6 + x_2 + x_3 = 11$
$x_1 + x_2 + x_3 = 5$
Number of invalid integral solutions = $\large \binom{7}{2} = 21$
Total possible solutions of $x_1 + x_2 + x_3 = 11$
= $\large \binom{13}{2} = 78$
Total number of valid integral solutions = $78 - 21 = 57$
Alternative approach by Generating Functions :-
$[x^{11}] \color{red}{\left ( x^0 + x^1+ x^2 + x^3 + x^4 + x^5 \right )} \color{green}{\left ( x^0 + x^1 + x^2 + x^3 ...... \right )} \color{blue}{\left ( x^0 + x^1 + x^2 + x^3 .......\right )}$
= $[x^{11}] \color{red}{\left ( x^0 + x^1+ x^2 + x^3 + x^4 + x^5 \right )} \color{green}{\left ( x^0 + x^1 + x^2 + x^3 ...... \right )^2} $
= $[x^{11}] \color{red}{\left ( \frac{1-x^6}{1-x} \right )} \color{green}{\left ( \frac{1}{1-x} \right )^2} $
= $[x^{11}] \color{red}{\left ( {1-x^6} \right )} \color{green}{\left ( \frac{1}{1-x} \right )^3} $
= $[x^{11}] \color{red}{\left ( {1-x^6} \right )} \color{green}{\left ( {1-x} \right )^{-3}} $
when will we get $x^{11}$ ?
when 1 is choose in red part we need $x^{11}$ coefficient in green part ==> $\color{red}{1}.\color{green}{\binom{-3}{11}}$
when $x^6$ is choose in red part we need $x^{5}$ coefficient in green part ==> $\color{red}{-1}.\color{green}{\binom{-3}{5}}$
$[x^{11}] \text{coeficient in} \color{red}{\left ( {1-x^6} \right )} \color{green}{\left ( {1-x} \right )^{-3}} $ = $\color{red}{1}.\color{green}{\binom{-3}{11}} + \color{red}{-1}.\color{green}{\binom{-3}{5}}$ = 57.
https://gateoverflow.in/300829/madeeasy-test-series?show=300901#a300901
