$x_1 + x_2 + x_3 + x_4 = 12 , \ \ \ 0< x_1 < 5, \ \ \ 0< x_2 < 5, \ \ \ 0< x_3 < 5, \ \ \ 0< x_4 < 5$
$[x^{12}] \ \ ( x + x^2 + x^3+x^4)( x + x^2 + x^3+x^4)( x + x^2 + x^3+x^4)( x + x^2 + x^3+x^4)$
$[x^{12}] \ \ x^4(1 + x + x^2 + x^3)(1 + x + x^2 + x^3)(1 + x + x^2 + x^3)(1 + x + x^2 + x^3)$
$[x^{8}] (1 + x + x^2 + x^3)(1 + x + x^2 + x^3)(1 + x + x^2 + x^3)(1 + x + x^2 + x^3)$
so now the equation is,
$x_1 + x_2 + x_3 + x_4 = 8, \ \ 0 \leq x_1 < 4, \ \ 0 \leq x_2 < 4, \ \ 0 \leq x_3 < 4, \ \ 0 \leq x_4 < 4$
Total valid solutions = Total solutions - invalid solutions
Total possible solutions for $x_1 + x_2 + x_3 + x_4 = 8$ are $\large \binom{11}{3}$$ = 165$
Now calculating invalid solutions,
When $x_1 \geq 4,$
$x_1 + 4 + x_2 + x_3 + x_4 = 8$
$x_1 + x_2 + x_3 + x_4 = 4$
total possible solutions = $\binom{7}{3} = 35$
similarly 35 solutions when $x_2 \geq 4$ or $x_3 \geq 4$ or $x_4 \geq 4$
when $x_1 \geq 4$ and $x_2 \geq 4,$
$x_1 + 4 + x_2 + 4 + x_3 + x_4 = 8$
$x_1 + x_2 + x_3 + x_4 = 0$
total possible solutions = $\binom{3}{3} = 1$
similarly for other combinations and there are total 6 combinations possible where $x_i \geq 4$ and $x_j \geq 4$
Now no 3 variables can be $\geq 4$
So total invalid solutions= $4 \times 35 - 6 \times 1 = 134$
Total valid solutions = Total solutions - invalid solutions
Total valid solutions $= 165 - 134 = 31$
$[x^{8}] (1 + x + x^2 + x^3)(1 + x + x^2 + x^3)(1 + x + x^2 + x^3)(1 + x + x^2 + x^3)$
$[x^8] \left ( \frac{1 - x^4}{1-x} \right )^4$
$[x^8] \left ( 1 - x^4 \right )^4 \left ( 1 - x \right )^{-4}$
$= \Large \binom{4}{0}\binom{-4}{8} + (-1)\binom{4}{1}\binom{-4}{4} + \binom{4}{2}\binom{-4}{0}$
$= \Large \binom{4}{0}\binom{11}{8} + (-1)\binom{4}{1}\binom{7}{4} + \binom{4}{2}\binom{3}{0}$
$ = 165 - 140 + 6 $
$ = \large 31$
