Actually this is a question of multinomial theorem .The problem is similar to dice sum problem.The problem can be formulated mathematically as :
x1 + x2 + x3 = 13 where each of the terms 0 <= xi <= 5 for i from 1 to 3..
So its solution is given by coefficient of xr in (1 + x + x2 + .................xk) n .
Here k = 5 , r = 13 and n = 3
So coefficient of x13 in (1 + x ... + x5)3 ,Now the inner term is :
1 + x ..+ x5 which is a G.P. which is = (1 - x6) / (1-x)
So the given term (1 + x ... + x5)3 can be rewritten as : (1 - x6)3 . (1 - x)-3
= (1 - 3x6 + 3x12 - x18).( 1 - x)-3
Now since x18 is already larger than x13 , so we ignore it .Here we have to remember the result:
Coefficient of xr in (1 - x)-n is given by n-1+r Cr.So
a) Taking 1 from the 1st term , we require r = 13 and n is 3 in this case.Therefore coefficient of x13 due to this term is given by :
3-1+13C13 = 15C13 = 15.14/2 = 105
b) Now taking -3x6 as the term , we need r = 7 and n = 3 .So the coefficient of x13 due to this term is given by :
(-3) . 3-1+7C7 = (-3) . 9C7 = -3 . 36 = -108
c) b) Now taking 3x12 as the term , we need r = 1 and n = 3 .So the coefficient of x13 due to this term is given by :
(3) . 3-1+1C1 = (3) . 3C1 = 3 . 3 = 9
So combining a), b) and c) , we get number of required solutions as : 105 - 108 + 9 = 6