Step 1: Find probability of getting sum as 5. It is 4/36 =1/9. Name it P(F).
Step 2: Find probability of getting sum as 5 or 8. It is 9/36=1/4.
Step 3: Find probability of not getting sum as 5 or 8. 1-1/4=3/4. Name if P(N).
Step 4:
You want to get sum=5 before sum=8.
In the 1st trial if you get sum as 5 then it's success. P(F)
If you don't then you should also ensure that sum=8 also doesn't appear in the 1st trial. So P(N)*P(F).
If you don't get 5 in the 2nd trial also then 8 should also not come in the 1st two trials. So P(N)*P(N)*P(F)
Keep doing this and add them up. You will get a gp series.
1/9+ 3/4*1/9+ 3/4*3/4*1/9...
Solve it to get 0.44.