Answer: $\mathbf A$
Explanation:
For equation $1$:
Let the $\mathbf {GP}$ be:
$\mathrm {A, AR, AR^2, AR^3},\dots$ (I choose capital letters because a and b already used in the question)
Sum of roots$ = x_1 + x_2 = \frac{-b}{a} = 3 \Rightarrow a + ar = 3 \Rightarrow a(1+r) = 3$ $\cdots \tag{1}$
Product of roots $= x_1.x_2 = \frac{c}{a} = a \cdots \tag{2}$
Similarly, for equation $2$:
Sum of roots $= x_3 + x_4 =12 \Rightarrow ar^2+ar^3 = ar^2(1+r) = 12 \cdots\tag{3}$
Product of roots $= x_3.x_4 = b \cdots \tag{4}$
Now, since all the roots are in $GP$
Dividing equation $1$ by $2$ we get:
$\frac{1}{r^2} = \frac{1}{4} \Rightarrow r = \pm2 \Rightarrow r = 2$
Now,
Substitute the value of r in equation 1,we get:
$A(1+R) = 3 \Rightarrow A(1+2) = 3 \Rightarrow A = 1$
So, $A = 1, R = 2$
From equation $2$
$x_1.x_2 = a \Rightarrow A.AR = a$
Now,
$x_1.x_2 = a \Rightarrow x_1.x_2 = a.ar = 1.1.2 = 2$
similarly,
$x_3.x_4 = b \Rightarrow ar^2.ar^3 = a^2r^5 = 1^2.2^5 = 32$
So, $ab = 2.32 = 64$
Hence, option $\mathbf A$ is the right answer.