$\underline{\textbf{Answer: A}}\Rightarrow$
$\underline{\textbf{Explanation:}}\Rightarrow$
When an equilateral triangle is inscribed in a circle then the radius of the circle is given by:
$r = \frac{b}{\sqrt3}$
When a square is inscribed in a circle its diagonal = $\sqrt a$ (Since, side = $a$, hence just use the Pythagoras theorem)
Now, equating both the radius, we get:
$\frac{b}{\sqrt{3}} = \frac{a}{\sqrt 2} = \frac{a}{b} = \sqrt\frac{2}{3}$
Therefore, the correct answer is option $\textbf A$.