Given:
$T(n)=3T(n/5)+(\log n)^2$
Using the Master Theorem:
$a=3$, $b=5$, and $f(n)=(\log n)^2$
Calculate:
$n^{\log_b a}=n^{\log_5 3}$
Since $(\log n)^2$ grows slower than $n^{\log_5 3}$, this is Master Theorem Case $1$.
Therefore:
$T(n)=\Theta(n^{\log_5 3})$
Answer: B $O(n^{\log_5 3})$