Reference:Solution of Arjun Sir
Case-I
$f(n)=n^{2} logn$
take $n=2^{10}$ So, $f(n)=2^{20} log2^{10}$
$f(n)=2^{20} $ *10 $
and $g(n)= n (logn)^{10}$
take $n=2^{10}$ So, $g(n)= 2^{10}(log2^{10})^{10}$
$g(n)= 2^{10}* 10^{10}$
Case-II
And now take one more value
take $n=2^{256}$
o, $f(n)=2^{512} log2^{256}$
$f(n)=2^{512} $ *256 $
and $g(n)= n (logn)^{10}$
take $n=2^{256}$ So, $g(n)= 2^{256}(log2^{256})^{10}$
$g(n)= 2^{256}* 256^{10}$
Now, main problem is how to check which one is bigger than other.
Just do one simple thing divide each other
like in case-I , if you'll divide f(n)/g(n)
f(n)/g(n) = $2^{20} $ *10 / $2^{10}$ * $10^{10}$
f(n)/g(n) = $2^{10} $ *10 / $10^{10}$
f(n)/g(n) = 10240 / $10^{10}$
So, here f(n)< g(n) , f(n) = O(g(n))
Now Case-II,
f(n)/g(n) = $2^{512} $ *256 / $2^{256}$ * $256^{10}$
f(n)/g(n) = $2^{256} $ *256 / $256^{10}$
Here, you can simply use Scientific calculator during GATE exam, and you will see quotient is >0.
So, here f(n)> g(n) , g(n) = O(f(n)) and you can find n where f(n) always greater than g(n)
So, Option B