(x+1)log(x 2+1) can be reduced to x log x 2 as addition of a constant term can be truncated.
Therefore, x log x 2 =2x log x = O(x log x)
f(x) = max ( complexity of (x+1)log(x 2+1) , complexity of x 2) = max ( O(x log x) , O( x 2) = O(x 2)
Answer : B
$O(x^2)$ is also $O(x^3)$ as per the definition of big-O. But such strict correctness is used for exams like GATE, not ISRO test.