From main function, square function is called with two arguments => square(&number, &number);
Hence, the second definition of square will be overloaded. And it has:
*x= (*x) * --(*y); where *x = 30 and *y = 30
-- (decrement) operator has the higher precedence than * (multiplication).
hence,
*x = 30 * 29
*x = 870
And the call to square function was a call by reference. Address of parameters was passed. So the value of number will be changed. And output will be 870.
And ofcourse, the line in first definition of square is typed wrongly. It should be:
*x= (*x)++ * (*x); // In exam paper, it was correct.