Above diagram depicts the scenario of our Question.
The length $P$ of our point $(x,y)$ selected randomly in $XY$ plane, from origin is given by
$P=\sqrt{x^{2} + y^{2}}$
$P^{2} = x^{2} + y^{2}$
Expected length of $P^{2}$ is given by
$E[P^{2}]=E[x^{2} + y^{2}]$
By linearity of expectation
$E[x^{2} + y^{2}]=E[x^{2}]+E[ y^{2}]$
Now we need to calculate the probability density function of $X$ and $Y.$
Since, distribution is Uniform
$X$ goes from $0$ to $1,$ so $PDF(x) = \frac{1}{1-0}=1$
$Y$ goes from $0$ to $2$ so $PDF(y) = \frac{1}{2-0}=\frac{1}{2}$
Now we evaluate
$E[X^2]= \int_{0}^{1}x^{2}.1dx = \frac{1}{3}$
$E[Y^2]=\int_{0}^{2}y^{2}\times(1/2)dy = \frac{4}{3}$
$E[P^2]=E[X^2] + E[Y^2]= \frac{5}{3}$
Correct Answer: $D$