@jay rathod let us suppose B is correct than
Step1: X adds digital signature to M, which can be done using X's private key, say XPR. So now we have
<M,σ> as <M,(XPR,M)>
Step2: X Encrypts <M,σ> (after signing digitally, we are encrypting it to send) using public key of Y, so we have YPU(M,(XPR,M)) then send to Y.
Step 3: At y, Decrypt using X's public key. XPU(YPU(M,(XPR,M))). which doesn't seems to generate the plain text as order is not correct.
Now instead, If I would have chosen D, then ill first decrypt using Y's private key i.e YPR(XPU(YPU(M,(XPR,M))) which will give (M,(XPR,M)).
Step 4: Now apply X public key to verify if sender's identity is correct or not by extracting out the digital signature and retrieving M, i.e XPU(XPR,M)=M
So correct order of operations is in D.