Answer is (C) .
First let me explain what is counting semaphore & How it works. Counting semaphore gives count, i.e. no of processes that can be in Critical section at same time. Here value of S denotes that count. So suppose S = 3, we need to be able to have 3 processes in Critical section at max. Also when counting semaphore S has negative value we need to have Absolute value of S as no of processes waiting for critical section.
A & B are out of option, because Xb must be 1, otherwise our counting semaphore will get blocked without doing anything. Now consider options C & D.
Option D :-
Yb = 1, Xb = 1
Assume that initial value of S = 2. (At max 2 processes must be in Critical Section.)
We have 4 processes, P1, P2, P3 & P4.
P1 enters critical section , It calls P(s) , S = S - 1 = 1. As S > 1, we do not call Pb(Yb).
P2 enters critical section , It calls P(s) , S = S - 1 = 0. As S >0 we do not call Pb(Yb).
Now P3 comes, it should be blocked but when it calls P(s) , S = S - 1 = 0-1 = -1 As S < 0 ,Now we do call Pb(Yb). Still P3 enters into critical section & We do not get blocked as Yb's Initial value was 1.
This violates property of counting semaphore. S is now -1, & No process is waiting. Also we are allowing 1 more process than what counting semaphore permits.
If Yb would have been 0, P3 would have been blocked here & So Answer is (C).