getbits(x,p,n) works like get n bit field of x from position p
Consider getbits(4,3,2)
Here, x = 4, p = 3 and n = 2
It is also mentioned that bit position 0 is at the right end.
x = 0000 0100
So, by definition it should return 2 bits value from position 3 i.e. 01 whose value is 1.
Now, let's perform
return (x >> (p+1-n)) & ~(~0 << n);
p + 1 - n = 3 + 1 - 2 = 2
So, x is right shifted by 2. Hence x becomes 0000 0001 ....(1)
~0 = 1111 1111 is left shifted by n i.e. 2
(~0 << n) = 1111 1100
and finally ~(~0 << n) = 0000 0011 ....(2)
Performing AND operation of (1) and (2) , we get 0000 0001 whose value is 1.
Basically ~(~0 << n) is acting like a mask of all 1's to get the right adjusted n bit field from position p of x.