R and S are relation schemas. in book small characters represent relation instance such as r and s, while capital letters represent relation schema ( attributes of a relation).
R = {A, B, C} S ={C}
R - S = {A,B}
$\prod$R-S, S(r) means $\prod$ A,B,C(r)
PS: The R.A expression given in the question is for 'division' operation, generally used for "All".