When port numbers of two clients are same then in that case server will confuse to which process port to be used hence port number is not enough to distinguish.Only using IP address will create same problem. So by using port number along with IP address will solve this problem.
Port number+IP address=Socket
This is how we can say server needs destination address so that it can distinguish packets from particular client connection.
for example A and B are two clients having IP addresses Ia and Ib and server Ip address is Is,port number of A is x and Port number of B is also x, x=80,and server port number is also 80
so from A it is Ia:Is+x(80) and from B it is Ib:Is +x(80) so using source Ip address ,destination ip address ,source port number and destination port number we can distinguish that packets are coming from a particular client.