fork(), copies the current process image, and creates a child process which starts executing from the next instruction onward. fork() returns the child process id to the parent and $0$ to the child process.
In the given code, $i$ and $j$ loop execute n times. But the if condition will be true for in total $0+1+2+\ldots n-1 = n(n-1)/2$ times. For each of these, the fork() ensures the number of processes doubles.
For $n=5$ we'll have $5*4/2 = 10$ times the if condition gets TRUE and so, the number of processes created will be $2^{10} = 1024.$ But out of this $1$ is the parent process and total number of child processes $=1023.$