There are 2 brothers and 6 different students. Firstly, the 6 students can be seated at a round table in (6-1)!=5! ways. Name the boys as B1, B2, B3, B4, B5, B6 in clockwise or anticlockwise order. Assume that between every 2 boys(say B1-B2 or B2-B3, or B3-B4, etc), there is a vacant seat which can be occupied by either of the brothers so that they don't seat next to each other. Following this, the total no of seats=6.
We can now select 2 seats out of 6 in 6C2 ways and both the brothers can arrange themselves in 2! ways.
Final answer=5!*6C2*2!=3600