It's clearly mentioned in the question that each page table occupies one page.
The usual way of solving is giving us an inner page table of size 4MB > PS. Thus, this approach is not asked in the question.
Hint: 1024 outer page table entries should point to 1024 inner page tables. Thus, outer page table size = 1024 entries * 4B entry size
And inner page table size = (1024 entries * 4B entry size) * 1024 such tables
Thus the overhead is : 1024 * 4B + 1024 * 4B * 1024 = 4 KB + 4096 KB = 4100 KB