Good one. You told everything perfectly and the only question remaining is "why can't we write to some random location in memory?". Well, that is due to OS protection. Suppose we write to a random location and changes the memory content of some other process, that can crash rt? So, OS allows a process to modify only the memory allowed by it- in Segmented memory managed systems, a process can only access the memory in its assigned segments and when it accesses a memory outside that it gets segmentation fault.
In a uniprocess system this protection is not needed. One of the main uses of virtual memory in addition to extension of memory address space is this memory protection.