I also hit my head on enough WvWvW castle walls by now to get irritated enough with the game to post a suggestion on this forum.
My main WvWvW gripe is that the points simply aren’t accessible for my kind of player (explorer) on a server with a low population. If those points of interest would move outside the gate of fortresses and keeps (and vistas would be accessible without having access to closed-down structures), it would really help.