If a the border population of a world greatly outnumbers that of the others, here are the downsides:
1) The larger pop world has very few challenging (i.e. fun) fights
2) The larger pop world spends a lot of time without people to fight at all
3) The lower pop world feels frustrated and helpless
4) The lower pop world starts losing people from the border making it worse
5) The lower pop world cannot achieve even basic in-game goals
6) There is no meaningful way you can contribute to fixing this situation as an individual. If you switch realms to avoid it, you make matters worse overall.
The fix that has been suggested before is to implement the following rule:
When trying to join a border zone, if your world outnumbers the second highest border population by more than 15% (or some other proportion, to be decided), you will join the queue.
Downsides of this:
1) Larger pop world ends up with longer queues.
2) As an individual, your only real fix for this is to move worlds.
Surely the downsides of this are a lot less severe than those of the current situation? Particularly since Anet could easily facilitate the moving of worlds from high WvW pop to low WvW pop.