Make score scale by percentage of total WvW population and whats ration between servers.
When servers are full and each server would have 1/3 share of total number of players in WvW then each server would get score like it is now. But when WvW servers are for example 5% full (night/mornings) each server would only get 5% of points they normally do. This would reduce impact by small group of players during night/morning, but they could still have small impact.
Also each server would get bonus for points if they have less than 1/3 of total number of players in WvW. Example if server 1 got 70%, server 2 got 20% and server 3 got 10% of total number of players in WvW then server 2 and 3 would get bonus points for their score.