I think making the price on Custom PvP server payable with Guild Influence would solve the problem of no one (individually) wanting to pay for a singular server that everyone in the guild would want to use.
By this I mean, to make the Influence price significant enough to were only large guilds can afford it. In a large guild, I could see one person feeling jaded paying individually for such a mass of people using. Aka them individually paying a service they have such a small fraction of its usage.
This would require smaller guilds to pool together their Gems for the custom arena. This works out because smaller guilds tend be be closer and trust each other enough for this Gem pooling. The only way around the Influence price for smaller PvP guilds is to invite large amounts of people. This removes the exclusivity/prestige of that guilds name and floods their custom server with randoms.