While the counter argument to this idea is usually “Walls can’t bleed”, neither can golems or half of the other non-organic things we fight.
It’s unfair to make certain tasks (like destroying the gate in CoF) nearly impossible to players who use condition damage as their main source of DPS.
This would obviously require balancing because 25 stacks of Vulnerability on a WvW gate would lead to some pretty ridiculous things. Maybe Walls and other structures can just be damaged by conditions, but aren’t affected by non-damaging conditions? Just an idea.