Locust Swarm
It’s worse in the Eternal Battlegrounds jumping puzzle. There is a point there where it starts hitting the rock that you are walking on (there is no health bar and it can’t be destroyed) which puts you into combat, more than negating the Swiftness.
On the other hand, it’s nice when your swarm fills your DS bar every time you run over a rabbit =)