During the race, people are always using swiftness, my guess is that they load up swiftness standing outside before they enter and the swiftness remains upon entry.
Furthermore, the challenges should put some kind of marker on your character to show that you’ve already done it once, preventing you from reentering it. People are dropping armors to reenter and win challenges specifically to kitten off achievement hunting players.
My suggestion to fix it, would be to enable some kind of boon strip upon entry of the challenges, and enable a marker like you have for silverwastes to signify they have already partaken in a challenge this time and cannot partake in any more for roughly 10-20 minutes?
Thank you for your time.