Indeed, it is frustrating and no fun at all…
This represents what I do not like with this game: hard is fine. I mean, I get angry at jumping puzzles or some dungeons fights, but at least, when you get the mechanics, it is fun, you can be proud of what you accomplished. Stupid is not fine, and no fun at all. With this joust, it is just plain luck, the guy does not have to follow the same rules we do. He can use his ability several times during the game, making him impossible to beat without any luck.
I would love to have a dev honestly comment about why they chose to have him not follow the same restrictions as we do, and to give us some pointers about the way to beat him if this was intentional…