To my knowledge this bug has been in game from the release, and probably been reported many times.
So, Phantasmal Haste -trait increases the attack speed of every other phantasm except Phantasmal Swordsman, it isn’t affected by this trait at all.
Come on, wouldn’t it be time to fix this already?