One of the biggest gripes is that there is a summoning cast time on phantasms, a lot of the time, the mob dies while you finish summoning, and pretty much have to wait a full CD.
Not to call you out, but that is part of knowing when to use your skills. If the target is almost dead, then it does not make sense to cast a phantasm.