I was having the same annoying problem, too. I could get him down to 25% then he would go invulnerable and swim back to his spawn point— unable to touch him again until he was 100% health, again.
But I was able to beat the boss by pulling him away from the shipwreck and dancing around him.
PS Nobody in the crew was fighting the boss, just me. As if they were just there to watch…