On the Sea of Sorrow server I have experienced this bug many times. The most recent is when trying to join my party member in Lornar’s Pass. We joined and I travelled to this area. When we were both on the map I was unable to join.
Just figured this out. My friend was transfered to Sorrows Furnace (something he didn’t choose to do but both of us started at nearly the same time). Not sure why this happened but perhaps this has been happening to other people?