The passive swiftness while in town and in home instances is bugged since latest update.
When loaded into a new town instance, the passive swiftness 33% movement speed from (Mastery) works, however when any swiftness (Boon) from any source is applied to your character, it will overwrite the swiftness (Mastery). When the swiftness (Boon) ends, you will permanently run at the regular speed while only keeping the swiftness visual effects from Leader of the Pact III (Mastery).
tl;dr Don’t stack swiftness (boon) with pact commander III (mastery) while in town or you will move at a regular pace again due to a bug that needs a fix.