I can confirm that this happens. I have a brand new PC, Intel I7 4770k (3.5 Ghz) with a GTX 770 Ltd (4 GB Memory) and a 750 Watt power supply. There is definitely no issue with cooling or low power.
I run the game with every setting cranked/turned on as high as it can go (with the exception of Vsync as I don’t need it) and I usually run between 90 and 140 FPS. I run at 50 while fighting Maw or Shadow Behemoth, which are very, very, graphic intense.
It is quite disturbing to see it drop to 23 FPS when looking towards the gate in CoF (I have noticed that it starts when the tall spire thing in the back ground comes up). The only thing I can think of is that the game is trying to render as high quality as possible, regardless of the LOD distance. Normally if there is a lot of detail far away, turning the LOD down will help (Lion’s Arch), but not here.