If your graphic card is an OC version, try set it to normal core and ram settings as it’s specified by it’s manufacturer. I’ve had the same problem and it helped me to maintain higher frame rates than it was OC’ed.
The same issue might happen when you have OC’ed CPU.
Good luck!