Low In-game Performance On Gtx 580 GXT 1536 Mb?

im playing Cities XL right now , and something very annoying is going on .( by the way happened in a couple of other games too ) but .. whenever im starting a new game , everything goes smooth as it should , but after a couple of neighbourhoods im building (lots of objects to render) , it becomes impossible to play ! fps drops to something like 2019 .and i seen the benchmarks for this game a year ago , the best performance was on gtx 480 ! with 40 fps Min ! the weirdest thing is , that when im exiting the game , my gpu fan works as in idle mode( i can hear the difference , in BF3 it's god damn loud as Jet ) , and the gpu mhz usage is really low , almost as when im just browsing my PC .