Thats because of the emulation. Compare a computer processor to that of the iPhone 4. Computer is gonna win. N64 games are a common culprit to this framerate issue, and Ive seen it happen on older computers too.
So basically, theres just not enough resources in the phone to keep up with what it takes to run a n64 game