FastRawViewer is performing Bayer demosaicking to display the image. It is very computational-intensive operation (chain of operations, in fact).
Couple of things worth looking at.
First, you can disable internal JPEG support in Preferences: RAW+JPEG: Ignore internal JPEGs.
Second, you can try different image resampling and downsampling options in Preferences: GPU Processing. Depending on the video card, it may speed things up.
One more thing: bottom bar contains file navigation buttons, and if you click on the file number, it allows to use Shift-ArrowUp/ArrowDown to jump through 5 files.
Can you please give more details: OS version you are running, is it 32 or 64 bits, what is the RAM size, what video card model is in the system, how much memory does video card have?