Since I doubt that an image or two is filling the 200GB cache, there's probably more going on here ... for example, LR probably writes out your current image into the cache and then reads the next one in. So maybe 5 seconds to write the current one out and 3 seconds to read the next one in? An IQ180 is probably a 400-500MB TIFF file (uncompressed), so the delay you're experiencing would be about right if the size of the cached data is similar to that for a TIFF file.
But with 64GB of RAM, you'd expect that LR would be able to hold an image or two or three in RAM as a first layer cache...
Or maybe photographers in front of computers just don't operate in a manner that makes that worthwhile?
I think Jeff's case is interesting because it suggests that there is very little performance benefit from the cache - maybe one or two seconds.
The raw cache directory would be more important on systems where the CPU was slow enough to be the bottleneck during the raw conversion.
Jeff, I'd be interested to know if setting the cache size to (say) 2GB changed the times at all.
I suppose a further point is, if CPUs become fast enough and images large enough that raw conversion is not a significant factor in the image load time, is there any point in keeping a raw cache?
Similarly, I'm curious as to if Wayne used the SSD for the cache if it made any real difference as the times from moving between images suggest that it is disk read/write that is the limiting factor and supposedly the SSD should provide better throughput. Update: that is contrary to testing that has showed negligible performance benefit from using an SSD with Lightroom - see Can LR benefit from SSD?