Thanks for your insight Damon. I was not aware of rapid photo downloader. Looks like a nice app and quite similar to mine in design goals. I also have a renaming feature, albeit quite rudimentary, which automatically creates a destination folder structure. I have a selection option, where you can filter the import to only the images you pick. I'm a wildlife photographer, and usually take many images of the same subject, hoping one or two will stand out. No point in importing all the loser shots.
Exiv2 is the go-to library for image metadata in C++. It's not as fast as single-purpose highly tuned custom code, of course, but given how long it has been around it will surely cover far more edge cases, such as where manufacturers have encoded metadata in non-standard, surprising and short-sighted ways. Having said that Exiftool is of course the gold standard for metadata coverage and reliability.
I tried Exiv2 but it had several problems. First, it was a royal pain getting the library to work on a PC. I never did get it to work with the 64 bit version of Qt. Second, it does not work with some unicode, and I have unicode in my metadata. Finally, this was an exercise for me to figure out how stuff works.
I haven't looked at your code Rory, but based on you mentioning having 4 threads, have you considered making the exif and thumbnail extraction an asynchronous operation after the file read? In other words, one thread that simply reads from disk and pretty much nothing else, and a small pool of threads that load the metadata and render the preview image in parallel to each other. That way you max out disk io, and take advantage of multiple cores to do the heavy rendering work. That's the approach my code takes. If all you're doing is extracting the jpeg preview (and not rendering from the RAW data), you can read from only the portion of the RAW file you actually need, i.e. the metadata you want and the jpeg preview itself. Unfortunately RAW files are not necessarily consistent in where they store that information in the file (some manufacturers are worse than others), but it can be made to work. Again that's what I do with my code.
Since reading the file is the most expensive operation, as you mention, I only read the parts I need: some metadata, the small thumbnail JPEG and the larger JPEG preview. I need to read the metadata first in order to find the small and larger JPEG data and their dimensions. I get thumbnails and the first couple of previews immediately, and then keep reading the rest to make the app as responsive as I can. Since all I am doing is displaying the JPEGs without any processing, that part goes pretty fast. I'll have to look at separating the read and the conversion to a bitmap - I had not thought about that. The only raw files I have worked on so far are NEF and CR2. If I get serious I'll likely have to go the third party route for metadata.
Also did you look at the FOSS code the folks behind FastRawViewer maintain? I'm not sure if their code to render actual RAW previews (not the preview) using the GPU is FOSS or not. It might be. They've certainly done a lot of work to take dcraw in new directions.
I'll check that out. To date I have not looked at other folks code for a couple of reasons. First I'm still learning and other code can be really hard to understand before you really know a language and the file structures - I've only been at this for 3 months and it has been one massive learning curve after another. Second, as I've mentioned, this has been a bit of a challenge for me to see if I can figure this stuff out for myself.
I do agree with John that we need to compare apples with apples. LR and other converters are doing much more than extracting a preview. They're also doing a lot of heavy processing work, e.g. adding sharpening.
Lets break this up into the import and the library preview functions.
At the import stage all they need to show is the embedded previews, which are presharpened based on your camera settings. It is pretty obvious to me that there is not any image precaching going on here. This appears to be completely apples to apples with what I am doing.
In the library I understand there is a lot of processing that occurs to
create the previews. However, after the previews have been built it should be an apples to apples comparison. However, Lr is not very smart at generating the previews I want when I want them. It is not very good at fast sequential image review. It is not very good at stopping when I do.
