What is the best method for quantitatively comparing the low light performance of different cameras?
In my experience, the almost universally adopted comparison method is as follows. For each camera under consideration, a photograph is taken of the same subject, at the same ISO setting, and either the exposure or aperture is adjusted for correct exposure. A full-scale crop of the image is then examined, and the “poorer performing” camera will show a noisier, less distinct image.
But what, exactly, does this test tell us? I contend that this test does not fundamentally tell us anything at all about which camera is capable of producing the best quality image in low light conditions. As a case in point, one of the cameras might have a very much faster lens, which surely would make it capable of taking lower noise images, yet this test will give no credit for that. Food for thought.
Therefore, there are two points that would appear to warrant further discussion.
(1) Does the almost universally adopted comparison method, as described, fundamentally tell us anything about which camera is capable of taking higher quality (=lower noise) images under low light conditions?
(2) If this comparison method is flawed, then what better method should be adopted?
Firstly, I wish to make it clear that when I speak of a “camera”, I refer to the camera body with lens attached, as in the real world you can’t actually take a photo any other way.
I have read enough online camera reviews to know that the reviewers and audience alike place great significance in tests of the type described above, carefully examining image noise over the full range of ISO values, and thus concluding that one camera has better low light/image noise performance than another. Despite this, I have never read exactly what this test is supposed to tell us, or why. I maintain that this test/comparison method does not fundamentally tell us anything at all about which camera is the capable of producing the best quality image in low light conditions, because no credit is given for the speed of the lens.
A more meaningful method of comparing low light/image noise performance would be as follows:
(1) For each camera under test, a photograph is taken of the same subject. The camera-to-subject distance and field of view must be the same for each camera.
(2) Each camera under test must use the same exposure time. Clearly, any camera permitted a longer exposure would be at an unfair advantage.
(3) Each camera is set to maximum aperture, as you would do in the real world when you are “flat out” having enough light to get a decent photo. Thus, credit is given to camera(s) with faster lenses, as it should be.
(4) Correct exposure is obtained by varying the ISO setting.
(5) Noise reduction should either be turned off, or at least all the cameras under test should use a similar amount of NR so that the resolved detail in the final images is similar.
(6) A full-scale crop of the image is then examined, and the “poorer performing” camera will show a noisier image.
(7) If the cameras have different numbers of pixels, the cropped images that are examined for image noise should be rescaled so they all appear the same absolute size on the output device.
In simple terms, all the cameras under test are doing the best they can in poor light and under identical constraints, and they are judged on the visible quality of the image. That is the test that matters in the real world, isn’t it?
Does anyone know what the more common test/comparison method is supposed to show, where the image noise of competing cameras is compared at the same ISO setting?
To my way of thinking this is an important topic, as a large part of this forum is devoted to discussion of which camera is better than which other camera in terms of low light/image noise. Does anyone else out there have any thoughts?
