Hi Jim,
I agree, but it does depend on what one is testing. Testing for field curvature, or chromatic aberration correction, spherical aberrations, or lens mount issues, can be done with multiple targets that cover the full field of view for the camera in a flat plane. That still does not exclude the possibility of a non-parallel orientation of the sensor/image-plane to the subject-plane (and even the sensor can be mounted slightly non-perpendicular to the optical axis, and it's usually very slightly rotated as well, within mounting tolerances).
So to test for lens performance, I prefer to test the best possible resolution in all corners separately, because that takes away all other variables. In a well-centered lens, the corners should resolve virtually identical on an approximately parallel image-plane / subject plane setup. If not, then there is already a lens issue. That's before we even consider the other possible issues. But if we do not first isolate the lens decentering we could be looking at several issues combined into one result.
Cheers,
Bart
P.S. Also, the use of high contrast bi-tonal features, leads to exacerbated aliasing artifacts on discrete sampling devices such as our sensors. That is then potentially worsened by Bayer CFA demosaicing that causes false color artifacts because the input signal was not properly high-pass filtered.
P.P.S. The ISO has added the sinusoidal grating version of the Siemens star to their resolution test standards (ISO 12233). Whether that had anything to do with my discussions with Norman Koren at the time, I do not know, but he did add (sinusoidally-modulated) Siemens star charts to his Imatest software after I suggested it to him, shortly thereafter followed by the ISO embracing it as an alternative to the earlier slanted edge test.
Bart, thanks for your insights. The Siemens Star test that I am developing -- and it is still a work in progress -- is an attempt to come up with a lens-screening test that can be performed by someone not skilled in the art of lens testing. The project stems from my belief that many many perfectly good lenses are returned as defective due to poor testing on the part of their would-be owners. Many folks were reporting de-centered lenses, so I took the OOF PSF test advocated by Prof Hank at DPR and tried to make it foolproof. Many people have been successful with it, but not everyone, including the OP in this thread. Every time someone has trouble with the test, I revise the instructions to plug that hole.
But it turns out that most all the lenses that people call decentered aren't. Of the ones that aren't perfectly fine, they seem to mostly suffer from field tilt, astigmatism, and/or local softness. The Siemens Star test is designed to find those things, even in the presence of mild symmetric focus curvature (which, I believe, is most often the result of design decisions rather than poor assembly.
I started out by rejecting all tests that require precise alignment because it's very difficult to do right (I have trouble myself sometimes), it can yield convincing-looking false positives, and I have found that almost no one is willing to do the checks that will detect it.
Your point about the effects of the step function edges in the Siemens Star is wel-taken. I will do some work with an anti-aliased star. However, at this point, I find the pronounced aliasing and false color useful. It is a test for focus error. It is a very fine indicator of astigmatism and coma. Now that I bring up coma, I'm not too happy that the Siemens Star is so good at detecting that, since I believe that it, too is mostly designed in to the lens. But I could be wrong.
With the help of people who I encounter on the Internet, I will continue to try to refine this test. I have already produced charts that tell you how far away you have to be as a function of lens focal length, sensor size, aperture, and allowed CoC. Turns out it is pretty darned far for many lenses and sensors. That in itself may be a problem in the acceptance of the test. We shall see.
I welcome your further comments; I have learned much from you.
Jim