Using DCamProf (
http://www.ludd.ltu.se/~torger/dcamprof.html) I've through spectral simulations virtually evaluated a number of real camera profiling targets. It works like this: by having measured spectral reflectance of the targets and the spectral sensitivity function of the camera, as well as the light source the whole profiling process can be completed without a single shot taken. The generated profiles can then be evaluated against real spectra to see how well they match.
The following targets have been evaulated:
* cc24: X-Rite Colorchecker 24
* ccsg: X-Rite Colorchecker Digital SG
* it8: Reflective IT-8 on photo paper by Wolf Faust
http://www.targets.coloraid.de/* qpc202: QP-Card 202
* pixma: homemade target printed with a Canon Pixma Pro-1 pigment inkjet on OBA-free semi-gloss baryta paper
The following real spectral data has been used for evaluating performance
* nordic-nature: leaves, flowers, etc natural colors from nature in Sweden/Finland
* lippmann2000: skin, hair, lips
Cameras used: Canon 5Dmk2 and Nikon D3x
Reference light: D50 (ie 5000K daylight)
Cameras have limited color matching ability so for reference I've generated a profile using lippman2000 + munsell colors to fill out, and same for nordic nature. These server as a baseline reference to show a limit of how good it can get. For both cameras I get below 1.0 DE at p90 with a max below 2 DE for nature and below 1 max for humans, ie very good match for an ideally designed profile. The human color set is somewhat simpler to match as it does not contain any high saturation colors, while the nature set has some saturated flower colors in it.
Comments about the target spectra:
cc24/ccsg patches has smooth spectra without any obvious colorant limitations. It8 while a photographic target has quite smooth and varied spectra, covers a large gamut, spectral variation looks better than I thought a photographic target would. QPC202 spectra is smooth but more similar to an inkjet print. The Pixma print has largest gamut but also have some limitations in spectral variation due to the limited number of colorants, but most likely considerably better than an ordinary CMYK print as there are more inks.
On to matching performance:
Both cameras show very similar results after profiling (which is expected), so to shorten the presentation I've only used numbers from the 5D mark II. The matching is with 2.5D LUT. The numbers are average, median, 90th percentile and max (ie worst), DE values are CIEDE2000, and the LCH is the error split in Luminance, Chroma (=saturation) and Hue. Generally speaking hue error is worst, luminance error the least.
NORDIC-NATURE matching:cc24
avg DE 0.75, DE LCh 0.36 0.39 0.41
mdn DE 0.68, DE LCh 0.34 0.35 0.27
p90 DE 1.27, DE LCh 0.69 0.74 0.92
max DE 3.59, DE LCh 1.58 1.85 3.28
ccsg
avg DE 0.64, DE LCh 0.25 0.32 0.40
mdn DE 0.51, DE LCh 0.19 0.28 0.27
p90 DE 1.26, DE LCh 0.51 0.63 1.03
max DE 2.95, DE LCh 1.99 1.68 2.21
it8
avg DE 0.91, DE LCh 0.35 0.51 0.54
mdn DE 0.86, DE LCh 0.31 0.47 0.33
p90 DE 1.59, DE LCh 0.75 1.00 1.20
max DE 4.32, DE LCh 1.85 2.44 3.43
qpc202
avg DE 0.71, DE LCh 0.29 0.32 0.46
mdn DE 0.62, DE LCh 0.23 0.25 0.35
p90 DE 1.23, DE LCh 0.63 0.70 1.00
max DE 3.72, DE LCh 1.72 1.49 3.35
pixma
avg DE 1.22, DE LCh 0.54 0.79 0.53
mdn DE 1.16, DE LCh 0.53 0.72 0.36
p90 DE 2.10, DE LCh 1.05 1.43 1.24
max DE 5.51, DE LCh 1.87 3.63 3.95
Here we see that the Pixma and IT8 target has some problems compared to the others, but not much and looking deeper into the stats we see that it's highly saturated colors, both has problem with a specific light vivid orange patch.
All the others, including the seemingly simplistic cc24 perform an equal level.
LIPPMANN2000 (skin/hair/lips) matching:cc24
avg DE 0.52, DE LCh 0.10 0.22 0.41
mdn DE 0.53, DE LCh 0.07 0.19 0.41
p90 DE 0.77, DE LCh 0.21 0.44 0.73
max DE 1.25, DE LCh 0.60 0.76 1.09
ccsg
avg DE 0.71, DE LCh 0.28 0.24 0.58
mdn DE 0.77, DE LCh 0.28 0.20 0.63
p90 DE 0.97, DE LCh 0.49 0.43 0.83
max DE 1.26, DE LCh 0.84 0.74 1.11
it8
avg DE 0.58, DE LCh 0.18 0.19 0.48
mdn DE 0.61, DE LCh 0.16 0.16 0.52
p90 DE 0.83, DE LCh 0.37 0.40 0.72
max DE 1.19, DE LCh 0.76 0.61 1.09
qpc202
avg DE 1.11, DE LCh 0.40 0.18 1.00
mdn DE 1.27, DE LCh 0.43 0.17 1.18
p90 DE 1.66, DE LCh 0.70 0.34 1.48
max DE 2.01, DE LCh 1.02 0.52 1.75
pixma
avg DE 0.97, DE LCh 0.17 0.55 0.74
mdn DE 1.10, DE LCh 0.12 0.55 0.82
p90 DE 1.22, DE LCh 0.40 0.84 1.04
max DE 1.50, DE LCh 0.91 1.03 1.35
Matching the human set, all targets do good. The ColorChecker SG has many "skintone" patches (foundation-tone patches I suppose) but it doesn't perform better than the CC24 still, and really when it's already below 1.0 you can make any real improvement. As long as the target is able to orient the camera into the skintone zone the the relative color differences within is handled well.
From this it looks like a CC24 is all you need, but isn't the supersaturated colors of the pixma target good at something? I haven't yet investigated thoroughly due to lack of spectral data but, I do suspect that it add some stability when dealing with subjects with super-saturated colors, for example what I see in running apparel when I shoot running competitions.
To just give some indication I generated artificial spectra along the pointer gamut border, that is a set with very saturated colors with smooth spectra:
POINTER BORDER GENERATED matching:cc24
avg DE 2.30, DE LCh 1.53 0.84 1.25
mdn DE 2.26, DE LCh 1.33 0.59 1.36
p90 DE 4.20, DE LCh 3.51 1.97 2.16
max DE 5.34, DE LCh 4.33 2.35 2.86
ccsg
avg DE 1.94, DE LCh 1.13 0.66 1.16
mdn DE 1.84, DE LCh 0.88 0.44 1.17
p90 DE 3.17, DE LCh 2.63 1.64 2.05
max DE 4.30, DE LCh 4.25 2.24 2.65
it8
avg DE 1.87, DE LCh 0.82 0.64 1.32
mdn DE 1.65, DE LCh 0.76 0.32 1.29
p90 DE 3.17, DE LCh 1.63 1.72 2.53
max DE 6.52, DE LCh 3.00 2.68 5.79
qpc202
avg DE 2.01, DE LCh 1.46 0.58 1.05
mdn DE 1.59, DE LCh 0.99 0.38 0.86
p90 DE 4.34, DE LCh 3.80 1.39 1.91
max DE 5.57, DE LCh 4.60 2.17 2.74
pixma
avg DE 1.70, DE LCh 0.90 0.78 0.88
mdn DE 1.20, DE LCh 0.87 0.22 0.54
p90 DE 4.25, DE LCh 1.50 2.72 2.63
max DE 7.12, DE LCh 1.97 5.75 4.20
combo cc24 + pixma
avg DE 1.26, DE LCh 0.75 0.39 0.70
mdn DE 1.12, DE LCh 0.77 0.18 0.46
p90 DE 2.22, DE LCh 1.52 1.25 1.90
max DE 4.60, DE LCh 1.68 2.56 3.75
Here we see that while having a quite bad max (which is not suprising due to colorant limitations), the pixma target has the best mean. The last one "combo" is a special one, where the cc24 is the primary target and then saturated colors are filled out with the pixma target. This is possible to do in a regular DCamProf workflow, just make sure you have stable setup and light, and shoot one shot with the cc24 and one with the pixma and then merge the two, letting the cc24 have priority (similar colors from the pixma is then excluded, but the supersaturated colors the cc24 lacks is included). A profile generated this way performs as the cc24 with some added boost on saturated colors.
I will personally test that combination more, getting the glossy Colorchecker Digital SG seems to be a waste of money, if you like me already have a printer and a spectrometer. And if you just have a cc24 and makes profiles with that, I think these simulations show that you should not really need to worry about the quality of those profiles, except possibly for supersaturated colors, cc24 is weakest in deep violet.
Note that this evaluation does not evaluate the performance of the bundled software, all profiles have been generated by DCamProf. The most interesting software to test I think would be QP-Card, which is supposed to have some novel technique to profiling, patented and all. The QP-Card does not contain skin color patches and such, but just patches which is supposed to be good at profiling the camera filters. I'm mildly skeptical but without trying the software I don't know if it can make a more accurate profile with the qpc202 chart than DCamProf can.
