State of the art camera profiling software?

[torger]

I'm continuing my project to drill deeper into camera profiling and is now trying to find out what the "current best" in camera profiling is.

It seems like camera profiling is a narrow genre with not so much software available at all, and there's a lot of old half-dead software out there too. I simply doesn't look like a vital software genre.

First let's sum up raw converters that support camera profiling:
* Adobe Lightroom (DNG Profiles = DCP)
* Phase One Capture One (ICC profile)
* Hasselblad Phocus (ICC profile)
* DxO Optics (ICC profile)
* Iridient Developer (ICC profile, seems also to have limited DNG profile support)
* Raw Photo Processor (ICC profile)
* RawTherapee (DNG profile and ICC profile)
* Corel Aftershot Pro (previously called Bibble, ICC profile)
* Photo Ninja (built-in format(?), has profiling built-in)

When it comes to ICC profiles it varies between software how the raw converters want them, Capture One wants ~1.8 gamma in them for example, but the profiling workflow usually fixes that automatically. However it seems like some ICC camera profiling software only supports a subset of the ICC raw converters. The DNG profile format has the advantage of being more well-defined.

Most profiling software seems to be sold together with a test target, like QPCard, Coloreyes, X-Rite.

When it comes it comes to DNG profiles (DCP), there's:
* Adobe DNG Profile Editor which only support MacBeth 24 patch colorchecker
* X-Rite has bundled software with their test targets, 24 patch colorchecker and the larger digital colorchecker SG (~96 patches)
* QPCard 35 patches with bundled software QPCalibration
* Datacolor SpyderCheckr 48 patches bundled software

For ICC (many same as the above as many support both DNG and ICC):
* X-Rite Profile Maker 5 (discontinued, ie interestingly enough X-Rite only supports DNG profiles nowadays)
* Datacolor SpyderCheckr, only Hasselblad Phocus
* Integrated Color Coloreyes, bundled with a 500 patch target
* QPCard
* Argyll, free open source supports most commercial targets and also custom-made

Photo Ninja raw converter seems to be a special case with built-in camera profiling support for the X-rite targets both the small 24 patch and the larger SG target.

Strangely enough it seems like today there's only Argyll that supports custom third-party targets, and there's no DNG profile maker that does it. Is this really the case or has I missed something? Lasersoft is selling a target without software and I doubt that they would if Argyll would be the only alternative to use it. Well, there's X-Rite Ipublish Pro2 which supports IT8 targets from Lasersoft (and Wolf Faust), but afaik it's the transparent targets for scanners rather than the reflective for cameras. For cameras it semes like X-rite only supports DNG and their own targets. Please correct me if wrong. X-rite's web site is a disaster and it's almost impossible to find any information of how their products work.

Only looking at the specs it seems like Coloreyes is the most serious product with it's bundled 500 individually measured target, but it can't do DNG profiles. For DNG profiles the current best seems to be X-rite Colorchecker SG. With Argyll you can use custom targets and most of the commercial ones, the weakness is that it's harder to use and supports only ICC and only "reproduction" type of profiles, ie there's no support for doing subjective hand-tuning like most(?) commercial profiling software do.

What I mainly miss here is DNG profile software that can do custom targets.

Another question is how valuable it really is to have many patches, or if a 24 patch profile will be as good. A limitation with test targets is that only a small gamut of what the camera can capture will be tested. A reflective target can't do very dark colors or very saturated colors, so even if you have a massive amount of patches you will sample only a smaller part of the camera's gamut. If you're going to do reproduction work of artwork it won't be a problem as the artwork will have limited gamut too, but for an all-around profile it might be a problem. I guess camera manufacturers use monochromators or some other sort of active target to get around this issue, but there seems to be no commercially available calibration solution with monochromators or the new multi-channel LEDs (10-22 channels full-spectrum programmable).

Is there any established "truth" of what the best commercially available camera profiling product is?

I think there are two aspects, 1) how accurate profiles can be made (for reproduction work) and 2) how subjectively-tunable it is for generic profile making. For example one may want to have a profile without hue-twists to make it more predictable, one may want to increase color separation in a certain color range, or reduce it, or one may want to add more saturation to the shadows etc.

And please add to the lists above (or correct if you find any errors)

[torger]

An additional question, anyone who knows how camera manufacturers does it in-house? With a monochromator you could measure the CFA response, like the examples shown here: https://spectralestimation.wordpress.com/data/

If you have the CFA response you could get CIE XYZ coordinates for them using the color matching functions, and voila you would have a matrix profile, but that would still only be approximate in practice as matrix-only profiles are. Maybe there's some better way to derive camera color from the CFA response? Or perhaps you would still combine the monochromator result with a reflective test target?

I think it would be wonderful if there was a color model where the only thing you would need to do is to measure the CFA response curves and then you could design any profile you'd like from that. I don't know if it's possible though.

A monochromator is not a cheap product though, but you can get a 300-800nm mini-chrom for about $1200... so if one could make high end stuff with it I would consider getting one, but I guess there's no software.

[digitaldog]

First things first. Is the raw converter going to expect a DNG camera profile or ICC profile? HUGE differences here both in terms of the sanity of moving forward, how the profile is applied and on what data (hugely a factor) then you can go down the ICC camera profile rabbit hole if necessary and start with targets and software.
TIP: this is potentially a time consuming and frustrating affair depending on what your answer for question 1.

[AlterEgo]

* Datacolor SpyderCheckr 48 patches bundled software

not again... they do not make dcp profiles, they make ACR/LR recipes (HSL tab)

[AlterEgo]

Is the raw converter going to expect a DNG camera profile or ICC profile?

code can be written to handle both

[AlterEgo]

* X-Rite has bundled software with their test targets, 24 patch colorchecker and the larger digital colorchecker SG (~96 patches)

XRite OEM software that makes dcp profiles only works with 24 patch target... or with subset of colorchecker SG representing 24 patches similar to xrite passport

[AlterEgo]

* Iridient Developer (ICC profile, seems also to have limited DNG profile support)

somebody published an exchange between him (somebody) and Brian Griffith about DNG support in Iridient (that was pre v3, but still) - search here...

[AlterEgo]

but there seems to be no commercially available calibration solution with monochromators or the new multi-channel LEDs (10-22 channels full-spectrum programmable).

argyll can do and it is not worse than commercial, no ?

[AlterEgo]

What I mainly miss here is DNG profile software that can do custom targets.

a decent software developer like you can write some kind of icc to dcp translator, granted it is not a replacement of full dcp functionality, but still.

[digitaldog]

code can be written to handle both

You're missing the point. One works really well, deals with raw scene referred data to do the job. The other doesn't. Kind of like the question: I have a DSLR, should I shoot raw or JPEG and you answering, well there are camera systems that allow both. Yes, but the differences in the approach and results is hugely different!

[AlterEgo]

You're missing the point. One works really well, deals with raw scene referred data to do the job. The other doesn't.

you are totally clueless... icc profiling can be done with raw data, that was pointed to you (yes, to you personally) many times in this forum - just use rawdigger in your workflow... and that is not to mention that argyll can work with monochromator data for example, while no publicly available dcp solution can... and you can do flatfielding again with raw digger for icc profiling workflow - you can't with dcp ones (publicly available... now I believe that all that Adobe has internally - but primitive tools for dcp profiling available to hoi polloi are what they are - primitive)

[digitaldog]

you are totally clueless...

As are the ICC apparently:
Using ICC profiles with digital camera images
http://www.color.org/whitepapers.xalter

Again to the OP, if you want to dig yourself into a color management rabbit hole, outside of copy work and very controlled shooting, you do want to look into ICC camera profiling. If you use a raw converter that supports DNG profiles, your life will be far easier.

[Tim Lookingbill]

An additional question, anyone who knows how camera manufacturers does it in-house? With a monochromator you could measure the CFA response, like the examples shown here: https://spectralestimation.wordpress.com/data/

If you have the CFA response you could get CIE XYZ coordinates for them using the color matching functions, and voila you would have a matrix profile, but that would still only be approximate in practice as matrix-only profiles are. Maybe there's some better way to derive camera color from the CFA response? Or perhaps you would still combine the monochromator result with a reflective test target?

I think it would be wonderful if there was a color model where the only thing you would need to do is to measure the CFA response curves and then you could design any profile you'd like from that. I don't know if it's possible though.

A monochromator is not a cheap product though, but you can get a 300-800nm mini-chrom for about $1200... so if one could make high end stuff with it I would consider getting one, but I guess there's no software.

Do you know the spectral response in (HSL) for the RGGB filters used to filter photons onto the sensor that produces a monochromatic (grayscale) response as voltage readings per each pixel site that has to be reconstructed and encoded by software?

Back in my illustration days when I was learning to use dyes and pigmented paint by visually examining the level of intensity each paint medium reflected back basic primary hues lit by daylight, I notice some were far more intense than others especially Doctor Martin dyes VS Windsor Newton watercolors.

I even asked Brian Griffith several years ago how he determined the intensity of the RGGB filters in order to build each ICC camera profile and he stated the majority of sensors (Sony?) out of Japan (engineers he spoke with) told him were close to NTSC color gamut. If you have a display that is exactly like NTSC gamut examine its individual full saturation of 255 Red, Green & Blue purities. Also examine gradients from full saturation to white and black, measure that and build a profile.

I tried to assign NTSC to a tiff converted to ProPhotoRGB out of Iridient Developer back then and it didn't work out so well. I gave up and just accepted what Adobe's DNG profiling did to my color which renders most of my images quite close to how it appears in front of my camera.

[torger]

The original way of applying an ICC profile was to convert to TIFF from the raw converter (using the raw converter's default color) and then handle color management outside the raw converter entirely. This is where the bad reputation of ICC profiles comes from.

However any raw converter with ICC support allows exporting a TIFF before color correction, ie what you get is the raw RGB channels with white balance applied, and possibly a curve. This is just as for DNG raw scene referred data, so there's not that big a difference as some seem to think. The DNG profile format is a lot easier to understand though, has floating point natively supported, and has more flexibility with dual illuminants and a pre-exposure and post-exposure LUT.

Writing an ICC to DNG Profile converter software is indeed possible and I'm thinking about doing that, but what you will have to do then is to specify which raw converter the ICC profile was made for and the profile converter (PC) must know how that raw converter pre-processes the image data before it puts it through the ICC profile, which is not the same for all raw converters. When knowing the PC would create a synthetic raw image with "all colors", pre-process it in the same way, put it through the ICC profile and save the result. Then it would run the same synthetic raw image through DNG pre-processing procedure and generate a DNG profile which produces as similar result as possible as the ICC profile. It's pretty massive work though, and seems to be a really long way to get a DNG profile, ie first profile using an ICC capable software like Capture One with Argyll and your custom target, and then convert that to a DNG profile to use in Lightroom.

[Tim Lookingbill]

The original way of applying an ICC profile was to convert to TIFF from the raw converter (using the raw converter's default color) and then handle color management outside the raw converter entirely. This is where the bad reputation of ICC profiles comes from.

However any raw converter with ICC support allows exporting a TIFF before color correction, ie what you get is the raw RGB channels with white balance applied, and possibly a curve. This is just as for DNG raw scene referred data, so there's not that big a difference as some seem to think. The DNG profile format is a lot easier to understand though, has floating point natively supported, and has more flexibility with dual illuminants and a pre-exposure and post-exposure LUT.

Writing an ICC to DNG Profile converter software is indeed possible and I'm thinking about doing that, but what you will have to do then is to specify which raw converter the ICC profile was made for and the profile converter (PC) must know how that raw converter pre-processes the image data before it puts it through the ICC profile, which is not the same for all raw converters. When knowing the PC would create a synthetic raw image with "all colors", pre-process it in the same way, put it through the ICC profile and save the result. Then it would run the same synthetic raw image through DNG pre-processing procedure and generate a DNG profile which produces as similar result as possible as the ICC profile. It's pretty massive work though, and seems to be a really long way to get a DNG profile, ie first profile using an ICC capable software like Capture One with Argyll and your custom target, and then convert that to a DNG profile to use in Lightroom.

Is there a question in there or are you just describing the hurdles of ICC vs DNG camera profiling no one seems to be concerned about? What are you trying to solve here? I haven't seen anything better and easier than what I'm currently using. I've put ICC camera profiling behind me because frankly I don't like the results it produces.

[torger]

If you have the CFA response you could get CIE XYZ coordinates for them using the color matching functions, and voila you would have a matrix profile

Hmm... I'm not sure if this is possible, I was thinking wrong when I wrote that stuff above. I tried to just make an integration, and sure that did not look good and of course it won't as I then just get the XYZ coordinates for the filter curve of red green and blue. I guess there should be some mathematical way to go from CFA curves to RGB primaries in XYZ coordinates, but I haven't figured it out. If someone knows how, please let me know :-).

I guess you could write software that runs through a large number of virtual test patches (each test patch a spectrum curve) against those filter curves and profile in the same way as you do with a physical test target, but here with any saturation you'd like. May that is how manufacturers do it?

[torger]

Is there a question in there or are you just describing the hurdles of ICC vs DNG camera profiling no one seems to be concerned about? What are you trying to solve here? I haven't seen anything better and easier than what I'm currently using. I've put ICC camera profiling behind me because frankly I don't like the results it produces.

No I was just describing the hurdles in there. As I'm writing an article about various aspects of camera profiling I'm drilling down as deep as I can into the subject. I've worked quite extensively with color management as a software developer, so I know quite a lot already, but still hungry for more :-). When it comes to camera profiling I've only made basic 24 patch color checker DNG profiles so far, and they've worked fine for my purposes but there's many unanswered questions. When I recently made printer profile experiments I came to the conclusion that at least 900 patches was sort of required for a high end profile. Can a camera profile be good with only 24 patches is then a question I'd like to investigate together with many more.

With ICC vs DNG it's much about which raw software you like. If you rather use Capture One than Lightroom and like to profile your camera you have to make an ICC profile because that is what Capture One supports. To make a complete article I must cover both DNG and ICC profiles.

[Tim Lookingbill]

Hmm... I'm not sure if this is possible, I was thinking wrong when I wrote that stuff above. I tried to just make an integration, and sure that did not look good and of course it won't as I then just get the XYZ coordinates for the filter curve of red green and blue. I guess there should be some mathematical way to go from CFA curves to RGB primaries in XYZ coordinates, but I haven't figured it out. If someone knows how, please let me know :-).

I guess you could write software that runs through a large number of virtual test patches (each test patch a spectrum curve) against those filter curves and profile in the same way as you do with a physical test target, but here with any saturation you'd like. May that is how manufacturers do it?

You left out the most influential aspect of your workflow and that being demosiaicing algrorithms which if you don't line 'em up right with your curves and matrices can skew the overall "petina" of the image making it look kinda' funky but in a way you can't put your finger on it.

Demosaicing is similar to nailing the angles determined to create the perfect rosette pattern in halftone dots on four color commercial press. Be slightly off in angle and the entire image's color cast shifts ever so slightly. I've seen it on press when I use to do color separations by hand for screenprint output. Screen printing 4 color process would really get screwed if the screen mesh fineness created a mosaiced pattern with t-shirt fabric pattern.

That's why I really respect demosaicing algorithm processes involved with the fine mesh sensor pattern reconstructed on my high rez display. Get it wrong, and you can forget about fixing it with curves and matrices.

[torger]

You left out the most influential aspect of your workflow and that being demosiaicing algrorithms which if you don't line 'em up right with your curves and matrices can skew the overall "petina" of the image making it look kinda' funky but in a way you can't put your finger on it.

I don't think demosaicing has that effect, and I have never seen it. In RawTherapee you can select 9 different demosaicers vastly different from each-other and they all produce the same color, *except* if you have an aliasing issue. A plain color woven fabric which provokes aliasing can probably in some circumstances cause different global color result depending on demosaicer. But any color in an area which is not plagued by aliasing will be rendered the same with different demosaicers.

In fact I think the DNG Profile Editor doesn't do demosaicing at all, as you can't zoom to 100% there it doesn't have to, it just takes blocks of 2x2 pixels (so you have all three channels represented) and produce a quarter size image from that which is used for profiling.

[Tim Lookingbill]

When I recently made printer profile experiments I came to the conclusion that at least 900 patches was sort of required for a high end profile. Can a camera profile be good with only 24 patches is then a question I'd like to investigate together with many more.

I'ld have to say the 900 patches method is a step in the right direction. I've come across discussions online where photographers complain that the 24 patch CCchart just isn't enough to account for properly capturing near full saturation objects under various lighting intensities.

But how to go about doing that appears to be easier dealt with on an image by image basis using the HSL panel that come with most Raw converters. That's what I've been having to do.

Lot of photographers like to shoot neon lights at night and think a camera profile will fix the HSL errors that they think is causing the posterization in the hue transitions. I think they're asking too much from a consumer level camera system. Build a profile using a CCchart made of neon lights. Yeah, that's gonna' work I'm sure.