Photoshop's Out of Gamut Masking for Printer Profiles - Details of operation

[Doug Gray]

First, a caveat. I am discussing only printer profiles here. Matrix based display profiles and working spaces such as sRGB or ProPhoto use different mechanisms for Out of Gamut determinations.

Printer profile Out of Gamut masking works in Photoshop by selecting a profile in View Proof then hitting Ctrl-Shift-Y to enable/disable masking. A neutral gray replaces all portions of the image that are out of gamut. This means that the image color can't be printed but that a less saturated color will be printed instead.

But there are problems with it. When does Photoshop consider a color to be out of gamut? How reliable is the indication? And what does it mean, if anything in the different printing modes of Perceptual, Relative, Saturation, and Absolute Intents?

I've closely approximated Photoshop's algorithm in Matlab by creating L*a*b* slices in increments of L:5 from 5 to 95. Then, the images were round tripped through the printer profile and Delta E's were generated. A close match on all slices with Photoshop occurred when the break between in gamut and out of gamut had a Delta E of 6. Printable colors with less than a Delta E of 6 were then compared to the requested color and their differences were added to a neutral Lab(50,0,0). This allows one to see the actual color deviation when printed close to but outside the gamut boundary while within the Delta E of 6. Close inspection of the Matlab gamut map at L=50 shows a magenta hue on the left boundary and green hue on the right. This shows the color error near the gamut edge prior to reaching the cutoff of 6 dE. Photoshop generates correct OOG masking at 6 dE only for Relative and Absolute Colorimetric Intents.

Unfortunately, Perceptual and Saturation Intents are not colorimetric intents so OOG masking has no clear meaning. Photoshop appears to simply deal with this by treating these as if they were Relative Colorimetric.  However, regular soft proofing with these intents works because, while the conversion to printer space is not defined colorimetrically, the colors shown (if within the display gamut) are colorimetric equivalents to the printed colors.

Attached are images of these Lab slices, processed by the Matlab program and a Photoshop masked view proof. There are fine grid lines at units of 10 along the a* and b*. Each pixel is at integer coordinates so the grid lines are spaced 10 pixels apart.

Here's things to keep in mind:

• Gamut masking is meaningful only in Relative and Absolute Intents.
• Gamut masking only occurs for colors that exceed 6 Delta E and that amount is easily visible but would provide no warning.
• Matrix based profiles use completely different algorithms. This only applies to LUT based output devices like printer profiles.

[Jack Hogan]

Good work Doug, quite interesting.

Jack

[bcnxavi]

I have always suspected that the softproof with Photoshop was from a high DE, but did not have the concrete data.

There should be some way to lower the DE, say to 2 or 3.

Good work Doug.

Thanks!

[bjanes]

First, a caveat. I am discussing only printer profiles here. Matrix based display profiles and working spaces such as sRGB or ProPhoto use different mechanisms for Out of Gamut determinations.

Printer profile Out of Gamut masking works in Photoshop by selecting a profile in View Proof then hitting Ctrl-Shift-Y to enable/disable masking. A neutral gray replaces all portions of the image that are out of gamut. This means that the image color can't be printed but that a less saturated color will be printed instead.

But there are problems with it. When does Photoshop consider a color to be out of gamut? How reliable is the indication? And what does it mean, if anything in the different printing modes of Perceptual, Relative, Saturation, and Absolute Intents?

I've closely approximated Photoshop's algorithm in Matlab by creating L*a*b* slices in increments of L:5 from 5 to 95. Then, the images were round tripped through the printer profile and Delta E's were generated. A close match on all slices with Photoshop occurred when the break between in gamut and out of gamut had a Delta E of 6. Printable colors with less than a Delta E of 6 were then compared to the requested color and their differences were added to a neutral Lab(50,0,0). This allows one to see the actual color deviation when printed close to but outside the gamut boundary while within the Delta E of 6. Close inspection of the Matlab gamut map at L=50 shows a magenta hue on the left boundary and green hue on the right. This shows the color error near the gamut edge prior to reaching the cutoff of 6 dE. Photoshop generates correct OOG masking at 6 dE only for Relative and Absolute Colorimetric Intents.

An excellent and very interesting post, Doug. Here is another look at this topic with tools available to me: ColotrhinkPro and Gamutvision, the latter of which is now freeware.

I selected an image of a colorful tulip which was taken with the Nikon D800e and rendered into ProPhotoRGB with Camera Raw using default parameters with no saturation boost. I then compared the gamut of the image with the gamut of my local Costco lab using their glossy paper profile for their Fuji Frontier printer. This is chromogenic paper (Fuji Crystal Archive) with a relatively narrow gamut.

This is the image (automatically converted to sRGB by the Smugmug software)



Here is the out of gamut indicator using Photoshop 2017.0.0


Here are the gamut comparisons using Colorthink after converting to colorlists and computing ΔEs, setting the lower range to 0-6 ΔE to conform with Doug's findings. The Photoshop out of gamut indicator does correspond to a delta E of greater than 6, consistent with Doug's findings.



Gamutvision is now a freeware program and it produces a pseudocolor gamut presentation using colors to show the out of gamut ΔEs. DeltaΔ2000 is not available in Gamutvision, so ΔE 94 was used here (see the Gamutvison docs for details).

Gamutvision is attractive since it can quickly produce the pseudocolor gamut map, whereas the process with ColorthinkPro is laborious and slow. One must first produce a color list for each image and then calculate the ΔEs, which is very slow even with a small image (267 x 400 pixels was used here).



Regards,

Bill

[Arlen]

Thanks for this info, Doug and Hans. It now seems clear that getting no out-of-gamut indicator in Photoshop's Gamut Warning does not mean that there is nothing to be concerned about for any particular image.

[bcnxavi]

bjanes, if I have not misunderstood, Gamutvision help in the chapter of read image for analysis says "The CIEDE2000 metrics replaced ΔE*CMC and ΔC*CMC in Gamutvision 1.3.7"

Then I think that we can use DE2000 in Gamutvision.

Thanks!

[bjanes]

bjanes, if I have not misunderstood, Gamutvision help in the chapter of read image for analysis says "The CIEDE2000 metrics replaced ΔE*CMC and ΔC*CMC in Gamutvision 1.3.7"

Then I think that we can use DE2000 in Gamutvision.

Thanks!

I have not had any luck showing DE2000 in Gamutvision. The dropdown menu does have an option for DeltaE 00, presumably DE2000, but when I select it nothing happens as shown below. This is on 64 bit Windows 2010 as shown below. If you have any luck, please let us know.



Bill

[Jack Hogan]

Nice Bill!

Jack
PS I sent you a PM.

[bcnxavi]

I have not had any luck showing DE2000 in Gamutvision. The dropdown menu does have an option for DeltaE 00, presumably DE2000, but when I select it nothing happens as shown below. This is on 64 bit Windows 2010 as shown below. If you have any luck, please let us know.



Bill

Here you have the Gamutvision copy using DeltaE-00, that I think is DeltaE 2000, according with his Help.

Windows 8.1 x64.

Thanks.

[Jack Hogan]

I can't talk about paper but thanks to Bill for sharing his raw capture I can contribute chromaticity diagrams for the fine tulip image.  Using an assumed illuminant of 5000K, say D50, and Adobe's DNG linear matrices, this is how the image appears in xy chromaticity space, one dot per pixel (sRGB, Adobe and ProPhoto RGB gamut outlines shown).



Note how some saturated yellows, oranges and reds fall outside of Adobe and sRGB's reach.  There are also a number of pixels outside of Helmholtz' horseshoe: those (few) pixels captured chromaticities outside of the visible range.  The following superimposes a histogram to get a rough idea of how many pixels are where in the diagram.  The counts represent the number of pixels of the given chromaticity within a .005x.005 xy square and are expressed in log10 units so blue represents less than 100 pixels per square, green less than 1000, brown a few thousand and yellow 10k or more.  Note the yellow 'lips' outside of Adobe RGB by the orange and red/purple chromaticities.



And here is where the pixels outside of Adobe RGB are in the image: the gray pixels are in gamut, the colored pixels are not.



You can see other examples of these plots here.
Jack

[Doug Gray]

One of the interesting things about the 6 dE Photoshop OOG clipping mask is that perceptually, it is only about 2.5 dEs.  Delta E 2000, which is a perceptually more accurate indication of color difference, gets compressed as the saturation increases. This was explored here:

http://forum.luminous-landscape.com/index.php?topic=111560.0

This shows that perceptually, 1 to 2 dE 1976 is very visible near the neutral tones where the dE 2000 is often larger than the dE 1976 but decreasingly so as the saturation, or distance from neutrals increases. So the dE 1976 of 6 from the Photoshop OOG mask is not as bad an indicator as one might think.