DxO PhotoLab 2 working space tests

[Doug Gray]

The specs say straight line + power curve for all those spaces.

What "specs?"

ICC profiles (V2) allow for 3 variations:
1. A linear (gamma=1)
2. A pure gamma curve (no toe) which is what Adobe RGB (1998) uses
3. A TRC, which is a set of points that are interpolated, this is used by sRGB:  IEC 61966-2-1:1999

The latter has a linear ramp followed by a roughly, gamma = 2.4. The overall effect is similar to gamma=2.2.

Adobe RGB (1998) uses a pure gamma curve of 2.199.

[Jack Hogan]

What "specs?"

ICC profiles (V2) allow for 3 variations:
1. A linear (gamma=1)
2. A pure gamma curve (no toe) which is what Adobe RGB (1998) uses
3. A TRC, which is a set of points that are interpolated, this is used by sRGB:  IEC 61966-2-1:1999

The latter has a linear ramp followed by a roughly, gamma = 2.4. The overall effect is similar to gamma=2.2.

Adobe RGB (1998) uses a pure gamma curve of 2.199.

Specs are specs, practice is practice. Graeme knows, he is one of the cognoscenti.

Back home on Wednesday, in due time I'll be able to fire up PS/Matlab and see what comes out of it then

Jack

[Doug Gray]

Specs are specs, practice is practice. Graeme knows, he is one of the cognoscenti.

Yes, Graeme certainly is. He has done great work. He also corrected an error I made that I was quite certain about. I  wrongly believed the I1Pro 2 spectro read with a uV cut filter, then a uV LED and generated M0,M1,M2 profiles from those two measurements. This is the way i1iSis does things. But not the I1Pro 2 as Graeme pointed out, quite correctly, that it measured M0 (no uV cut), then did a uV LED measurement and generated the M's from those.

I found what may be the source for Adobe RGB (1998) generating a linear ramp which merges into a gamma curve. It's in an Adobe PDF document titled "Adobe RGB (1998) Color Image Encoding" here:

https://www.adobe.com/digitalimag/pdfs/AdobeRGB1998.pdf

At then end of the colorspace description, is this in Annex C:

Annex C. Implementation notes (Informative)

Slope limits in a color converter

Many color converters and products impose slope limits on gamma curves found in the rTRC, gTRC, and bTRC tags of ICC profiles. For an arbitrary slope limit of x (where x < 1), the effective gamma curve as used in the color converter has a slope of x or greater. When used with the Adobe RGB (1998) ICC profile, the slope limit should not be greater than 1/32. A slope limit of 1/32 affects 8-bit integer values 1 to 14. At the time of writing, the Adobe color conversion engine, ACE, included with Adobe Photoshop and other products from Adobe Systems, imposes a slope limit of 1/32. The effective inverse transfer function for Adobe RGB (1998) when used with ACE thus becomes:

-  see graphic equation in PDF -   my one line equiv:  max(pow(C,2.19921875), C/32) for C[0:1]

Note that the above slope limit is an implementation aspect, not an attribute of the Adobe RGB (1998) color space encoding. Different implementations may use different slope limits.

However, Adobe ACE in current versions of Photoshop do not do this. It's easy to test with the image posted above of all 64k 16 bit neutrals. Just assign Adobe RGB (1998) and convert to any linear RGB space using Photoshop's ACE. There is no slope limiting. Perhaps it did do this at the time the document was written. I've found no indication anyone has noticed that ACE doesn't now do what is stated in the pdf let alone when/if Adobe changed it.

Back home on Wednesday, in due time I'll be able to fire up PS/Matlab and see what comes out of it then

Jack

Jack,
I'm interested to see what background info you can dig up. Test my posted image and whatever else you come up with.

Should be interesting. I've found nothing on the web referring to Annex C and any discussion of what Adobe's ACE actually does in that regard. It may be that other imaging apps implement the discussed slope limiting, I have no idea,  but ACE does not. I find that rather odd.

[GWGill]

What "specs?"

The relevant colorspace specifications for all those spaces (i.e. the definitions I used when creating ICC profiles for them.)

I found what may be the source for Adobe RGB (1998) generating a linear ramp which merges into a gamma curve. It's in an Adobe PDF document titled "Adobe RGB (1998) Color Image Encoding" here:

I'd forgotten that Annex, probably because it applies to their CMM not the profile. Not mathematically the same as the usual straight line + power curve either, since a slope limit generates a smoothness discontinuity (i.e. the power curve is not offset).

[Doug Gray]

The relevant colorspace specifications for all those spaces (i.e. the definitions I used when creating ICC profiles for them.)

Are you saying you created Adobe RGB (1998)?  Then why is it using a pure gamma curve and Adobe ACE is not implementing a slope limit contrary to their spec's Annex C? Is Adobe now doing it wrong?

I'd forgotten that Annex, probably because it applies to their CMM not the profile. Not mathematically the same as the usual straight line + power curve either, since a slope limit generates a smoothness discontinuity (i.e. the power curve is not offset).

Well, since Photoshop's ACE neither currently implements Annex C nor does the Adobe RGB (1998) profile they install, which has a pure gamma of 2.19921875 without a ramp, is this something you are doing elsewhere?

Edit:
I looked at the rec2020 profile you made and see that it uses a TRC with a linear ramp, so I presume your other profiles do as well. Any idea why Adobe doesn't do that? Also ProPhoto RGB uses a pure gamma and ACE doesn't modify it. I assume some apps use ramps while others (Adobe) don't. Is that your understanding?

[Jack Hogan]

Any idea why Adobe doesn't do that? Also ProPhoto RGB uses a pure gamma and ACE doesn't modify it. I assume some apps use ramps while others (Adobe) don't. Is that your understanding?

I wouldn't assume PS 'doesn't do that'. It used to do it and I would be surprised if it didn't do it any more. There are good reasons why they do it.

The test is not PS to PS. ACE is well behaved, so if it does do whatever it probably does it throughout. The test is ACE vs a literal interpretation of the spec (e.g. Microsoft ICM or similar).

Jack

[Doug Gray]

I wouldn't assume PS 'doesn't do that'. It used to do it and I would be surprised if it didn't do it any more. There are good reasons why they do it.

The test is not PS to PS. ACE is well behaved, so if it does do whatever it probably does it throughout. The test is ACE vs a literal interpretation of the spec (e.g. Microsoft ICM or similar).

Jack

Microsoft ICM is not as well behaved as ACE and produces larger errors. Particularly with sRGB conversions.

A good approach is to compare a linear to Adobe RGB (or inverse) conversion in ACE to the same conversion in Matlab or any other good math package using a pure, floating point gamma and the two match within 33 parts per million. There is no ramp and the miniscule differences are because ACE uses scaled fix point, 16 bit (15 bits and a sign bit) math while Matlab uses a double precision pure gamma.

[GWGill]

Are you saying you created Adobe RGB (1998)?

I'm saying I created an ICC profile ("ClayRGB1998.icm") to implement Adobe RGB (1998) based on the Adobe specification.

Then why is it using a pure gamma curve and Adobe ACE is not implementing a slope limit contrary to their spec's Annex C? Is Adobe now doing it wrong?

No idea - you'd probably have to ask Chris Cox of Adobe that question.

Edit:
I looked at the rec2020 profile you made and see that it uses a TRC with a linear ramp, so I presume your other profiles do as well. Any idea why Adobe doesn't do that?

No - I haven't heard anything about how Adobe RGB(1998) came to be.

Also ProPhoto RGB uses a pure gamma and ACE doesn't modify it.

ProPhoto RGB implemented according to the specifications should have power curve with straight line slope limit. It is based on ROMM RGB, and the spec says:

   If Rr, Gr, or Br are less than 0.001953
      R = Rr*16
      G = Gr*16
      B = Br*16
   If Rr, Gr, or Br are greater than or equal to 0.001953
      R = Rr1/1.8
      G = Gr1/1.8
      B = Br1/1.8

I assume some apps use ramps while others (Adobe) don't. Is that your understanding?

All I know is that (mainly from testing Marti did some time ago, as well as my own testing) that Adobe ACE, littleCMS and ArgyllCMS are in very good agreement in the interpretation of ICC profiles, while other CMM (i.e. Microsoft) are less reliable. I'm not sure about Apples CMM.

[digitaldog]

I'm saying I created an ICC profile ("ClayRGB1998.icm") to implement Adobe RGB (1998) based on the Adobe specification.No idea - you'd probably have to ask Chris Cox of Adobe that question.No - I haven't heard anything about how Adobe RGB(1998) came to be.ProPhoto RGB implemented according to the specifications should have power curve with straight line slope limit. It is based on ROMM RGB, and the spec says:

   If Rr, Gr, or Br are less than 0.001953
      R = Rr*16
      G = Gr*16
      B = Br*16
   If Rr, Gr, or Br are greater than or equal to 0.001953
      R = Rr1/1.8
      G = Gr1/1.8
      B = Br1/1.8
All I know is that (mainly from testing Marti did some time ago, as well as my own testing) that Adobe ACE, littleCMS and ArgyllCMS are in very good agreement in the interpretation of ICC profiles, while other CMM (i.e. Microsoft) are less reliable. I'm not sure about Apples CMM.

Chris is long gone from Adobe 
Adobe RGB (1998) resulted from a mistake (they, Adobe thought they were specifying SMPTE-240M; got it wrong). Doug will have to ping SMPTE.

[Doug Gray]

I'm saying I created an ICC profile ("ClayRGB1998.icm") to implement Adobe RGB (1998) based on the Adobe specification.No idea - you'd probably have to ask Chris Cox of Adobe that question.No - I haven't heard anything about how Adobe RGB(1998) came to be.ProPhoto RGB implemented according to the specifications should have power curve with straight line slope limit. It is based on ROMM RGB, and the spec says:

   If Rr, Gr, or Br are less than 0.001953
      R = Rr*16
      G = Gr*16
      B = Br*16
   If Rr, Gr, or Br are greater than or equal to 0.001953
      R = Rr1/1.8
      G = Gr1/1.8
      B = Br1/1.8
All I know is that (mainly from testing Marti did some time ago, as well as my own testing) that Adobe ACE, littleCMS and ArgyllCMS are in very good agreement in the interpretation of ICC profiles, while other CMM (i.e. Microsoft) are less reliable. I'm not sure about Apples CMM.

Thanks Graeme, much appreciated.

I checked the ProPhoto RGB profile Adobe uses as well and it is also a pure gamma profile! But I also recall ROMM had a linear ramp as well so I don't know why that has filtered into Adobe stuff.

I haven't found a good rationale for a linear ramp on the front or matrix spaces. but it seems to have been more in common a decade or so ago than now. Hard to find much on it now. It is quite a small effect and only at really low luminance.

Perhaps high dynamic range stuff and Hollywood is a factor. Their (Academy of motion picture arts and sciences) 16 bit small float storage of video allows for a good dynamic range where a pure gamma fits better.

Added:

I've found a few discussions of Adobe's "Slope Limiting CMM" and the original ROMM paper. There is even a discussion that the Kodak copyrited ProPhoto RGB does not do it in the ICC profile in the expectation Adobe's CMM does it. See here:
http://www.photo-lovers.org/pdf/color/romm.pdf

It seems quite apparent that, at some point in time Adobe ACE did slope limiting but everything I've tried with Photoshop does not produce slope limited results. Very weird.

The rationale for slope limiting is cleaner inversions of near black colors from linear back to the gamma encoded space. This makes sense for a lot of the cpu tech that existed >10 or 20 years ago where small, lookup tables were used. But inversions using floating point have become faster on CPU's than lookup tables when working with higher precision data because of cache/memory latency.

I'm beginning to think Adobe ACE at one time did do slope limiting using older coding practices and at some point modernized it and eliminated slope limiting for efficiency as there is no invertability issue with modern processors.

I'm like a dog on a bone about this for some reason.  Hi Andrew

[GWGill]

I've found a few discussions of Adobe's "Slope Limiting CMM" and the original ROMM paper. There is even a discussion that the Kodak copyrited ProPhoto RGB does not do it in the ICC profile in the expectation Adobe's CMM does it. See here:
http://www.photo-lovers.org/pdf/color/romm.pdf

I could ask Geoff Woolfe if he recalls any of that detail if you like.

[Jack Hogan]

I wouldn't assume PS 'doesn't do that'. It used to do it and I would be surprised if it didn't do it any more. There are good reasons why they did it.

The test is not PS to PS. ACE is well behaved, so if it does do whatever it probably does it throughout. The test is ACE vs a literal interpretation of the spec (e.g. Microsoft ICM or similar).

So the Adobe Color Engine does indeed use a linear toe near the origin when dealing with Adobe RGB, at least in CS5, which is the version of PS I use.  In fact it uses two distinct linear sections. This is what the curves look like in 16-bit DNs:



Note how the conversion from sRGB linear to sRGB gamma performed by adhering to the spec in Matlab and by CS5's Adobe Color Engine independently track perfectly - and are well behaved near the origin, thanks to the linear toe.

However, Matlab's perfect 'spec' Adobe RGB gamma conversion results in some junk near the origin (can we call it noise Andrew?) and then follows the trajectory of a 1/2.2 exponential.  On the other hand the smarter Color Engine in CS5 introduces two linear sections near the origin and avoids such messes, in a win for practicality vs unattainable perfection.

Cheers,
Jack

[Doug Gray]

So the Adobe Color Engine does indeed use a linear toe near the origin when dealing with Adobe RGB, at least in CS5, which is the version of PS I use.  In fact it uses two distinct linear sections. This is what the curves look like in 16-bit DNs:



Note how the conversion from sRGB linear to sRGB gamma performed by adhering to the spec in Matlab and by CS5's Adobe Color Engine independently track perfectly - and are well behaved near the origin, thanks to the linear toe.

However, Matlab's perfect 'spec' Adobe RGB gamma conversion results in some junk near the origin (can we call it noise Andrew?) and then follows the trajectory of a 1/2.2 exponential.  On the other hand the smarter Color Engine in CS5 introduces two linear sections near the origin and avoids such messes, in a win for practicality vs unattainable perfection.

Cheers,
Jack

That's the first I've seen mentioned of 2 linear ramps with Adobe RGB. The paper on Adobe RGB linked previously would only have 1 ramp slope. Do you have the protocol that generated the second graph? I don't have CS5 but can test it against the current ACE.

The ramp I get, using PS CC, and converting the posted 16 bit tif with 65536 pixels (256x256) assigned to a linear gamma, shows the 2.2 "perfect" curve. It matches MATLAB.

The searches I've done on this strongly suggests that Adobe's ACE initially implemented a ramp but I see no indication it's in the current product.

The oldest PS I have is CS6 which I will install and test.

[Jack Hogan]

That's the first I've seen mentioned of 2 linear ramps with Adobe RGB. The paper on Adobe RGB linked previously would only have 1 ramp slope. Do you have the protocol that generated the second graph? I don't have CS5 but can test it against the current ACE.

The ramp I get, using PS CC, and converting the posted 16 bit tif with 65536 pixels (256x256) assigned to a linear gamma, shows the 2.2 "perfect" curve. It matches MATLAB.

The searches I've done on this strongly suggests that Adobe's ACE initially implemented a ramp but I see no indication it's in the current product.

The oldest PS I have is CS6 which I will install and test.

That's what ACE does in CS5, glad for you to double check it on a more current version of PS.

I think you don't see it because from what I understand of your comments you are not forcing PS to do the conversion.  The protocol I use is very simple:

1 ) Generate a plausible XYZD65 read noise 'black' frame
2 ) Project it into linear Adobe RGB via matrix multiplication
3 ) Write the file to tiff
4 ) Open the tiff with CS5
5 ) Assign a previously prepared linear Adobe RGB (1998) profile to the open image
6 ) Convert to the standard Adobe RGB (1998) profile with its gamma
7 ) Save the newly converted Adobe RGB image to tiff
8 ) Load this last tiff into Matlab and compare it to 3) raised to 'spec' aRGB gamma.

Code snippet attached.  If you PM me your email address I can send you scripts and xyz data.

Jack

[Doug Gray]

That's what ACE does in CS5, glad for you to double check it on a more current version of PS.

I think you don't see it because from what I understand of your comments you are not forcing PS to do the conversion.  The protocol I use is very simple:

1 ) Generate a plausible XYZD65 read noise 'black' frame
2 ) Project it into linear Adobe RGB via matrix multiplication
3 ) Write the file to tiff
4 ) Open the tiff with CS5
5 ) Assign a previously prepared linear Adobe RGB (1998) profile to the open image
6 ) Convert to the standard Adobe RGB (1998) profile with its gamma
7 ) Save the newly converted Adobe RGB image to tiff
8 ) Load this last tiff into Matlab and compare it to 3) raised to 'spec' aRGB gamma.

Code snippet attached.  If you PM me your email address I can send you scripts and xyz data.

Jack

Got it, your process is perfectly fine. The difference is that you start from a noisy linear source then convert to Adobe RGB and compare the mapping of linear RGB values to the converted, Adobe RGB ones so, for instance, a histogram of the result shows the noise distribution from the camera. You should see exactly the same linear to Adobe RGB plot with the tif16all.tif file I posted since it has all 64k, 16 pit pixels in it. Just load the tif1ll16.tif image at step 4 and run through the rest of the steps.

Confirmed!  I installed CS6.  PS CS6 exhibits exactly the same 2 linear ramps!  PS CC 2018 doesn't.

Somewhere between CS6 and the current CC (V20.0.n), Adobe dropped the slope limit in ACE conversions. sRGB is unchanged which is to be expected since the sRGB TRC has a longer slope limit than where the earlier ACE cuts in.

[Doug Gray]

Yep, CS6 has the linear ramp. Actually 2 of them as Jack noted.

Jack,

CS6 exhibits exactly the same ramp you see. No idea when Adobe ACE dropped the slope limiting. Here's the image and code.

% AdobeRGB_toe_test
imag_CC=reshape(imread('tif16all_aRGB_CC.tif'), 65536,3);
imag_CS6=reshape(imread('tif16all_aRGB_CS6.tif'), 65536,3);
plot(imag_CC(:,2)); hold on
plot(imag_CS6(:,2)); hold off
title('PS CC v CS6, ACE Toe')
legend('PS CC', 'PS CS6'); grid on

[32BT]

It could also indicate linear interpolation of some internal lut.

Does it make any difference if you enable/disable gpu?

[Doug Gray]

It could also indicate linear interpolation of some internal lut.

Does it make any difference if you enable/disable gpu?

No difference. GPUs presence doesn't affect the conversion.

It's technically possible for Adobe to use LUTs but more likely they used to use LUTs and dropping the linear ramp was a consequence of converting to SSE math. Back in the old days of processors, floating point math was slow relative to memory access so LUT solutions provided the best performance. But floating point and multiple core processors rapidly changed the tradeoff. Implementing a ramp would, these days, incur a small performance hit compared to just calculating the gamma. sRGB and a few others like L* based spaces still require LUTs. But Adobe RGB and ProPhoto can calculate gamma faster than using LUTs.

[Jack Hogan]

CS6 exhibits exactly the same ramp you see. No idea when Adobe ACE dropped the slope limiting.

Huh, interesting Doug.

Since I had the stuff out, here is a histogram to support my earlier obvious point on floating point: if you stick with floating point, the original luma Y is all there and the same after a round trip to Adobe RGB and back (the two histograms overlay each other) - while of course that's not the case if the same trip is performed via 16-bits.  The chart shows what happens near the origin, of course the same would hold true at the other end, the edges of the cube where clipping occurs.

Jack

[Doug Gray]

Since I had the stuff out, here is a histogram to support my earlier obvious point on floating point: if you stick with floating point, the original luma Y is all there and the same after a round trip to Adobe RGB and back (the two histograms overlay each other) - while of course that's not the case if the same trip is performed via 16-bits.  The chart shows what happens near the origin, of course the same would hold true at the other end, the edges of the cube where clipping occurs.

Jack

Hi Jack!

Where do the negative numbers come from. The histogram shows for the floating point. Was this created by adding noise to your data set? Negative numbers, by definition, represent out of gamut colors in an RGB space. Does this relate to the title of the histogram? 16 bit numbers in tiff files are unsigned and Adobe PS drops the low bit but still treats the numbers as unsigned which explains the right side of the histogram.

That aside, I'm not sure what the import of this is? Could you explain more how the dataset was generated and processed. Matlab, where applicable, is fine and a lot shorter than words.