Pages: [1]   Go Down

Author Topic: X-Rite Passport Inaccuracies  (Read 23726 times)

JustinK

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
X-Rite Passport Inaccuracies
« on: September 19, 2013, 02:37:13 pm »

I was directed here by a helpful photographer whom I contacted regarding my issues with the X-rite Passport.  My primary purpose for using it is to get accurate and consistent colors across different ambient and artificial light sources for architectural photography.  Also, I wanted to use it to match color between cameras.  On both counts, it has failed, and when I subjected it to a test using the reference values available on X-rite's website, it did not perform well.  I would like to get some feedback from experienced members here regarding my findings and the response I received from X-rite after sending them details of my testing.

Here is an outline of my test:
1) I took the photo of the X-rite Passport under overcast daylight (no artificial light) with two cameras, the Canon 5d and Canon 60d.
2) Then I created custom profiles from each photo using the X-rite Passport software.
3) Next, I applied the custom profile to each photo of the Passport
4) White balance was corrected in ACR by clicking the WB on the square two over from black (#22)
5) Both photos were loaded into Photoshop and color sampler points were applied to the blue, green, red, and yellow squares (#'s 13,14,15,16)
6) The image was converted to Lab color and the a and b values were compared against the reference values on X-rite's website.
7) The same process was repeated for the photo from each camera using the Adobe Standard profile


What I discovered was that the color was just as accurate if not more accurate in the Adobe Standard profile. Also, the colors did not match between the Canon 60d and 5d after applying the X-rite profiles. 
A couple sample points from the Canon 5d (again, a,b values):
Daylight overcast test-- X-rite: red a,b= 69,41;  Adobe Std: red a,b= 61,37; Reference (rounded to nearest whole number): red a,b=53,28.
X-rite: green a,b=-51,39; Adobe Std: green a,b=-47,36 Reference: green a,b= -38,31

I did the same test under artificial light (CFL I think) and the results were a little closer between the Adobe Standard and the X-rite, but for $100 and promises of accurate color, I was very disappointed.

After contacting X-rite, and sending over my findings and photos, I waited over a month to hear back, after their engineers were going to review my results.  The response I received was surprising.  X-rite said "To make a long story short, your standards for performance and data analysis would not be well suited to use ColorChecker Passport and its DNG profiles." And also "ColorChecker Passport software was designed to be a simple camera profiling solution for casual and non-professional photographers..." The recommendation was that I use an ICC profile as opposed to a DNG profile." They also offered me a refund on the Passport.

What do you think of my test and X-rite's response? Have you conducted your own objective testing? I don't need perfect color, but I do want consistent and accurate looking color.  Thanks in advance.


Logged

Vladimirovich

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1311
Re: X-Rite Passport Inaccuracies
« Reply #1 on: September 19, 2013, 02:43:26 pm »

What I discovered was that the color was just as accurate if not more accurate in the Adobe Standard profile.

it was said that Adobe Standard profile is not intended for a reproduction work (so the word "accurate" shall not be used), but rather to provide some consistent color rendering between different camera models according to what Adobe thinks is the pleasing one...

also it is useful to have two totally different targets (like xrite and qpcard) in your frame, build profile using one of them, render raw and check accuracy by another one... that is a better test.
Logged

jwlimages

  • Full Member
  • ***
  • Offline Offline
  • Posts: 135
Re: X-Rite Passport Inaccuracies
« Reply #2 on: September 21, 2013, 12:52:49 am »

I think you got excellent answers on the colorsync-users' list.    ;)

John
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: X-Rite Passport Inaccuracies
« Reply #3 on: September 21, 2013, 04:03:58 am »

Hi,

I would suggest that both XRite Passport software and Adobe DNG Profile editor oversaturate images. You should not complain about the cost of ColorChecker Passport, as it is free. What you pay for is the Color Checker Passport reference card.

You could just try to reduce saturation a bit and see if the results are better?

With DNG Profile Editor you can also tune the individual colors.

Best regards
Erik


I was directed here by a helpful photographer whom I contacted regarding my issues with the X-rite Passport.  My primary purpose for using it is to get accurate and consistent colors across different ambient and artificial light sources for architectural photography.  Also, I wanted to use it to match color between cameras.  On both counts, it has failed, and when I subjected it to a test using the reference values available on X-rite's website, it did not perform well.  I would like to get some feedback from experienced members here regarding my findings and the response I received from X-rite after sending them details of my testing.

Here is an outline of my test:
1) I took the photo of the X-rite Passport under overcast daylight (no artificial light) with two cameras, the Canon 5d and Canon 60d.
2) Then I created custom profiles from each photo using the X-rite Passport software.
3) Next, I applied the custom profile to each photo of the Passport
4) White balance was corrected in ACR by clicking the WB on the square two over from black (#22)
5) Both photos were loaded into Photoshop and color sampler points were applied to the blue, green, red, and yellow squares (#'s 13,14,15,16)
6) The image was converted to Lab color and the a and b values were compared against the reference values on X-rite's website.
7) The same process was repeated for the photo from each camera using the Adobe Standard profile


What I discovered was that the color was just as accurate if not more accurate in the Adobe Standard profile. Also, the colors did not match between the Canon 60d and 5d after applying the X-rite profiles. 
A couple sample points from the Canon 5d (again, a,b values):
Daylight overcast test-- X-rite: red a,b= 69,41;  Adobe Std: red a,b= 61,37; Reference (rounded to nearest whole number): red a,b=53,28.
X-rite: green a,b=-51,39; Adobe Std: green a,b=-47,36 Reference: green a,b= -38,31

I did the same test under artificial light (CFL I think) and the results were a little closer between the Adobe Standard and the X-rite, but for $100 and promises of accurate color, I was very disappointed.

After contacting X-rite, and sending over my findings and photos, I waited over a month to hear back, after their engineers were going to review my results.  The response I received was surprising.  X-rite said "To make a long story short, your standards for performance and data analysis would not be well suited to use ColorChecker Passport and its DNG profiles." And also "ColorChecker Passport software was designed to be a simple camera profiling solution for casual and non-professional photographers..." The recommendation was that I use an ICC profile as opposed to a DNG profile." They also offered me a refund on the Passport.

What do you think of my test and X-rite's response? Have you conducted your own objective testing? I don't need perfect color, but I do want consistent and accurate looking color.  Thanks in advance.



Logged
Erik Kaffehr
 

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: X-Rite Passport Inaccuracies
« Reply #4 on: September 21, 2013, 04:06:13 am »

Sorry,

Do you happen to have a link to the colorsync users list?

Best regards
Erik

I think you got excellent answers on the colorsync-users' list.    ;)

John
Logged
Erik Kaffehr
 

Vladimirovich

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1311
Re: X-Rite Passport Inaccuracies
« Reply #5 on: September 21, 2013, 11:31:12 am »

Logged

JustinK

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
Re: X-Rite Passport Inaccuracies
« Reply #6 on: September 21, 2013, 04:42:30 pm »

Yes I did receive good responses from the Colorsync Group.  Unfortunately, I believe there is a problem with the way the Passport is marketed. #1 It's not sold as a standalone unit. It's sold as a package for which the software is integral.  #2 X-rite claims the software and color checker will give you accurate color, and that it will match color responses between cameras.  I found both of those to be extremely off the mark.

Also, up to this point, no one has been able to explain why the Adobe Standard profile was just as accurate if not more accurate than the Passport software in my test.  Nor does it appear that anyone else I've contacted has performed a similar test.  The issue isn't just saturation, although the Passport software clearly produces more saturated colors than Adobe Standard (measured using HSB with my overcast daylight test).  I've used Adobe Editor in a separate CFL test, but not for my daylight test. I plan to do that to compare.

I did find this little gem from some online research. "How good are these profiles created by Colorchecker Passport? Are they as good as the standard Dual Illuminant ones that Adobe creates and includes with camera Raw and Lightroom?
Likely not, since Adobe uses calibrated light sources and a much larger set of colour patches to make their profiles."  The author goes on to say the Passport is a Must-Have, which I find very strange. I guess it depends if you want accurate color, or have some other purpose in mind.
http://www.luminous-landscape.com/reviews/accessories/colorchecker-psssport.shtml
Here's another review that was interesting, in which the author said one must use the Prophoto RGB with the Passport (using the proprietary profile generator) to retain nuances in color:
http://translate.google.se/translate?sl=sv&tl=en&js=n&prev=_t&hl=sv&ie=UTF-8&eotf=1&u=http://www.sfoto.se/teknik/farghantering-kalibrering-och-utskrifter/kalibrera-din-kamera&act=url
Logged

Vladimirovich

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1311
Re: X-Rite Passport Inaccuracies
« Reply #7 on: September 21, 2013, 07:51:03 pm »

Adobe uses calibrated light sources and a much larger set of colour patches to make their profiles.

they use monochromator and they do not use naturally Adobe PE... why 'd somebody use reflective targets when they have resources (money, people, time) to use very good, properly calibrated monochromators ?

Logged

JRSmit

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 922
    • Jan R. Smit Fine Art Printing Specialist
Re: X-Rite Passport Inaccuracies
« Reply #9 on: September 22, 2013, 03:17:06 am »

To my knowledge and testing one  needs prophoto rgb working space to e compass the camera profile.  This btw also true for camprofs of
C1. Simply put the camprof is too large for adobergb
So prophotorgb and 16bit per channel.
Logged
Fine art photography: janrsmit.com
Fine Art Printing Specialist: www.fineartprintingspecialist.nl


Jan R. Smit

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: X-Rite Passport Inaccuracies
« Reply #10 on: September 22, 2013, 03:43:02 pm »

Hi,

I don't see how this relates to the topic. Anyway, the ColorChecker reference colors fit comfortably in AdobeRGB.

Using Prophoto RGB is reasonable but less than optimal when publishing images digitally, as most displays only handle sRGB. High en display handle Adobe RGB, but if the colors fit in a smaller color space there is little advantage to use a greater color space.

Best regards
Erik


To my knowledge and testing one  needs prophoto rgb working space to e compass the camera profile.  This btw also true for camprofs of
C1. Simply put the camprof is too large for adobergb
So prophotorgb and 16bit per channel.
Logged
Erik Kaffehr
 

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20614
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: X-Rite Passport Inaccuracies
« Reply #11 on: September 22, 2013, 05:39:41 pm »

if the colors fit in a smaller color space there is little advantage to use a greater color space.
True but that's a bit of work**, examining the image, rendered from ProPhoto primaries in an Adobe converter to both to evaluate what will fit and which to pick. The alternative is not checking and picking either ProPhoto or Adobe RGB (1998) in this context. If you pick the wrong one, you clip colors. That seems the greater of the two evils.

** I suppose one could toggle between the two in soft proof, then examine the histogram clipping indicators.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

JustinK

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
Re: X-Rite Passport Inaccuracies
« Reply #12 on: September 23, 2013, 01:25:52 am »

I changed the color profile of the picture of the Passport to Prophoto RGB in ACR and then converted to Lab.  I compared the blue,green,red, yellow, swatch values to the Adobe RGB and Prophoto RGB versions.  All stayed exactly the same except blue, which went from L,a,b= 22,26,-67 in Adobe RGB to L,a,b=20,23,-70 in Prophoto RGB.

ErikKaffehr, thanks for those links.  It would be interesting if you compared the corrected photo of the Passport using each profile-- Adobe Std, X-rite Passport, and DNG editor, looking at the a,b color values and comparing them to the reference values here: http://xritephoto.com/ph_product_overview.aspx?ID=1257&Action=Support&SupportID=5159
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: X-Rite Passport Inaccuracies
« Reply #13 on: September 24, 2013, 01:15:33 am »

Hi,

I have run Imatest on a shot of the CCP that I used for calibration. Saturation was reduced by 13 (-13) on the CCP and DNG Profile Editor shots.

I will make a test with flourescent this evening.

Best regards
Erik

Logged
Erik Kaffehr
 

JustinK

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
Re: X-Rite Passport Inaccuracies
« Reply #14 on: September 24, 2013, 01:35:20 am »

Erik, really interesting. Thanks for posting this.  Can you post the original RAW shot of the colorchecker used in those tests as well, or would you be willing to send it to me?
What camera profile was used in the control shot (the one on the far left)?
I've never studied Imatest results before, but in looking at the color plot, I see one of the values is the deltaE ab, which I take to mean the difference in the a,b color values between the reference and the measured charts. Is that correct? In the Passport shot, the deltaE ab, both mean and max, are greater in the Passport and DNG shots than for the original. However many of the colors do appear closer to the ideal, save for yellow. In my test (looking at red, green, blue, yellow) yellow was the one color that was slightly more accurate with the Passport. I also notice that the mean camera chroma (saturation)% increases with the Passport over the orginal, and the DNG is slightly more saturated yet.
If you are up for analyzing the shot I took of the Passport, I'll send it to you.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: X-Rite Passport Inaccuracies
« Reply #15 on: September 24, 2013, 05:03:34 pm »

Hi,

I have put two DNG files here: http://echophoto.dnsalias.net/ekr/Articles/Color_profiles/DNGFiles/

The first one is the one I used for the first comparison

The second one is in a room illuminated by an energy saving lamp. As you see, the Adobe Standard Profile falls apart. The DNG Profile editor worked fine. Color checker passport stopped with erroneous error message.

This time I left saturation at default for both images. The colors on the AS profile were widely off anyway.

If you send me a DNG file I will gladly evaluate it with Imatest.

Best regards
Erik



Erik, really interesting. Thanks for posting this.  Can you post the original RAW shot of the colorchecker used in those tests as well, or would you be willing to send it to me?
What camera profile was used in the control shot (the one on the far left)?
I've never studied Imatest results before, but in looking at the color plot, I see one of the values is the deltaE ab, which I take to mean the difference in the a,b color values between the reference and the measured charts. Is that correct? In the Passport shot, the deltaE ab, both mean and max, are greater in the Passport and DNG shots than for the original. However many of the colors do appear closer to the ideal, save for yellow. In my test (looking at red, green, blue, yellow) yellow was the one color that was slightly more accurate with the Passport. I also notice that the mean camera chroma (saturation)% increases with the Passport over the orginal, and the DNG is slightly more saturated yet.
If you are up for analyzing the shot I took of the Passport, I'll send it to you.

Logged
Erik Kaffehr
 

JustinK

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
Re: X-Rite Passport Inaccuracies
« Reply #16 on: September 24, 2013, 10:22:50 pm »

Got the photos, thank you!  First, after some initial testing (using blue, green, red, yellow swatches), the colors in the image under fluorescent light with the Adobe Std. profile applied are terrible, as you stated, and more accurate with the Passport profile I generated. With the daylight shot, the Passport also appears to be more accurate than Adobe Standard.  Could you send/post the entire daylight photo rather than a crop?  I would like to see the environment it was taken in.  If the Passport had demonstrated that level of accuracy for me, I would have been more pleased.

For example, in my test, after applying the Passport profile, I measured the blue swatch at a,b= 23,-70
In yours I measure blue as a,b= 19,-58. The reference value is a,b=14,-50 (rounded to nearest whole number).

I would like to try to reproduce your fluorescent light test to the extent I can.  Do you know the color temperature of the light source? I wonder if for some reason the Adobe Standard profile is not calibrated effectively for your particular camera under fluorescent light.  I have no idea if that is the case-- it's just a thought.

I will send you the full raw file of my overcast daylight shot.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: X-Rite Passport Inaccuracies
« Reply #17 on: September 24, 2013, 11:28:56 pm »

Hi Justin,

Unfortunately I don't know the color temperature of the ES lamp. It's one of those you can buy anywhere sitting above my kitchen table, I suspected it to be horrible.

Regarding the Adobe Standard Profile it is measured with a monochromator AFAIK.

The cropped DNG file contains the whole image, you just need to uncrop it, I think.

I will retest with another camera, the one I used for these tests is a P45+.

Best regards
Erik


Got the photos, thank you!  First, after some initial testing (using blue, green, red, yellow swatches), the colors in the image under fluorescent light with the Adobe Std. profile applied are terrible, as you stated, and more accurate with the Passport profile I generated. With the daylight shot, the Passport also appears to be more accurate than Adobe Standard.  Could you send/post the entire daylight photo rather than a crop?  I would like to see the environment it was taken in.  If the Passport had demonstrated that level of accuracy for me, I would have been more pleased.

For example, in my test, after applying the Passport profile, I measured the blue swatch at a,b= 23,-70
In yours I measure blue as a,b= 19,-58. The reference value is a,b=14,-50 (rounded to nearest whole number).

I would like to try to reproduce your fluorescent light test to the extent I can.  Do you know the color temperature of the light source? I wonder if for some reason the Adobe Standard profile is not calibrated effectively for your particular camera under fluorescent light.  I have no idea if that is the case-- it's just a thought.

I will send you the full raw file of my overcast daylight shot.
Logged
Erik Kaffehr
 
Pages: [1]   Go Up