Luminous Landscape Forum

Raw & Post Processing, Printing => Colour Management => Topic started by: Doug Gray on November 12, 2017, 12:54:13 am

Title: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 12, 2017, 12:54:13 am
I was evaluating the Lab distances of a 3500 patch set and noticed a very large number of patches had lab values that were close to some other one in the patch set. This seems pointless and may even lead to anomalies around colors with different RGB values but nearly identical Lab values when printed. The maximum dE is 12.0 while the minimum is .08.

So, I'm currently recursively eliminating those that are small. I've reduced the RGB patch set to about 2,000 which results in patches that are at least a dE of 5 from all others yet the maximum has remained at dE of 12.

This will create a patch set with "holes" but holes that don't provide much useful info. Question is: can the profiling software handle these?

If so, is the quality of profiles generated as good as that of the full patch set or at least very close? If so, I will expand the set to 10000 or so patches and apply the same removal process. Hopefully, I can get a much smaller patch set with a lower maximum distance that is as good as the extreme patch set. Once I have such a patch set then it should be good to create great profiles on that printer for any future paper with a lot less paper waste.

But it really depends on how well the profiler s/w handles patch sets with "holes."

Anyone ever try this?
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: GWGill on November 12, 2017, 02:15:49 am
This will create a patch set with "holes" but holes that don't provide much useful info. Question is: can the profiling software handle these?
I'd imagine that many can. ArgyllCMS certainly.
Quote
If so, I will expand the set to 10000 or so patches and apply the same removal process. Hopefully, I can get a much smaller patch set with a lower maximum distance that is as good as the extreme patch set.
...
Anyone ever try this?
Well, ArgyllCMS default ofps algorithm attempts to do the opposite - generate a patch set with well distributed patches. The default criteria using a previously generated profile is more subtle though - it takes curvature into account, increasing the density of patches where the device response is more curved, and reducing it where it is flatter. There is also a (tweak-able) bias towards increased patches near neutral, mimicking the characteristics of CIE DE94/2000/DIN99 in compensating for the increased visibility of near neutral inaccuracy.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Mark D Segal on November 12, 2017, 03:19:09 am
Ethan Hansen published very useful research on optimal patch set sizes (which would include for their underlying configuration) in this Forum. I tested the 1877 set against the 2033 set (the former "optimal", the latter less so according to his information) and determined that he was largely correct. Instead of starting from scratch re-inventing patch sets - which may be useful, maybe not, I decided the most practical and time-efficient approach would be to test what's out there and if satisfactory jut use it. But then again, you may come up with something yet better in some respect - hard to know without trying, so if you do, looking forward to your results, but to answer your question - no I haven't tried re-configuring patch sets but I remain interested in results from others who are doing so.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Alan Goldhammer on November 12, 2017, 07:46:12 am
I'd imagine that many can. ArgyllCMS certainly.Well, ArgyllCMS default ofps algorithm attempts to do the opposite - generate a patch set with well distributed patches. The default criteria using a previously generated profile is more subtle though - it takes curvature into account, increasing the density of patches where the device response is more curved, and reducing it where it is flatter. There is also a (tweak-able) bias towards increased patches near neutral, mimicking the characteristics of CIE DE94/2000/DIN99 in compensating for the increased visibility of near neutral inaccuracy.
My approach with ArgyllCMS is to use a small patch set to generate the first profile (which is generally quite good) and then a slightly larger profile that incorporates both a B/W patch set and near neutral tweak mentioned by Graeme for the final profile.  It takes an extra day to do this (two sets of paper targets to let dry) but the profiles are quite good for my use.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 12, 2017, 02:49:33 pm
My approach with ArgyllCMS is to use a small patch set to generate the first profile (which is generally quite good) and then a slightly larger profile that incorporates both a B/W patch set and near neutral tweak mentioned by Graeme for the final profile.  It takes an extra day to do this (two sets of paper targets to let dry) but the profiles are quite good for my use.
I suspect this is may be effective. There is a great deal of fairly sophisticated stuff going on in Graeme's s/w. I'm also curious about the impact of adjusting the instrument plus device uncertainty option he has. The Isis has proven highly repeatable compared to the I1Pro 2 scans I previously did. However, that repeatability has identified printer related errors on both my 9500 and 9800. The errors are of similar magnitude but differ in nature. The 9800's greatest variation correlates to the horizontal position across the paper but is quite stable varying little between consecutive prints or a 24 hr lag, first print. The 9500 II, OTOH, has the most sensitivity to the colors of the patches in the same row. For instance printing successive black patches results in a shift in L*, something I've not observed with the 9800.  It also exhibits some shift between the first and consecutive prints. But then the 9500 II is also much less lumpy in the device response.

So I'm focusing on the 9800 now because it is more repeatable.

As an aside, I've done a bit of testing of drying time characteristics.

"Dry" is a relative term. Inked paper has two components, water and some sort of water soluble glycol used to suspend the pigments. While a print appears dry immediately, it isn't and measuring it immediately after printing results in an average dE76 of about 1 compared to a print dried for a few days. However, by measuring the paper with a thermal imager with 0.05C resolution an estimate can be made of the amount of residual moisture. The temp delta is proportional to the rate of evaporation to a first order. Initially, after printing, the surface is about 2C below ambient due to evaporative heat transfer. After about 10 minutes the temperature delta drops to 1C gradually decreasing over the next 20 minutes to under .05C. However, the glycol evaporative rate is much slower and doesn't show up with the imager.

There is another effect related to humidity I've noticed small shifts when the paper is dried for days at low humidity (10%) with dE's around .2 compared to 50% RH.  Returning it to normal humidity also results in small color changes back towards that of the 50% samples. This occurs within an hour or so and one can see the adsorption of water from the air by noting weight increases with an analytic scale.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Alan Goldhammer on November 12, 2017, 04:30:09 pm
I suspect this is may be effective. There is a great deal of fairly sophisticated stuff going on in Graeme's s/w. I'm also curious about the impact of adjusting the instrument plus device uncertainty option he has. The Isis has proven highly repeatable compared to the I1Pro 2 scans I previously did.
I'm using an i1Pro to do the readings with the manual chart reader.  I do duplicate readings and use the average utility before running the profile program.  I've always downloaded the readings into Excel and looked for outliers and occasionally there will be one but overall the realtive dEs using the 'profcheck' utility are quite low (I don't have any other profile making or checking software other than ArgyllCMS).  Several months ago I profiled Moab Entrada Natural on my Epson 3880, a paper that I really like.  The average error was 0.18 which is the lowest I've seen. 

Quote
As an aside, I've done a bit of testing of drying time characteristics.

"Dry" is a relative term. Inked paper has two components, water and some sort of water soluble glycol used to suspend the pigments. While a print appears dry immediately, it isn't and measuring it immediately after printing results in an average dE76 of about 1 compared to a print dried for a few days. However, by measuring the paper with a thermal imager with 0.05C resolution an estimate can be made of the amount of residual moisture. The temp delta is proportional to the rate of evaporation to a first order. Initially, after printing, the surface is about 2C below ambient due to evaporative heat transfer. After about 10 minutes the temperature delta drops to 1C gradually decreasing over the next 20 minutes to under .05C. However, the glycol evaporative rate is much slower and doesn't show up with the imager.

There is another effect related to humidity I've noticed small shifts when the paper is dried for days at low humidity (10%) with dE's around .2 compared to 50% RH.  Returning it to normal humidity also results in small color changes back towards that of the 50% samples. This occurs within an hour or so and one can see the adsorption of water from the air by noting weight increases with an analytic scale.
The drying characteristics are also paper dependent as well.  I've not done anything to the same extent that you have.  I only looked at differences at 6, 24, and 48 hours.  I was pretty satisfied that 24 hours was just fine for my purposes.

The glycol evaporation rate is dependent on what types of glycols are in the ink mixtures and Epson don't give much of a clue here.  a couple of years ago I went back to look at the Material Safety Data Sheets and if I recall they only said it was a mixture of proprietary glycols.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 12, 2017, 07:03:09 pm
I'm using an i1Pro to do the readings with the manual chart reader.  I do duplicate readings and use the average utility before running the profile program.  I've always downloaded the readings into Excel and looked for outliers and occasionally there will be one but overall the realtive dEs using the 'profcheck' utility are quite low (I don't have any other profile making or checking software other than ArgyllCMS).  Several months ago I profiled Moab Entrada Natural on my Epson 3880, a paper that I really like.  The average error was 0.18 which is the lowest I've seen. 
I just ran profcheck on the Argyll profile and it had exactly the same dE00 as the data shown in the histograms. They are calculated the same way. Don't rely on them too much, They only describe the accuracy of the AtoB table. While it's the most accurate, the BtoA tables are the ones used for printing and they tend to be not as good. That's why I run an additional check printing an in-gamut image set in Abs. Col. and measuring them with a spectro for match stats.

What I did with the I1Pro 2 was similar. Multiple scans tended to identify the outliers. Oddly, the i1iSiS tends not to because they repeat and can be quite large. Yet the average dE of sequential scans is almost always <= .05 dE00. Careful positioning on the patches with an I1Pro 2 confirms the outlier is real.  Because the i1iSiS reads the same location on the patches within about  .25mm, outliers are sticky and tend to remain in multiple scans more than they do with the I1Pro 2.

One of the tricks I did with the I1 Pro was make the patches slightly taller (10mm)  then make 2 or 3 scans shifting the alignment rule from the top, center, bottom of the patches. There is no way to do this with the i1iSiS as it tracks the center regardless of height. So to identify outliers you have to print at least 2 charts or one chart with duplicate colors.

Sticky outliers aside, the I1Pro 2 is not as consistent as the i1iSiS! I would be lucky to get repeat reads of charts within .2 dE00 ave with the Pro. That's both bad and good. The bad part of it is instrument variation from the tungsten lamp, the good part is you are reading different parts of the patches so you can identify outliers and average results effectively w/o duplicating. Mixed blessing but really both are excellent instruments.

As for the i1iSiS, changing the width from 6mm to 12mm reduces the size of the outliers but I find it better to just use additional patches of the same color and check their differences to find outliers. This applies most to the 9800 because of it's horizontal position effecting patch color.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Mark D Segal on November 13, 2017, 02:44:07 am
Then again, dE values of 0.2 will be undetectable to the naked eye. There is a limit to how fussy and precise one needs to be about this stuff, and that limit depends on the nature of the photo, the colours being measured and whether there are close comparisons at hand when viewing the version of primary interest.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 13, 2017, 10:34:41 am
Then again, dE values of 0.2 will be undetectable to the naked eye. There is a limit to how fussy and precise one needs to be about this stuff, and that limit depends on the nature of the photo, the colours being measured and whether there are close comparisons at hand when viewing the version of primary interest.
Certainly true. The outliers do need to be located because one, large enough, outlier can create visible ripples in a gradient if one happens to have that gradient in their printed image.  Still, quite rare to see that.

I probably should make clear that much of this is just trying to get the most out of my setup for producing specialized charts. That comes from some prior work setting up QA processes for lenses and imager subassemblies where optimizing buys margin improving process control.

Pretty much no, in gamut, prints I've made in quite a long time from patch sets of 1500 or more can be visibly discerned from each other outside of instrumentation.

Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 15, 2017, 11:07:35 pm
Trimming patches with small dEs was a bust. They worked on Argyll but not I1Profiler which just says it can't make a profile. And even on Argyll the benefits were minimal and only occurred with about a 10% pruning.

So I examined the problem more closely. Turns out the 9800 has a very different problem than the 9500. The 9800 color variation is correlated with its location on paper and it is color dependent. By printing a large number of duplicate patches in each color then examining their statistics it turns out that colors exhibiting the largest variation are in the more saturated greens and, to a lesser degree, the oranges.

Specifically, out of a set of 70 distributed colors repeated 13 times in random locations, the mean dE00 of the worst case color, a fairly saturated green, was .38*. However the median color set of 13 patches was .17. Of the 70 sets of colors, 90% had a mean dE00 of .26 or less.

To mitigate this one can create a patch set where colors exhibiting the greatest variation are repeated N times as required to reduce the variance of the mean to the same levels as, say, just over the median color set. For instance 3 duplicates of colors closest to the worst case 10% color sets, and 2 duplicates for colors in the next 20% should significantly reduce the errors that will be baked into a profile.

These numbers may seem small but they are average dE00s.  The variation across the 13 colors typically maxes around 3 times larger. So baking in fewer errors in the profile, which then compound with the unavoidable intrinsic error when printing seems desirable.

Another approach could be just printing two sets of charts with the colors scrambled differently. Then, after drying and scanning them, identifying the deviates and printing a third set of just those, perhaps duplicated once again and averaging the whole lot. Simple, but a lot of paper.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: GrahamBy on November 16, 2017, 07:31:00 am
I wonder if there might be some value in doing some L*a*b*-space smoothing?

My experience of this is in a completely different domain: software for automatic engine tuning, where you measure exhaust-gas oxygen and use it to apply corrections to a 2-d map of injector duration as a function of rpm and throttle opening. Because of the measurement errors, you can end up with some pretty implausible maps, but these can be improved if you apply a Gaussian smooth to the corrections (which are already averaged for each cell... ideally you could imagine a regression that would take into account the number of observations and their variance, but I didn't go there for various reasons).

One problem is knowing what is variance, since if the original map is "rough", a perfect set of corrections must also be rough... I had the luxury of iteration and damping (ie making only half the suggested change at each step) without the cost and hassle of making repeated prints.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: GWGill on November 16, 2017, 07:56:06 pm
I wonder if there might be some value in doing some L*a*b*-space smoothing?
I can't speak for how Profile Maker works (the news that it failed when points were deleted from the chart is interesting), but ArgyllCMS uses a scattered data interpolation routine to turn the patch data into an A2B table (plus per channel curves, which is a tweak with different story). There is a smoothing magic number involved, and the "-r" parameter allows indirect control of that smoothness/accuracy balance. It's this smoothness balance that prevents "over fitting". Using a scattered data routine makes it fairly agnostic about the chart point distribution.

[ I had a clever idea a while ago for auto tuning this smoothing factor in a gamut-local way, and got as far as proving to myself that it would probably work, but turning that sketch into working code is still on my "nice to do sometime" list, as it is not simple, and would mean revisiting every use of the scattered fitting function in the whole code base. ]
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 17, 2017, 12:36:45 am
I can't speak for how Profile Maker works (the news that it failed when points were deleted from the chart is interesting), but ArgyllCMS uses a scattered data interpolation routine to turn the patch data into an A2B table (plus per channel curves, which is a tweak with different story). There is a smoothing magic number involved, and the "-r" parameter allows indirect control of that smoothness/accuracy balance. It's this smoothness balance that prevents "over fitting". Using a scattered data routine makes it fairly agnostic about the chart point distribution.

[ I had a clever idea a while ago for auto tuning this smoothing factor in a gamut-local way, and got as far as proving to myself that it would probably work, but turning that sketch into working code is still on my "nice to do sometime" list, as it is not simple, and would mean revisiting every use of the scattered fitting function in the whole code base. ]
Hi Graeme
Actually it was I1 Profiler that failed. I didn't try PM5.

Indeed, I played a bit with the -r option. Reducing it from the default of .5% to .1% grouped the A2B much closer though still with a distribution greater than I1Profiler.

Indeed, I think I1 Profiler by default overfits in the broader gamut but appears to treat the near neutrals which were at twice the density differently. Overfitting isn't good if there is significant variance in the individual color patches as is the case for the 9800. However, I got good results with a 2500 patch set duplicated and re-randomized then averaged. This reduced the impact of overfitting that would be more problematic with a 5000 patch set w/o duplicates.

Further pursuing the notion that increasing the patch duplication only on colors that exhibit high variance looks very promising. I did a near color check and indeed, patches with high color variance also had closest neighbors with high variance.

Therefor, using a patch set where the expected variance for a particular color is reduced by N duplication through simple lookup to the closest color that I have good statistics on should improve the overall color variance without duplicating all patches the majority of which have only a small variance.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: GWGill on November 17, 2017, 02:27:00 am
Therefor, using a patch set where the expected variance for a particular color is reduced by N duplication through simple lookup to the closest color that I have good statistics on should improve the overall color variance without duplicating all patches the majority of which have only a small variance.
I've always wondered whether such an approach is optimal though - if the smoothing is at the appropriate level, it should be better to use the extra patch budget for more unique samples, thereby getting more detail and averaging out variance at the same time.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 17, 2017, 10:55:32 am
I've always wondered whether such an approach is optimal though - if the smoothing is at the appropriate level, it should be better to use the extra patch budget for more unique samples, thereby getting more detail and averaging out variance at the same time.
Hi Graeme,
I think you're right and that should provide somewhat better results. At least if the profiling software can handle the variation and smooth reasonably. It might be worse if the s/w overfits like I1Profile seems to do.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 17, 2017, 01:08:47 pm
I wonder if there might be some value in doing some L*a*b*-space smoothing?

My experience of this is in a completely different domain: software for automatic engine tuning, where you measure exhaust-gas oxygen and use it to apply corrections to a 2-d map of injector duration as a function of rpm and throttle opening. Because of the measurement errors, you can end up with some pretty implausible maps, but these can be improved if you apply a Gaussian smooth to the corrections (which are already averaged for each cell... ideally you could imagine a regression that would take into account the number of observations and their variance, but I didn't go there for various reasons).

One problem is knowing what is variance, since if the original map is "rough", a perfect set of corrections must also be rough... I had the luxury of iteration and damping (ie making only half the suggested change at each step) without the cost and hassle of making repeated prints.

Graeme's comment about the -r option in Argyll does implement degrees of smoothing in the device space measurements. The issue with printer targets is that there are occasional large outliers in addition to the bulk of variation which is more consistent with a normal distribution. The outliers are >5 std dev out and appear to be mostly due to variations in the paper surface itself rather than the printer. For instance with Canson rag matte the outliers are close to non-existent. Not sure why they are more frequent on glossy and worse, on semigloss/luster, but they are.  I'm not sure whether just excluding them at some nominal threshold or doing a Gaussian smoothing would be better. Interesting idea though.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: digitaldog on November 17, 2017, 01:45:31 pm
Whatever test of patches you use, be sure to also include Bill Atkinson's 1728-patch target!
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: digitaldog on November 17, 2017, 01:47:56 pm
My approach with ArgyllCMS is to use a small patch set to generate the first profile (which is generally quite good) and then a slightly larger profile that incorporates both a B/W patch set and near neutral tweak mentioned by Graeme for the final profile.  It takes an extra day to do this (two sets of paper targets to let dry) but the profiles are quite good for my use.
I do the same in i1Profiler where my optimization target is larger than the original (Bill Atkinson's 1728-patch target) by nearly 2. And I've seen it produce better results, sometimes on some papers/printers. Both in the soft proof and the output in some areas of color space. It's like the later patches 'fill in the holes' in some cases, not all. So I just always do a two step profile process.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 18, 2017, 11:12:20 pm
I created a patch set with:
1. 15x15x15 grid points, evenly spaced.
2. 30x3x3 neutrals and near neutrals
3. About 2400 Duplicate patches (2 or more) colors with high variance

Measured ave dE00 against reference Lab colors: .40.

This compares to about .45 for duplicate/averaged 2553 patches. Both require 6 US letter size Isis target prints.

Included is a histogram comparing reference Lab colors to measured, printed of the same. Also, the CGATS file suitable for loading into I1Profiler optimized for the Epson 9800. It uses up 6 pages but goes pretty fast with M2 on an Isis.

The largest dE00 values are associated with Lab values of L=95,0,0 which is just outside the paper/printer gamut which is trying to bring the a* and b* in to 0 while the paper's white point is already at L=95.  Also, the Epson 9800 response is lumpy with a particularly strong deviation in b* from -1 to -4 around L*=90 from roughly L*85 to 93 and this is exceeding what the 3D Luts can deal with. Outside of those the ave dE00 is about .38.

This lumpy anomaly in the 9800 is intrinsic. It can be seen running the A2B tables of the canned profiles from 2005 so it's not a function of wear and tear. However, it's weird in that the Advanced B&W is quite well behaved and smoothly transitions along the entire range from 0 to 255 with only gradual, smallish changes in a* and b*. Apparently, it's just something intrinsic in how the microweave color algos are done.

While it's not smaller, it does have better performance than anything I've made less than or equal to targets with 6 US letter pages.

Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: GWGill on November 19, 2017, 03:55:53 am
This lumpy anomaly in the 9800 is intrinsic. It can be seen running the A2B tables of the canned profiles from 2005 so it's not a function of wear and tear. However, it's weird in that the Advanced B&W is quite well behaved and smoothly transitions along the entire range from 0 to 255 with only gradual, smallish changes in a* and b*. Apparently, it's just something intrinsic in how the microweave color algos are done.
Presumably you're driving it with in RGB, so such weirdness could just be a limitation of the profiling used to create its RGB->CMYKLCLMLLK lookup table.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 19, 2017, 03:04:19 pm
Presumably you're driving it with in RGB, so such weirdness could just be a limitation of the profiling used to create its RGB->CMYKLCLMLLK lookup table.

Yes, device space RGB.  However, further investigation shows that only about half the abrupt  b* shift is from the RGB->CMYKLCLMLLK lookup table. The other half is due to differences in inking and how the Isis does M2.  Turns out there is significant fluorescence from the LED's spectral components around 400nm to 430nm which is showing up as a significant bump at 480nm. It's particularly exacerbated because the LED luminance at 480nm is at a significant minimum which greatly magnifies the impact of fluorescence. This  bump does not occur with a standard uV cut tungsten spectro and it is sensitive to the mix of Y, LM and LC relative to the LLK proportions.

Here's ABW compared to printing in color mode RGB w/o color management. Also the spectrum of Costco's glossy showing the bump around 480nm. This isn't there with a regular I1Pro uV Cut.

For the tri color plot, Blue=L*  Orange=a*   Yellow=b*
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: GWGill on November 19, 2017, 07:17:34 pm
The issue with printer targets is that there are occasional large outliers in addition to the bulk of variation which is more consistent with a normal distribution.
Quote
I'm not sure whether just excluding them at some nominal threshold or doing a Gaussian smoothing would be better. Interesting idea though.
A few people use a workflow that excludes such outliers using ArgyllCMS's profcheck -P (http://www.argyllcms.com/doc/profcheck.html) functionality.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on November 19, 2017, 11:33:19 pm
A few people use a workflow that excludes such outliers using ArgyllCMS's profcheck -P (http://www.argyllcms.com/doc/profcheck.html) functionality.

Graeme,

You have quite a nice set of tools. I've made some similar ones in Matlab but only with options I've needed. Far less extensive that what you have done. I've found Matlab's interactive approach is really handy for exploring the numerical data in different ways. One of the more useful recent tools is creating time stamped, labeled I1Isis compatible tiff files from Cgats at a selectable DPI. Also, I have some specialized (weird) tif files for testing I1Isis alignment tracking. The I1Isis has proved a surprisingly good instrument.

One of the things I'm looking at is trying to identify outliers w/o printing duplicate patch sets. It appears that when an outlier occurs, the variation is higher for successive scans than patches that aren't outliers. This works quite well already for I1Pro scans where outliers aren't as "sticky" but the tracking consistency is so good with the I1Isis that the main variation is from the time slice sampling which creates about .5mm uncertainty on the patch locations. That's pretty small but enough to bounce the outliers around a bit.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Ethan Hansen on December 05, 2017, 08:32:07 pm
Here are a few general thoughts on profiling patch sets.

Quote
This will create a patch set with "holes" but holes that don't provide much useful info. Question is: can the profiling software handle these?

This depends on the assumptions made by the profiling software. ArgyllCMS happily handles these situations and the default space-filling patch set does exactly this. In the case of X-Rite, my experience is that the software behavior reflects the code origins. The i1P profiling engine appears to draw heavily on MonacoProfiler which required equally spaced source data points for RGB profiles. i1P inherits some flexibility from ProfileMaker, but it quickly gets brittle with oddball patch sets.

Are you absolutely sure that the variations you see are caused by the printer and not the iSis? One step of the target analysis we perform is to both check a small set of duplicate patch values and validate whether differences between "neighboring color" patches makes sense. Several times a week we flag a measurement that is dubious because of variation.

Patch sizes below 6.6 mm or so on the iSis are more susceptible to spurious readings. Mismeasurements due to angled papers are easier to detect than errors from the timing resolution you mentioned above.

That said, some printers do have variations on how they lay down color. When we start evaluating a particular printer family this is something we check. Depending on the printer we'll either add duplicate patches or use some flavor of radial basis function to derive a correction matrix.

I'm an empirical guy, so I prefer to confirm measurements on hardware less prone to orientation variability than the iSis. If you want, shoot me a PM or email and we can measure one of your targets on a Spectroscan.

Lumpy bumpy RGB output behavior has been an Epson trademark for nearly two decades. Each model improves upon the last, but I wonder whether Epson never resolved the issue completely because doing so would be tacit admission there was a problem in the first place. ABW mode gave a clean slate to the software team.

The spurious peak at 480 nm in iSis M2 measurements moves around somewhat depending on the paper stock and brightener levels for Epson x900 printers. There are two approaches you can use here. First, measure in M1 mode with the caveat that your prints are only going to match when viewed in an ISO 3664:2009 compliant viewing booth. The other is to again measure in M1 but use profiling software with useful software-based optical brightener compensation. Yes, I know i1P has the manual OBC module but its utility is limited at best in the real world.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on December 09, 2017, 12:03:24 am
Here are a few general thoughts on profiling patch sets.

This depends on the assumptions made by the profiling software. ArgyllCMS happily handles these situations and the default space-filling patch set does exactly this. In the case of X-Rite, my experience is that the software behavior reflects the code origins. The i1P profiling engine appears to draw heavily on MonacoProfiler which required equally spaced source data points for RGB profiles. i1P inherits some flexibility from ProfileMaker, but it quickly gets brittle with oddball patch sets.

Are you absolutely sure that the variations you see are caused by the printer and not the iSis? One step of the target analysis we perform is to both check a small set of duplicate patch values and validate whether differences between "neighboring color" patches makes sense. Several times a week we flag a measurement that is dubious because of variation.

Patch sizes below 6.6 mm or so on the iSis are more susceptible to spurious readings. Mismeasurements due to angled papers are easier to detect than errors from the timing resolution you mentioned above.

That said, some printers do have variations on how they lay down color. When we start evaluating a particular printer family this is something we check. Depending on the printer we'll either add duplicate patches or use some flavor of radial basis function to derive a correction matrix.

I'm an empirical guy, so I prefer to confirm measurements on hardware less prone to orientation variability than the iSis. If you want, shoot me a PM or email and we can measure one of your targets on a Spectroscan.

Lumpy bumpy RGB output behavior has been an Epson trademark for nearly two decades. Each model improves upon the last, but I wonder whether Epson never resolved the issue completely because doing so would be tacit admission there was a problem in the first place. ABW mode gave a clean slate to the software team.

The spurious peak at 480 nm in iSis M2 measurements moves around somewhat depending on the paper stock and brightener levels for Epson x900 printers. There are two approaches you can use here. First, measure in M1 mode with the caveat that your prints are only going to match when viewed in an ISO 3664:2009 compliant viewing booth. The other is to again measure in M1 but use profiling software with useful software-based optical brightener compensation. Yes, I know i1P has the manual OBC module but its utility is limited at best in the real world.

Ethan,
Isis and I1Pro 2 both purport to measure M0, M1, and M2 but they operate differently with slightly different results. The I1Pro 2 measures M1 by default, Isis M2 (uV cut) by default. Then they illuminate with a uV LED and measure the fluorescence. The I1Pro 2 calculates M2 by subtracting some portion of the uV response from the M1 default and adding in some to determine M1.  Isis adds in uV response to come up with M0 and M1 from its default, M2.

I've been able to deal with the Isis wheel track issue by using 10.5mm patches and aligning the paper so the wheel tracks run near/on the patch transitions. I find the improvements mostly affect some Baryta luster paper that with a surface that tends to crush a bit on the tracks. As for intrinsic variation in a paper without significant track marks, I find printing duplicate patches then sorting the outliers is more effective. I'm pretty well automated for either as needed.
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on December 25, 2017, 11:27:32 pm
After a bit of experimenting I decided to create a patch set with a much larger set of neutral and near neutral patches. I combined this with a standard grid of 9x9x9 (729 patches). The near neutrals comprised of 1546 patches.

I duplicated the near neutral patches and added the standard grid patches and randomized the set. This made 4 pages of letter size Isis targets.

I made profiles for both Canson Rag Photo Matte and Costco Glossy and printed several targets that comprised of the Colorchecker colors and a neutral ram from L*=2.5 to 100 in steps of 2.5. Each had 2 additional duplicates that were also randomized. Since each set had 64 patches there were a total of 192 patches. These targets were in  16 bit Adobe RGB tif files Isis readable and were printed using Absolute Colorimetric Intent. Only in gamut colors were analyzed.

The results finally eliminated the Costco paper perturbations I was seeing in *b and the neutrals at last had lower dE00s than the more saturated colorchecker patches. Average dE00 for neutrals was .42 on each of the neutral colors and about .72 for ColorChecker's colored patches.

The Canson Rag Photo was even better, coming in at about dE00=.34 for the Colorchecker colors and dE00=.32 for the neutral ramp. So for the Matte paper this particular target patch set does a good job on both color patches and neutrals.

One note of interest is that the dE00 variation for neutrals exceeded the dE76 while the reverse is true for colored patches. This is one of the more curious aspects of color perceptual sensitivity. It's easier to see differences in color when the colors are neutral to near neutral and much more difficult as the colors become more saturated.


These are the results from the Costco Glossy paper which has more variation than good smooth matte papers:

2 Printouts, Epson 9800, Gostco Glossy
ColorCheckerColors dE00: 0.76, dE76: 1.35,   Neutrals (Steps of 2.5 L*) dE00: 0.52, dE76: 0.43
ColorCheckerColors dE00: 0.64, dE76: 1.09,   Neutrals (Steps of 2.5 L*) dE00: 0.40, dE76: 0.35
ColorCheckerColors dE00: 0.58, dE76: 0.91,   Neutrals (Steps of 2.5 L*) dE00: 0.40, dE76: 0.37

ColorCheckerColors dE00: 0.83, dE76: 1.54,   Neutrals (Steps of 2.5 L*) dE00: 0.51, dE76: 0.44
ColorCheckerColors dE00: 0.65, dE76: 1.18,   Neutrals (Steps of 2.5 L*) dE00: 0.36, dE76: 0.31
ColorCheckerColors dE00: 0.55, dE76: 0.87,   Neutrals (Steps of 2.5 L*) dE00: 0.36, dE76: 0.34


And these are the results from the Canson Rag Photo Matte:
3 Printouts, Epson 9800, Epson Enhanced Matte
ColorCheckerColors dE00: 0.33, dE76: 0.50,   Neutrals (Steps of 2.5 L*) dE00: 0.29, dE76: 0.26
ColorCheckerColors dE00: 0.35, dE76: 0.51,   Neutrals (Steps of 2.5 L*) dE00: 0.30, dE76: 0.27
ColorCheckerColors dE00: 0.34, dE76: 0.51,   Neutrals (Steps of 2.5 L*) dE00: 0.31, dE76: 0.28

ColorCheckerColors dE00: 0.32, dE76: 0.49,   Neutrals (Steps of 2.5 L*) dE00: 0.31, dE76: 0.29
ColorCheckerColors dE00: 0.34, dE76: 0.49,   Neutrals (Steps of 2.5 L*) dE00: 0.31, dE76: 0.28
ColorCheckerColors dE00: 0.35, dE76: 0.52,   Neutrals (Steps of 2.5 L*) dE00: 0.33, dE76: 0.31

ColorCheckerColors dE00: 0.35, dE76: 0.51,   Neutrals (Steps of 2.5 L*) dE00: 0.34, dE76: 0.32
ColorCheckerColors dE00: 0.36, dE76: 0.56,   Neutrals (Steps of 2.5 L*) dE00: 0.32, dE76: 0.29
ColorCheckerColors dE00: 0.35, dE76: 0.54,   Neutrals (Steps of 2.5 L*) dE00: 0.33, dE76: 0.30


Each of the printouts (Two Costco, Three Canson) had 64 total colors duplicated 3 times, hence each of the groups of 3 lines. While the groups were randomized, the same tif file was used for each of the 5 printouts.

I have attached the Isis ready test 192 patch Adobe RGB tif file as well as the CGATS file of the 4 page Isis target from which the profile targets were made.


Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Brad P on January 25, 2018, 10:44:30 pm
Doug, thanks as always for sharing your thoughts.  Just printed out the patches successfully on my Z and waiting to scan. 

Interesting thinking on all the near neutrals - quite a different approach from other patch sets I’ve been using.  I imagine images with this would best be printed in RelCol if possible (rather than P) and that you might have taken this approach even with another spectro for the reasons you develop here.

Addendum later today:  Thinking about it all and watching my patch set dry, actually maybe perceptual would benefit too.  Certainly B/W would be assisted over a more “standard” patch set with the same number of patches.   The neutrals and white/black transitions seem across the luminosity range to be pretty well nailed down by this approach.... hmm.   The proof of the pudding will be in the printing.   
Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Doug Gray on January 26, 2018, 03:05:38 pm
Doug, thanks as always for sharing your thoughts.  Just printed out the patches successfully on my Z and waiting to scan. 

Interesting thinking on all the near neutrals - quite a different approach from other patch sets I’ve been using.  I imagine images with this would best be printed in RelCol if possible (rather than P) and that you might have taken this approach even with another spectro for the reasons you develop here.

Addendum later today:  Thinking about it all and watching my patch set dry, actually maybe perceptual would benefit too.  Certainly B/W would be assisted over a more “standard” patch set with the same number of patches.   The neutrals and white/black transitions seem across the luminosity range to be pretty well nailed down by this approach.... hmm.   The proof of the pudding will be in the printing.

Yes, to the degree Rel Col benefits, Perc. will too. Not that it will be better per se, but Perc. tables are built from the same info and reflect, with I1Profiler, relatively small changes. The tone curve has a very slight "S" curve and the chromaticity is rounded near the gamut edges. As for the neutral renderings. There are sliders in I1Profiler that lets you adjust the way the neutrals are rendered. Generally, the default  the white point is gradually brought in from the paper white to D50 neutral around L*=50. This differs from Rel. Col. which maintains the same paper white point chromaticity coordinates as much as possible down to the DMax. Chromaticity is set by the xy coordinates in the familiar CIEXY horseshoe gamut. You will likely see little effect as the HP spectro is an M2 spectro and most of the Perceptual neutral shift shows up with high OBA media.

Keep in mind that this patch set was what finally made my 9800 printer print neutrals very smoothly without the large b* changes due to ink shifts around L*=80+ I suspect the HP or most other printers don't suffer from this. However, the inclusion ofsuch a large number of neutrals does benefit things because the Delta E 2000 metric is far more sensitive to slight hue/saturation shifts on or near the neutrals.  In most colors Delta E 2000 is much smaller than dE76 but not near neutrals. Especially as saturation goes beyond about 30 (sqrt(a^2+b^2) where dE00 typically gets much smaller than dE76. There can be a difference between dE00 and dE76 of over 4 to 1.  So it's arguably desirable to have a heavier patch concentration of near neutrals than further out.

Title: Re: Starting a search for a smaller, but more optimal, profile patch set
Post by: Brad P on January 27, 2018, 03:12:57 pm
Doug - just a quick note to say I’m somewhat blown away by the absolutely visible positive difference your patch set is making to an image I’m have been pretty intensively working on, and that difference over a more equally weighted somewhat larger i1P generated 4357 patch set.  Soft proofing only in P so far, the colors look more realistic and like a dose of clarity had been provided both in relcol and perceptual proofs. Working now on a device link print.  So anyway, it’s not just the Epson.