Pages: 1 [2] 3   Go Down

Author Topic: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests  (Read 4532 times)

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #20 on: July 23, 2019, 03:31:49 pm »

I've found a way to evaluate profile quality without a two step process. Eliminating the second step means I don't have to create in gamut colors which first requires profiles from the first step and then print charts with separate colors for each of the profiles being tested which uses a lot of paper.

The technique is quite simple and I should have thought of this before.

1. Create patch sets that contain all the patches for each desired patch count as well as additional near neutrals which are added into the larger patch counts.

2. Create about a thousand random RGB triplets which includes 200, near device neutral values.

3. Combine, randomize, and print all patches. This eliminates bias associated with printer warmup or ink jiggling half way through the print job. This also produces identical images and RGB sets for testing any printer.

4. Scan the targets. De-randomize and extract the RGB/Lab pairs for each of the patch sets.

5. Create Argyll profiles for each of the patch sets using colprof and the defaults except for -qh (high quality profile).

Evaluate the statistics of the thousand, random RGB triplets. Since these are independent of the patches used for creating the profiles, they can be used to evaluate the quality of each of the generated profiles w/o requiring a second printing of in gamut colors.  By definition, device RGB values are in gamut. Each represents printable colors.

Except for printing and scanning, the entire process requires less than half an hour per printer/paper. It requires 8 US Letter pages of 957 patches each which includes all the patches for each profile as well as about a thousand independent patches for evaluating the profile.

Attached are the 8 pages of targets and CGATs file of the RGB values. The listing below is each of the 12 profiles generated for each printer using Costco Glossy paper. Other glossy type papers show virtually identical results and the differences between printers remain pretty stationary. Matt paper (not shown) shows a significant improvement and is also stationary between the MP101 Canon Matte (lots of OBA) and the Canson Photographique (No OBA).

Descriptions:
PchCnt: Patch count used to create profile
PchdE: Ave dE of the profile patches. This is similar to the average of the dE's reported by I1Profiler for each patch.
Ave dE:  This is the overall dE average on the independent patches.
LowL,  MidL,  HighL: These are the average of colors L* < 35, L* between 35 and 65, and L* over 65. Interesting because different printers show different error averages depending on luminosity.

As for the profiles, the number after the "M" is the number of grid points on one axis. For instance the 125 patch count profile uses 5x5x5 point grids. and would have the profile suffix be "M5."  Some of the profiles also have a "P" after the number. These indicate packed grids. These tend to produce slightly better quality relative to their increase in size. Additionally, a "N" appears after a few of the larger profiles. These profiles have 134 additional near neutral, staggered RGB triplets. For instance M12PN is a profile with 12x12x12 points and an additional 134 near neutrals. One can see that the near neutrals significantly reduce the dE average errors on the neutral colors. While not shown, they improve near neutral performance to roughly the same level for any of the patch counts.

Lastly, the profiles ending in "MALL" have the full set of RGB patches from all of the smaller patch sets. In addition, they have another 500 or so near neutrals. The additional near neutrals shows minimal improvement in the performance of the profiles' near neutral accuracy in comparison to adding the 134 near neutrals.

I also did a few tests using the same data sets with I1Profiler and Profile Maker 5. PM5 was significantly inferior with dE averages at least 25% higher and many cases exceeded 50%. OTOH, I1Profiler was quite close overall. The accuracy on the color patches was slightly better with I1P while Argyll had the advantage with the near neutrals. It is, however, a PITA to make I1P profiles while automating the Argyll workflow was a piece of cake.




Printer P1000
Colors
PchCnt PchdE     Ave dE   LowL  MidL  HighL
  125  0.34      1.01     1.30  1.06  0.68  Printer-P1000_M5.icm
  216  0.35      0.84     1.04  0.89  0.58  Printer-P1000_M6.icm
  343  0.36      0.72     0.88  0.74  0.53  Printer-P1000_M7.icm
  512  0.34      0.62     0.75  0.64  0.48  Printer-P1000_M8.icm
  729  0.34      0.57     0.69  0.57  0.44  Printer-P1000_M9.icm
  855  0.36      0.53     0.64  0.53  0.40  Printer-P1000_M8P.icm
 1241  0.36      0.50     0.62  0.50  0.39  Printer-P1000_M9P.icm
 1728  0.38      0.51     0.65  0.49  0.39  Printer-P1000_M12.icm
 1859  0.38      0.50     0.63  0.49  0.38  Printer-P1000_M12N.icm
 3059  0.37      0.45     0.58  0.43  0.35  Printer-P1000_M12P.icm
 3188  0.37      0.45     0.57  0.43  0.35  Printer-P1000_M12PN.icm
 6616  0.34      0.43     0.54  0.42  0.34  Printer-P1000_MALL.icm

NearNeutrals
PchCnt           Ave dE   LowL  MidL  HighL
  125            1.09     1.73  0.79  0.76  Printer-P1000_M5.icm
  216            1.04     1.70  0.77  0.65  Printer-P1000_M6.icm
  343            1.01     1.57  0.81  0.64  Printer-P1000_M7.icm
  512            0.83     1.21  0.68  0.61  Printer-P1000_M8.icm
  729            0.72     1.02  0.59  0.55  Printer-P1000_M9.icm
  855            0.65     0.88  0.59  0.49  Printer-P1000_M8P.icm
 1241            0.64     0.88  0.56  0.48  Printer-P1000_M9P.icm
 1728            0.64     0.98  0.52  0.44  Printer-P1000_M12.icm
 1859            0.46     0.60  0.43  0.36  Printer-P1000_M12N.icm
 3059            0.57     0.78  0.49  0.43  Printer-P1000_M12P.icm
 3188            0.46     0.58  0.43  0.35  Printer-P1000_M12PN.icm
 6616            0.40     0.48  0.38  0.34  Printer-P1000_MALL.icm


Printer 9800
Colors
PchCnt PchdE     Ave dE   LowL  MidL  HighL
  125  0.62      1.63     1.63  1.47  1.81  Printer-9800_M5.icm
  216  0.55      1.15     1.30  0.97  1.16  Printer-9800_M6.icm
  343  0.51      0.96     1.05  0.85  0.98  Printer-9800_M7.icm
  512  0.46      0.82     0.94  0.74  0.78  Printer-9800_M8.icm
  729  0.46      0.70     0.83  0.64  0.63  Printer-9800_M9.icm
  855  0.45      0.67     0.78  0.60  0.63  Printer-9800_M8P.icm
 1241  0.46      0.59     0.72  0.56  0.50  Printer-9800_M9P.icm
 1728  0.44      0.58     0.69  0.56  0.47  Printer-9800_M12.icm
 1859  0.44      0.57     0.69  0.56  0.46  Printer-9800_M12N.icm
 3059  0.44      0.52     0.63  0.51  0.42  Printer-9800_M12P.icm
 3188  0.42      0.51     0.61  0.51  0.40  Printer-9800_M12PN.icm
 6616  0.40      0.49     0.60  0.48  0.38  Printer-9800_MALL.icm

NearNeutrals
PchCnt           Ave dE   LowL  MidL  HighL
  125            1.82     1.55  1.49  2.41  Printer-9800_M5.icm
  216            1.38     1.29  1.11  1.74  Printer-9800_M6.icm
  343            1.04     0.90  0.89  1.33  Printer-9800_M7.icm
  512            0.99     0.99  0.89  1.10  Printer-9800_M8.icm
  729            0.73     0.84  0.61  0.75  Printer-9800_M9.icm
  855            0.76     0.79  0.71  0.80  Printer-9800_M8P.icm
 1241            0.63     0.68  0.61  0.61  Printer-9800_M9P.icm
 1728            0.70     0.73  0.64  0.72  Printer-9800_M12.icm
 1859            0.50     0.55  0.52  0.43  Printer-9800_M12N.icm
 3059            0.57     0.64  0.59  0.48  Printer-9800_M12P.icm
 3188            0.47     0.54  0.50  0.38  Printer-9800_M12PN.icm
 6616            0.41     0.48  0.42  0.34  Printer-9800_MALL.icm


Printer 9500
Colors
PchCnt PchdE     Ave dE   LowL  MidL  HighL
  125  0.49      1.51     1.72  1.36  1.45  Printer-9500_M5.icm
  216  0.49      1.15     1.31  1.03  1.10  Printer-9500_M6.icm
  343  0.45      0.93     1.01  0.86  0.92  Printer-9500_M7.icm
  512  0.42      0.81     0.95  0.74  0.74  Printer-9500_M8.icm
  729  0.42      0.70     0.85  0.64  0.61  Printer-9500_M9.icm
  855  0.42      0.64     0.83  0.59  0.51  Printer-9500_M8P.icm
 1241  0.43      0.58     0.74  0.54  0.46  Printer-9500_M9P.icm
 1728  0.42      0.60     0.78  0.55  0.48  Printer-9500_M12.icm
 1859  0.42      0.59     0.75  0.54  0.48  Printer-9500_M12N.icm
 3059  0.40      0.54     0.70  0.49  0.42  Printer-9500_M12P.icm
 3188  0.39      0.53     0.69  0.48  0.42  Printer-9500_M12PN.icm
 6616  0.37      0.50     0.63  0.46  0.40  Printer-9500_MALL.icm

NearNeutrals
PchCnt           Ave dE   LowL  MidL  HighL
  125            2.09     1.38  2.87  2.02  Printer-9500_M5.icm
  216            1.78     1.74  2.49  1.12  Printer-9500_M6.icm
  343            1.42     1.67  1.80  0.80  Printer-9500_M7.icm
  512            1.23     1.69  1.37  0.63  Printer-9500_M8.icm
  729            1.05     1.55  1.09  0.49  Printer-9500_M9.icm
  855            0.84     1.27  0.81  0.44  Printer-9500_M8P.icm
 1241            0.73     1.16  0.67  0.36  Printer-9500_M9P.icm
 1728            0.79     1.27  0.75  0.36  Printer-9500_M12.icm
 1859            0.51     0.70  0.53  0.30  Printer-9500_M12N.icm
 3059            0.60     0.96  0.53  0.31  Printer-9500_M12P.icm
 3188            0.49     0.63  0.54  0.30  Printer-9500_M12PN.icm
 6616            0.48     0.52  0.64  0.30  Printer-9500_MALL.icm
« Last Edit: July 23, 2019, 04:02:39 pm by Doug Gray »
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #21 on: July 23, 2019, 07:27:04 pm »

Just added MP101 (Canon Matte) on the Pro1000 to the group. Results show the expected improvements over glossy types:

Printer P1000M
Colors
PchCnt PchdE     Ave dE   LowL  MidL  HighL
  125  0.19      0.67     0.77  0.69  0.55  Printer-P1000M_M5.icm
  216  0.19      0.56     0.61  0.59  0.50  Printer-P1000M_M6.icm
  343  0.20      0.48     0.51  0.49  0.43  Printer-P1000M_M7.icm
  512  0.19      0.44     0.46  0.44  0.40  Printer-P1000M_M8.icm
  729  0.19      0.42     0.45  0.41  0.40  Printer-P1000M_M9.icm
  855  0.21      0.38     0.40  0.38  0.36  Printer-P1000M_M8P.icm
 1241  0.21      0.36     0.38  0.35  0.34  Printer-P1000M_M9P.icm
 1728  0.22      0.35     0.37  0.35  0.33  Printer-P1000M_M12.icm
 1859  0.23      0.34     0.36  0.35  0.32  Printer-P1000M_M12N.icm
 3059  0.23      0.31     0.31  0.32  0.30  Printer-P1000M_M12P.icm
 3188  0.23      0.31     0.31  0.32  0.30  Printer-P1000M_M12PN.icm
 6616  0.22      0.29     0.28  0.30  0.28  Printer-P1000M_MALL.icm
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #22 on: July 24, 2019, 07:43:06 am »

Thanks for providing this data.  Unfortunately, those of us who still do manual scans with an i1 Pro are not in a position to do such a high patch count.  I do a two step profiling with Argyll, a two page preconditioning profile followed by a four page (1848 patches) with added neutrals using the new command that Graeme added a couple of months back.  This is a manageable approach in terms of manually reading patches and average errors are in the 0.2 to 0.3 range.  Standard test prints come out with accurate color rendition which is the bottom line.
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #23 on: July 24, 2019, 01:49:32 pm »

Thanks for providing this data.  Unfortunately, those of us who still do manual scans with an i1 Pro are not in a position to do such a high patch count.
Actually the goal of this is to minimize the number of patches people need to make good profiles. I absolutely do not encourage the highest possible patch count making profiles.

By showing the dE00 averages as patch count increases for each printer/paper type, one cahn select an appropriate number of patches for an initial profile. Then adding additional patches may be used to further improve things. For instance the Pro1000 needs considerably fewer patches to get the same quality profile as the 9800 and especially the 9500 II.
Quote

I do a two step profiling with Argyll, a two page preconditioning profile followed by a four page (1848 patches) with added neutrals using the new command that Graeme added a couple of months back.  This is a manageable approach in terms of manually reading patches and average errors are in the 0.2 to 0.3 range.  Standard test prints come out with accurate color rendition which is the bottom line.

That should work quite well. BTW, how do you determine average dE's of your profiles?
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #24 on: July 24, 2019, 03:17:40 pm »

That should work quite well. BTW, how do you determine average dE's of your profiles?
I use the 'profcheck' utility in Argyll and direct the output to a logfile.  It gives the data read data for all the patches along with the deviation from what it should be.  At the end of the Excel spreadsheet it lists the max error, average error, and rms error.  You can easily sort the patch set according to error value and determine whether you need to re-read any of the patches.  I always do two readings and average those so unless there is a very bad outlier (>2), I don't bother.  It also allows for the creation of a 3-D profile that can be used for comparison purposes.
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #25 on: July 24, 2019, 04:36:01 pm »

I use the 'profcheck' utility in Argyll and direct the output to a logfile.  It gives the data read data for all the patches along with the deviation from what it should be.  At the end of the Excel spreadsheet it lists the max error, average error, and rms error.  You can easily sort the patch set according to error value and determine whether you need to re-read any of the patches.  I always do two readings and average those so unless there is a very bad outlier (>2), I don't bother.  It also allows for the creation of a 3-D profile that can be used for comparison purposes.

Thanks Alan,

This is a good tool provided an independent set of RGB patches is compared against the profile rather than the patches used to create the profile. By way of explanation, let me quote Graeme from profcheck:

Quote
profcheck provides a way of checking how well an ICC profile conforms to the test sample data that was used to create it (or other test samples that are from the same device). This is the same sort of check done within the profile making tool (colprof), but having a separate tool provides some flexibility.  The absolute forward table in the profile is used to create PCS values from the sample points, and the profiles PCS value then compared to the PCS values of the measured sample points. Note the lower delta E values are not always a better measure of how good a profile is. The aim of a profile is to model the underlying characteristics of a device, not to slavishly reproduce the sampled data point values. Sampled data point values contain device variation and instrument reading inaccuracies, and a good profiler will try and filter out this noise, resulting in some deliberate differences between the profile and the sample points used to create it.

Profcheck does an AtoB table lookup on the RGB values and compares the result, in Lab, to the measured Lab values. This corresponds exactly to the PchdE columns in my posts as only the patches used to create each profile are used.  This tends to slightly understate profile accuracy dEs since they are the same patches used to create the profile.  Graeme's caution is particularly apropos because, for instance, making a profile of an evenly spaced grid  in I1Profiler will see very little smoothing and produce what looks like excellent dEs (under .1 !) from the same data set that colprof might show .3 dE. But that's because colprof smooths the A2B tables to reduce the impact of measurement noise/printer variation. But these very low values out of I1PRofiler profiles do not indicate its profiles are better or achieve anywhere near that kind of accuracy.

The other columns show dE00 values based on an independent, large random set of RGB values and so are more representative of the dE00 error one would have printing arbitrary, but in-gamut colors.

The breakthrough I made in my approach was to combine and scramble all the patches used by the different profiles I tested as well as a completely independent set which are used to determine profile accuracy.

One really cool side effect is that this same data can be plugged into any profile generating program so it's also a way to compare different products w/o the vagaries of printer variation, warmup, time since last used, temperature, drying time, humidity, and so on. These things would, of course, affect the relative values shifting dE's slightly higher or lower but they don't change the differences between the different profiles or profile making software.
« Last Edit: July 24, 2019, 11:38:47 pm by Doug Gray »
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #26 on: July 24, 2019, 04:46:53 pm »

Thanks Alan,

This is a good tool provided an independent set of RGB patches is compared against the profile rather than the patches used to create the profile. By way of explanation, let me quote Graeme from profcheck:


The breakthrough I made in my approach was to combine and scramble all the patches used by the different profiles I tested as well as a completely independent set which are used to determine profile accuracy.

One really cool side effect is that this same data can be plugged into any profile generating program so it's also a way to compare different products w/o the vagaries of printer variation, warmup, time since last used, temperature, drying time, humidity, and so on. These things would, of course, affect the relative values shifting dE's slightly higher or lower but they don't change the differences between the different profiles or profile making software.
What would you say is the optimal number of patches to check the profile against.  I only use Argyll and an generate a random patch set and then see how that compares to the profile in question. 
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #27 on: July 24, 2019, 05:12:26 pm »

What would you say is the optimal number of patches to check the profile against.  I only use Argyll and an generate a random patch set and then see how that compares to the profile in question.

You can get a reasonably close estimate of overall dE with a single page or less of patches. 100 random device RGB patches should produce an average dE that is within +/-.06 dE of the actual average dE and 400 patches will drop that to about +/-.03. The important thing is that they differ from those used to create the profile. However, you may see more variation than that just printing the same patches a second time. My 9500 II is particularly bad at that and can produce .2 dE differences printing the exact same chart but on different days.
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #28 on: July 24, 2019, 05:33:23 pm »

You can get a reasonably close estimate of overall dE with a single page or less of patches. 100 random device RGB patches should produce an average dE that is within +/-.06 dE of the actual average dE and 400 patches will drop that to about +/-.03. The important thing is that they differ from those used to create the profile. However, you may see more variation than that just printing the same patches a second time. My 9500 II is particularly bad at that and can produce .2 dE differences printing the exact same chart but on different days.
It's not clear to me that I could prepare a totally random set of patches using Argyll targen command.  I'll need to look carefully at the commands.  Do you use MatLab to create synthetic color patch sets?
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #29 on: July 24, 2019, 05:43:41 pm »

It's not clear to me that I could prepare a totally random set of patches using Argyll targen command.  I'll need to look carefully at the commands.  Do you use MatLab to create synthetic color patch sets?

Yes, MATLAB for virtually everything including making charts. Fast and powerful with lot's of built in functions for analysis. For instance I can read a 4kx3k image in ProPhoto RGB, convert it to sRGB, and get a histogram of the dE2000 differences of all 12M pixels in about 10 seconds with about 4 lines of code.

I made a couple of random RGB CGATs patch sets for 100 and 400 patch counts and attached them here.
Logged

GWGill

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 595
  • Author of ArgyllCMS & ArgyllPRO ColorMeter
    • ArgyllCMS
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #30 on: July 24, 2019, 10:10:16 pm »

The breakthrough I made in my approach was to combine and scramble all the patches used by the different profiles I tested as well as a completely independent set which are used to determine profile accuracy.
So, a cross validation type approach ?

If time is no object, then it's possible to do multiple passes of "leave N out" cross validation on such a data set, which can give some idea as to the expected accuracy of a model. Ideally it could also be used to tune the model parameters, but making such an approach computationally feasible is challenging.
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #31 on: July 25, 2019, 01:49:00 am »

So, a cross validation type approach ?

If time is no object, then it's possible to do multiple passes of "leave N out" cross validation on such a data set, which can give some idea as to the expected accuracy of a model. Ideally it could also be used to tune the model parameters, but making such an approach computationally feasible is challenging.

Interesting idea. Does sound computation intensive. I'm attached to the idea of using random, independent RGB patch sets to get the profile accuracy.  Easy to get the AtoB Lab estimates, but since different profile generators turn out to have different algos for the A2B and B2A processes, the only way to get an accurate assessment of different profiler packages is to find the Lab value that, after going through the B2A, yields the RGB values of the independent test patch set. This, after all, is what determines the actual print accuracy since prints only use the B2A tables.

I've been experimenting with Matlab's Optimization toolkit but it is slow. Matlab is fast with arrays of Lab and RGB sets but the optimizer uses individual triplets so it can't vectorize things. Takes about 10 minutes to go through the independent RGB set to find the Lab values that match the RGB sets going thorough the B2A's. And that's for each of the profiles.  Not awful but it can take many hours for the profile data sets I posted.  So I'm looking at another approach that may be faster. Hoping to do in strictly in Matlab but may wind up writing a mex addin.

Added:
Graeme, cross checking against all the unused patches may be doable. My revised alternate approach to MATLAB's optimization algo works well. It find the Lab values that produce an RGB/Lab pair to  about 1/16th bit (based on 0 to 255) on 1000 different RGB values in about 1 sec. And it's not all that long a program. I'm a bit wary. Even though the sample set of excluded RGB values would larger than the added random set, there may be undesired correlates as they are heavily patterned. I had that problem some weeks ago when making Lab sets to print and measure. I used a fixed deltas on the L*, a*, and b* to create what I thought would be a good spread. It wasn't. Changing the deltas changed the accuracy averages because then tended to align in the same places on the LUT cubes. A particular problem on I1Profiler profiles since they don't use any curves prior to the LUT lookups.

Now I can compare vendor tools w/o concern re A2B variations since it now only uses B2A.
« Last Edit: July 25, 2019, 03:28:29 pm by Doug Gray »
Logged

GWGill

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 595
  • Author of ArgyllCMS & ArgyllPRO ColorMeter
    • ArgyllCMS
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #32 on: July 25, 2019, 09:30:18 pm »

Matlab is fast with arrays of Lab and RGB sets but the optimizer uses individual triplets so it can't vectorize things. Takes about 10 minutes to go through the independent RGB set to find the Lab values that match the RGB sets going thorough the B2A's. And that's for each of the profiles.  Not awful but it can take many hours for the profile data sets I posted.  So I'm looking at another approach that may be faster. Hoping to do in strictly in Matlab but may wind up writing a mex addin.
xicclu -fib may do what you want a lot faster (i.e. msec per sample, once initialized).
Quote
Graeme, cross checking against all the unused patches may be doable.
Depends what you mean by that !

What I meant was the process of:

  Remove N samples from the measurement set
  Build a profile from the remaining samples
  Check the fit of the N samples to the model (i.e. A2B).
  Repeat.

This could be done systematically (i.e. say divide the set randomly into 5 parts, then do
5 profiles with one part left out.)

Or it could be done completely randomly:
  Remove N% of the samples from random positions.
  Build a profile from the remaining samples
  Check the fit of the N% samples to the model (i.e. profile).
  Repeat indefinitely.

As I understand it, the resulting statistics are not identical to the underlying accuracy, but are related to them.
Logged

RobWignell

  • Newbie
  • *
  • Offline Offline
  • Posts: 28
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #33 on: August 06, 2019, 08:28:04 am »

I have been on holidays for a couple of weeks and didn't get around to sending the prints before I went away.

Will they still be useful?  Do you have a preference for a print of the 8 Medium patch sets or the 4 Tiny patch sets?

I have two options for printing: Qimage One (Q1) or Roy Harrington's Print Tool.  I believe that they produce the same results and are both designed for printing with or without colour management.  I use international A4 (210 x 290 mm) size paper which is slightly longer and slightly narrower than the Letter standard used in USA.  A test print with both sets of software gives patch set dimensions (for patches but not guides) of 164mm wide by 186mm high.  This makes the 29 columns 5.655mm wide and the 33 rows 5.666 mm high.  The distance from the leading edge of the paper to the black bar at the top is 45 mm and the distance from the black bar at the bottom of the patch set to the bottom of the paper is 43 mm.  Are any of these measurements a deal breaker? 

The extra page of correspondence on this topic since I last posted is rich with information.  It seems like you have made progress. 

Your observation of the diminishing return on improved DE as a function of patch numbers encourages me to use smaller patch sets as a starting point for profiling different papers. 

I have combined your two CGATS files and added 127 extra near neutral steps to create a 627 set CGATS file.  ColorPort ingests this file and creates patch sets for numerous older spectros, including my DTP70.  It also lets me set the patch size so that it almost fills my A4 page size.

I will report on results using Argyll colprof defaults plus your -qh and -r0.3 settings using this patch set.
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #34 on: August 06, 2019, 05:53:37 pm »

I have been on holidays for a couple of weeks and didn't get around to sending the prints before I went away.

Will they still be useful?  Do you have a preference for a print of the 8 Medium patch sets or the 4 Tiny patch sets?

I have two options for printing: Qimage One (Q1) or Roy Harrington's Print Tool.  I believe that they produce the same results and are both designed for printing with or without colour management.  I use international A4 (210 x 290 mm) size paper which is slightly longer and slightly narrower than the Letter standard used in USA.  A test print with both sets of software gives patch set dimensions (for patches but not guides) of 164mm wide by 186mm high.  This makes the 29 columns 5.655mm wide and the 33 rows 5.666 mm high.  The distance from the leading edge of the paper to the black bar at the top is 45 mm and the distance from the black bar at the bottom of the patch set to the bottom of the paper is 43 mm.  Are any of these measurements a deal breaker? 

The extra page of correspondence on this topic since I last posted is rich with information.  It seems like you have made progress. 

Your observation of the diminishing return on improved DE as a function of patch numbers encourages me to use smaller patch sets as a starting point for profiling different papers. 

I have combined your two CGATS files and added 127 extra near neutral steps to create a 627 set CGATS file.  ColorPort ingests this file and creates patch sets for numerous older spectros, including my DTP70.  It also lets me set the patch size so that it almost fills my A4 page size.

I will report on results using Argyll colprof defaults plus your -qh and -r0.3 settings using this patch set.

Yes, please use the target files at this post. While it's 8 US letter pages, lot's more information can be derived.

https://forum.luminous-landscape.com/index.php?topic=130971.msg1120870#msg1120870
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #35 on: August 08, 2019, 01:58:37 am »

I have added charts for printing with ACPU from Windows and rounded the device RGB near neutrals to 8 bits.

There are two sets of 8 US Letter size pages. The ones with ACPU should be used for printing using Adobe's ACPU utility when running Windows. The ACPU utility in windows shrinks the printed image a bit and these targets expand the image 2% to accommodate ACPU in Windows. Mac's, or other programs that can print images the proper size and w/o color management enabled should use the images without the ACPU tag.

These targets are 8 bit tiffs. The earlier ones were 16 bit tiffs though all the patches except for 192 near device neutrals are actually 8 bits. ACPU is will print them as 8 bits and will result in slightly worse (.04 dE) or so results on near neutrals if used with the earlier chart. That only effected results on near neutrals and had no impact on profile generation which sticks strictly with 8 bit values. Otherwise the charts are the same.


FOR THOSE WITH i1iSis spectros:

Also included are CGATs files to load into the spectro. You can scan them and post or email the results rather than mail the printed targets and I will generate an analysis of the printer/paper.

The CGATs with the trailing "r" can be used to read the charts in reverse. That is by feeding the bottom end in and going from chart 8 down to chart 1. These are really only of use for people with i1iSis scanners setting the patch size to 6x6mm and 957 patches per page. I only need the standard forward measurement file but providing the reverse as well lets my system check for measurement error.
« Last Edit: August 08, 2019, 02:01:55 am by Doug Gray »
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #36 on: August 11, 2019, 01:22:29 am »

All the RGB CGATs files that were used to construct each of the profiles shown here all come from the 8 page target attached in the previous post. RGB and LAB pairs were extracted from the measured charts and used to construct these profiles using Argyll's software. Each of the profiles was used to analyze the deltaE 2000 between 848 measured patches distributed randomly in gamut and 192 near neutrals. I consider near neutral performance important because many prints are in B&W or have large areas that are de-saturated. Human vision is more sensitive to small hue changes near neutrals and this is reflected in the higher sensitivity that delta E 2000 shows even though printers have multiple gray inks to help neutral accuracy.

Each of these charts shows the overall average dE2000 of 848 color patches.  It groups the 192  near neutrals in steps of 10 L*.

Profile Structure
Each of the profiles is constructed with spacing such that there are 5,6,7,8,9, or 12 points on each of the RGB axis. The number of RGB colors are thus 5^3, 6^3, 12^3.  Additionally, so called "packed" grids, where additional RGB colors are placed in the center of each of  cube. This adds exactly (N-1)^3 more patches. Experiments show a small improvement in profile quality even accounting for their increased size as can be seen by looking at the results for 343 patch set (7^3 patches) v the 341 packed set (6^3 + 5^3).

Near Neutrals
These tables also show the impact of adding 3 different, near neutral patch sets. These 3 sets are 130, 260, and 476 in additional, near neutral patches. They show differing impact depending on how large the base, color patch count is. The largest set, immediately helps the near neutral performance of even the smallest patch profiles. However, as the color patch size increases, the differences between the 3 near neutral sets become very small.

Decoding the Profile
The profile names follow a pattern that defines their construction. Their last letters consist of an "M", a Number, an optional "P", and ends with an optional "N", "N5", or "N3". The number following the "M" is the grid space count. For instance, 9 would mean 9^3 RGB patches. The "P" means that an additional set of RGB values are added in the center of each cube. The three, near neutral sets are "N" 134 patches, "N5" 260 patches, and "N3" 476 patches.

The attached CGAT zip files are not scrambled and easily examined in a text editor. The main RGB grid is placed first, the centered grid, if present, second, and the near neutrals, if present, last. They can, if you wish, be used for your own printer profile construction and can be used with Profile Maker 5, Argyll or I1Profiler.

Summary and recommendation
For the best overall accuracy, the M12PN5 profiles with 3314 patches, are best. For excellent B&W work, the M5PN3 and M6PN3 are very good, and small enough to profile w/o lots of paper and time.


Pro1000 with Canon Platinum Pro Glossy Paper
Black Point Lab: 1.89 -0.13  0.13, White Point: 94.83  -1.41  -2.18
Profile performance for standard grid sizes of 5,6,7,8,9,12

        ProfilePatchesAveColAveNeuts    <10  10-20  20-30  30-40  40-50  50-60  60-70  70-80   >80
Printer-P1000PlatPro_M5.icm   125  1.16  1.51      1.60   1.95   2.67   2.20   1.32   1.29   1.26   0.70   0.62
Printer-P1000PlatPro_M6.icm   216  0.89  1.13      1.54   1.85   2.17   1.20   0.70   0.85   0.54   0.65   0.64
Printer-P1000PlatPro_M7.icm   343  0.74  1.06      1.46   1.44   2.14   0.82   1.14   0.72   0.59   0.59   0.63
Printer-P1000PlatPro_M8.icm   512  0.68  0.90      1.36   1.31   1.38   0.76   0.80   0.72   0.58   0.59   0.63
Printer-P1000PlatPro_M9.icm   729  0.61  0.83      1.37   1.19   0.99   0.83   0.65   0.77   0.55   0.51   0.59
Printer-P1000PlatPro_M12.icm  1728  0.51  0.72      1.37   1.04   0.75   0.57   0.61   0.66   0.47   0.45   0.56



Profile performance for standard grid sizes of 5,6,7,8,9,12
including inset grid at center points of cubes

        ProfilePatchesAveColAveNeuts    <10  10-20  20-30  30-40  40-50  50-60  60-70  70-80   >80
Printer-P1000PlatPro_M5P.icm   189  0.86  1.22      1.91   1.39   2.26   1.31   0.73   0.93   0.72   0.84   0.85
Printer-P1000PlatPro_M6P.icm   341  0.68  0.94      1.52   1.28   1.53   0.82   0.78   0.75   0.59   0.51   0.63
Printer-P1000PlatPro_M7P.icm   559  0.60  0.85      1.40   1.22   1.19   0.74   0.86   0.66   0.51   0.47   0.62
Printer-P1000PlatPro_M8P.icm   855  0.57  0.76      1.28   1.02   0.79   0.71   0.65   0.79   0.54   0.46   0.57
Printer-P1000PlatPro_M9P.icm  1241  0.52  0.73      1.25   0.98   0.76   0.81   0.64   0.68   0.49   0.46   0.52
Printer-P1000PlatPro_M12P.icm  3059  0.45  0.61      1.01   0.76   0.57   0.61   0.57   0.63   0.46   0.40   0.47

As Above but also including 134 near neutrals

        ProfilePatchesAveColAveNeuts    <10  10-20  20-30  30-40  40-50  50-60  60-70  70-80   >80
Printer-P1000PlatPro_M5PN.icm   320  0.79  0.60      1.10   0.65   0.48   0.57   0.62   0.63   0.42   0.42   0.49
Printer-P1000PlatPro_M6PN.icm   472  0.67  0.57      1.06   0.64   0.49   0.55   0.58   0.59   0.38   0.38   0.45
Printer-P1000PlatPro_M7PN.icm   686  0.58  0.57      1.06   0.69   0.44   0.56   0.60   0.59   0.40   0.37   0.46
Printer-P1000PlatPro_M8PN.icm   986  0.56  0.56      1.02   0.61   0.38   0.57   0.61   0.61   0.40   0.39   0.45
Printer-P1000PlatPro_M9PN.icm  1371  0.51  0.57      1.01   0.61   0.39   0.61   0.63   0.60   0.40   0.38   0.47
Printer-P1000PlatPro_M12PN.icm  3188  0.45  0.53      0.90   0.55   0.39   0.57   0.60   0.58   0.39   0.36   0.42

... Including 260 near neutrals

        ProfilePatchesAveColAveNeuts    <10  10-20  20-30  30-40  40-50  50-60  60-70  70-80   >80
Printer-P1000PlatPro_M5PN5.icm   446  0.81  0.61      1.14   0.76   0.49   0.55   0.63   0.56   0.42   0.44   0.52
Printer-P1000PlatPro_M6PN5.icm   598  0.68  0.56      1.07   0.69   0.43   0.53   0.56   0.54   0.37   0.36   0.48
Printer-P1000PlatPro_M7PN5.icm   806  0.59  0.57      1.08   0.68   0.45   0.54   0.61   0.54   0.39   0.36   0.49
Printer-P1000PlatPro_M8PN5.icm  1112  0.56  0.55      0.98   0.65   0.39   0.55   0.58   0.55   0.40   0.38   0.46
Printer-P1000PlatPro_M9PN5.icm  1497  0.51  0.54      0.94   0.61   0.38   0.57   0.58   0.55   0.39   0.36   0.47
Printer-P1000PlatPro_M12PN5.icm  3314  0.45  0.50      0.83   0.51   0.40   0.54   0.55   0.53   0.38   0.35   0.43

... Including 476 near neutrals

        ProfilePatchesAveColAveNeuts    <10  10-20  20-30  30-40  40-50  50-60  60-70  70-80   >80
Printer-P1000PlatPro_M5PN3.icm   662  0.78  0.52      0.87   0.55   0.34   0.54   0.59   0.59   0.38   0.40   0.46
Printer-P1000PlatPro_M6PN3.icm   814  0.67  0.50      0.87   0.53   0.34   0.52   0.56   0.53   0.37   0.35   0.43
Printer-P1000PlatPro_M7PN3.icm  1022  0.58  0.50      0.87   0.55   0.32   0.53   0.56   0.54   0.38   0.34   0.44
Printer-P1000PlatPro_M8PN3.icm  1328  0.56  0.50      0.86   0.50   0.34   0.53   0.56   0.55   0.38   0.37   0.42
Printer-P1000PlatPro_M9PN3.icm  1713  0.51  0.50      0.84   0.55   0.32   0.54   0.57   0.55   0.38   0.36   0.43
Printer-P1000PlatPro_M12PN3.icm  3530  0.44  0.49      0.81   0.47   0.34   0.53   0.57   0.54   0.37   0.35   0.41
Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #37 on: August 16, 2019, 11:21:32 pm »

I notice that you are using a -r setting of 0.3 in Argyll's colprof instead of the default 0.5.  For me, this usually increases the Gamut Volume slightly but can introduce kinks in the gamut model surface, especially in the darker areas.  Have you had this experience?
I'll examine your chart info for this. My 9800 exhibits clumpy behavior while the 9500II and Pro1000 do not. This can be seen my examining a large qty of device space RGB values and round tripping them to Lab PCS and back to device RGB. The 9800 exhibits as well over a difference in RGB values of 50 that print the same Lab value. The math term for this is that device space RGB and in-gamut Lab PCS are not isomorphic. A well designed RGB to inking DPI algorithm will minimize this. That is, different RGB values map more or less smoothly to different Lab values.

However, this does not mean the printer can't print smoothly. My 9800 prints quite smoothly. It does strongly suggest more patches are needed for the same quality profile. And that is the case with the 9800. When wildly different device RGB values print the same Lab value this means that some portion of device RGB is duplicated. In a sense this is just wasted RGB space as it decreases the total number of device RGB values that correspond to a color.
Quote

For experimenting, I use Kodak Ultra Premium High Gloss paper on my Epson 3880 and Epson 7880 (COSTCO stopped selling Kirkland Glossy in Australia a couple of years ago).  I bought the Epson 7880 back from the dead about 18 months ago using cheap 3rd party ink and, once I got it going properly I loaded a set of Cone inks. 

I would be pleased to send you a set of prints on the Kodak paper.

I'm currently processing your target prints. My first cut looking at the gamut is that it's quite close to the 9800 on Costco glossy excepting that the Costco differs on the white and black points.

It would help for getting a database or printer/paper combos if you could provide the printer model you used and driver settings such as bi-directional or uni-directional as well as DPI settings. I've found these make a difference in some areas. Also what ink set you used if it isn't OEM ink.

I'm batch processing a large set of Argyll profiles from your charts. This involves extracting the  RGB/Lab values for each of the different patch sets that are embedded in the profile. Currently running about 30 different profiles. It's all automated on my end but the calculations still take some time, about 10 mins. each for the -qu option on my workstation.

To get a quick read, I ran the data with the -qh option which is much faster, about a minute per profile. Results are quite good. Device space RGB is better mapped much like my 9500II and Pro1000. This gets a better dE00 at lower patch counts than the 9800 which really needs over 1k patches for good results.
« Last Edit: August 17, 2019, 01:27:27 am by Doug Gray »
Logged

RobWignell

  • Newbie
  • *
  • Offline Offline
  • Posts: 28
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #38 on: August 17, 2019, 10:30:28 pm »

Doug Gray
In your post of 16 August, 2019, 11:21:32pm, you asked a couple of questions:
The printer I used for printing the medium set of targets is an Epson 3880.  It is running Epson OEM inks but most have passed their use by date.
I have attached files showing the printer settings:  I printed your targets using PrintTool which I believe gets around any attempts by macOS Mojave version 10.14.5 to interfere with colour (file RW-Epson3880-PrintTool.png);  The next two files show the Epson Print Driver Settings (files RW-Epson3880-PrinterSettings.png and RW-Epson-AdvancedMediaControl.png).


Logged

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2173
Re: Possible Crowdsourcing Printer Profile Optimal Patch Count Tests
« Reply #39 on: August 18, 2019, 06:25:27 pm »

Thanks Rob,

Indeed, The print tool did a good of printing the targets are the correct size. And w/o color management. Further, it seems the tool prints 16 bit tiff images correctly as well. The targets you printed are all 8 bit RGB values except for the additional near neutrals used to test neutral accuracy. Those were inadvertently fractional so produced 16 bit tiff files. This has no impact as my software can be switched for 16 bit tiff files. Even so the differences just rounding are only on the order of .03 dE00 and only affect neutrals in any case. All the target patches and independent color patches are 8 bit resolution.

So when I checked it, it's clearly printing 16 bit equiv. The tool isn't available on Windows but Photoshop can effectively emulate 16 bit by a combination of the null trick and setting the dither option which adds just enough low bit noise to get the same results for an 8 bit driver. Canon purports to offer 16 bit drivers through their Pro1000 plugin but it doesn't actually work. It produces 8 bit prints even when set to 16 bit. But the null trick works just fine on Windows.

I ran your 8 pages of charts though the I1iSis forward and reverse. Turned out one patch was 3.9 dE read reverse v read forward. Upon examining it the color was a close to neutral at L*=81. First time I've seen a large delta that wasn't associated with L* below 20. Appears to be lint, as re-reading the page it was on forward and reverse yielded matching values to a dE < .1.  What's interesting to me is that the erroneous reading lower saturation and L*.  This is the opposite of what happens with low L* read differences. There, the lint increases L* as it reflects more light back to the sensor. For the high L* it seems the lint decreased the reflected print light more than reflected light from the lint hence the reversal.

I re-ran the profile batch set with the labels changed to reflect the 3880 printer. They have finished except for the last profile. I also added a perceptual table based on Adobe RGB. However, I rarely use perceptual and it's not really a defined process. It also takes a lot longer, about 25 minutes per profile.

In any case profiles on your 3880 improve with patch count almost as fast those of the Pro1000 and much faster than the 9800. I'll be posting a link to a zip file you can download for all 31 profiles that resulted. It's going to be about .8 GB and eats into my Onedrive quota so I probably won't leave it up for too long. If that's too large for you to download I can attach just the better ones and will probably attach the best profile here with the test results.

Thanks for contributing. I am happy that your target prints turned out quite well.
Logged
Pages: 1 [2] 3   Go Up