Luminous Landscape Forum

Raw & Post Processing, Printing => Colour Management => Topic started by: guyburns on May 09, 2011, 11:00:46 am

Title: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 09, 2011, 11:00:46 am
I'm trying to generate a Kodachrome profile from a Kodak Kodachrome IT8 target on a Nikon Coolscan V ED. I've actually generated a successful profile, and now that I know how to do it, I want to go back to the start and make sure I have the optimum profile. So I'm scanning the target again – and here's where I've come unstuck.

I assume that the most accurate profile will be based on a scan that is itself the most accurate scan of the target. What I mean is: if you scan a target and it is way off, then the profile-generating software should, in theory, be able to generate an accurate profile – but it may have to make large corrections. My theory (correct me if I'm wrong) is that if you scan a target and the scan is very accurate, you will get a better profile because the corrections required will be smaller.

So, I'm going back to square one, and am trying to get the most accurate scan. That's where the problems lie.

The Coolscan allows a variety of profiles to be applied to the scan, or you can turn off the colour management. Since Nikon gives no information about this in their literature, I have done some tests. I placed the IT8 target in the scanner and scanned it with various settings. Then I opened the image in Photoshop, and checked the assigned profile (Edit > Assign Profile). The Assign Profile window has three options:

A. Don't Colour Manage This Document
B. Working RGB
C. Assign Profile

To ensure we are all on the same wavelength, my understanding is that if the image has an embedded profile, option C is highlighted with the name of the profile. If the image does not have an embedded profile, option B is highlighted, and the image is automatically assigned Photoshop's "Working RGB" profile (in my case sRGB 2.1). Option A is provided in case you want to turn off colour management. I assume that if you select option A, you are seeing the image on screen based solely on it's RGB numbers with no corrections applied.

I scanned my target several times with various options selected:

1. CM on/Apple RGB profile. This scan had an assigned profile which I'll call Apple RGB.

2. CM on/Adobe RGB profile. This scan had an assigned profile which I'll call Adobe RGB.

3. CM on/Nikon sRGB profile. This scan had an assigned profile which I'll call Nikon sRGB.

4. CM on/Scanner RGB profile. When this image was loaded into Photoshop, it arrived with a "Working Profile" of sRGB 2.1 (my system default). I assume this means the scanned image had no embedded profile and PS applied one (the default sRGB).

5. CM (colour management) off. As per 4 – the image arrived in PS with a "Working Profile" of sRGB 2.1 (my system default).

At the end of the scanning, I therefore had 5 images. Which one should I use as the basis for generating a Kodachrome profile? It gets rather complicated. These are the characteristics of the images.

• Scans 1, 2 & 3, with their profiles applied, look identical on screen.
• Scans 4 & 5 look different from each other, and different from 1, 2 & 3.
• Scans 3, 4 & 5 appear identical (compared to themselves) no matter which of the A, B & C options are applied in Photoshop under "Assign Profile". This must mean that the native PS display on my computer, and sRGB, and Nikon sRGB all result in the same colours.
• When I choose "Don't Colour Manage This Document" for all scans, none of the images (as scanned, without profiles) looked the same. All five differed from each other.

Five scans resulted in five different sets of RGB numbers. In the case of Scans 1, 2 & 3, it seems the scanner distorts the RGB values its sees so that when the profile is applied, the result looks the same. The RGB numbers are distorted even when no profile is applied at the scanner stage – Scans 4 & 5 arrived in PS without embedded profiles (PS applied sRGB), but for some reason, the scanner sent different RGB values.

Any suggestions as to which of the scans would be the most accurate? Scans 1, 2 & 3 are probably not in the running because the RGB values have been distorted; so it's down to 4 & 5. But 4 does not have as good a rendition of "Kodak yellow" as 1, 2 & 3, but it does have better definition on the woman's shoulder (her shoulder is quite clearly defined on the slide).

The scans can be downloaded as a zip file (1.6 MB) from within this folder: http://www.mediafire.com/?thogddgfozxi2 (Five Kodachrome IT8 Scans.zip)
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: Ethan_Hansen on May 09, 2011, 11:48:30 am
Although I have a Nikon Coolscan, I admit it's been some years since I even took it out of the box. I'll need to go from memory here, but the process was not that involved.

Enabling the scanner's color management is not what you want. This converts the image using Nikon's default profile to the desired color space. Option #5, CM off, is what you want. This gives the raw (albeit gamma-corrected) RGB values. Photoshop needs to assume some color space for an image to display it properly. With nothing else to go on, PS assigns your working color space. The software you use to build your scanner profile should pay no attention to embedded image profiles. The raw numbers are the best place to start.

Reading your post, it also appears that you are using the Nikon software. If you have not done so already, download an evaluation copy of VVueScan (http://hamrick.com/). This program manages to get the absolute best performance out of film scanners. It also allows extracting the truly raw RGB values read from the scanner and using them to build a profile. Doing so often produces superior results than beginning with gamma-corrected values.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 09, 2011, 04:07:50 pm
Here is an article from Hutch Color that may be helpful: http://www.hutchcolor.com/PDF/Scanning_Guide.pdf
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 09, 2011, 05:37:22 pm
... extracting the truly raw RGB values read from the scanner and using them to build a profile. Doing so often produces superior results than beginning with gamma-corrected values.

Here is an article, though not exactly on point, that may help in producing a "raw" linear scan with the Nikon scanning software.  As Ethan states, try using a "raw" linear scan for profiling, then after you create your profile, scan your film using the same settings (i.e. to produce "raw" linear scans), then assign your profile to your scans in PS. The next step is up to you, but I usually convert my scans to ProPhoto then do the bulk of my adjustments in Adobe Camera Raw.

http://www.colorneg.com/nikonscan.html?lang=en
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 09, 2011, 11:27:23 pm
Thanks for the responses. The "colorneg" link was particularly informative, though not correct in its description of how to set gamma to 1.0. You have to turn off CM first, otherwise gamma correction is grayed.

Now that I have set gamma to 1.0, CM off, I have one more question. The colorneg site suggested activating the "auto exposure for positive film" option, whereas I had already turned that off, thinking that the best results would be obtained if the exposure was constant when scanning the target and when subsequently scanning other slides.

Is it desirable to turn auto exposure on?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 10, 2011, 03:17:54 am
Okay, I've scanned the target at gamma = 1, CM off; generated two profiles (an XYX look-up table, and a gamma-matrix type) and applied them to an image that was scanned with identical scanner settings, and…

1. The colour difference between this profile and several other profiles I have generated using a variety of scanner settings is imperceptible.
2. However, using the gamma = 1 profile results in banding in gradations, banding that is almost non-existent when I applied all the other profiles to the relevant image (i.e profiles applied to an image that was scanned with the same settings that originally generated that particular profile).

I'm using Coca, based on Argyll, to generate profiles.

I'll go back to the idea I posited in my original post: that it is best to start from a target scan that looks on screen to be a close match to the target slide when the target is viewed on a slide viewer. The gamma = 1 scans came out exceptionally dark, but there was nothing wrong with them: when I applied a gamma of 2.2 in PS they looked fine. But it seems to me that if you start with a linear scan there may be problems: (1) large corrections are required by the profile because the image will ultimately be displayed on a monitor that effectively has a gamma of 2.2 [?]; (2) linear scans require more bits to accurately map the luminance range of an image.

I came across an explanation about digitising images in Reproduction of Colour (RWG Hunt). I don't know how applicable this is, but he says (p546, slightly edited for clarity):

Digital imaging systems often operate with 8 bits in each of three channels, making a total of 24 bits. This nymber of bits generates 224 or 16,777,216 different colour signals. The number of colours that can be distinguished by human colour vision is a matter of some debate, but it has been quoted as 10 million… For a 24-bit system to reproduce all these colours, it would have to sample colour space almost uniformly in visual terms. But many systems digitise linearly, and these are notoriously non-uniform perceptually.

A perceptually uniform grey scale which is widely used is that provided by the L* function of the CIELAB colour space… The range of L* values used in imaging depends on the display and the viewing conditions, but the maximum range can be taken as extending from 10 (for a black) to 100 (for a white). If a just-noticeable difference is taken to be one unit of L*, then the number of tonal levels required to avoid spurious contouring in gradually changing areas would be 100-10 = 90, which would need 7 bits… But, when linear signals are used… the number of tonal levels rises to 733, requiring 10 bits.


Hunt certainly knows his stuff, but he is not the best teacher, and he doesn't clearly explain how he arrived at those figures. I've spent a few hours reading some of his other chapters, and this is how he derived those figures. As he describes on p109 (edited for clarity):

…a uniform linear scale of luminance does not represent a uniform visual scale: for example, the apparent difference between two luminances of 10% and 15% is much greater than that between luminances of 70% and 75%. To allow for this, a quantity called CIE 1976 lightness, L*, is calculated as:

 L* = 116 (Y/Yn)0.33 - 16


Where Y is the luminance being examined, and Yn is the luminance of the reference white. Below a certain value of Y/Yn this formula is replaced by another, so that L never goes negative (as I understand it).

A luminance scale of L*, ranging from black (~10) to white (100) requires only 7 bits (128 values possible). Therefore, 8-bits will be more than ample to avoid banding, if the levels are non-linearly spaced (i.e. if gamma is higher than 1.0). Problems arise when you use a linear scale, because depending how you set up the scale, a jump from one level to the next may exceed what humans can detect – and we will see banding. Humans have greatest sensitivity to changes in luminance when the luminance is low. Therefore, to limit banding to being just perceptible, you choose the change in luminance between L*(10) and L*(11) as the basis for the increments between linear levels for the entire range from black to white. What this means, however, is that as the brightness increases, you have a lot of unnecessary levels (requiring more bits). The advantage is: now matter what the luminance (black, white, grey), the change from one level to the next will never be more than just perceptible, and then only in the darker regions of the image. And that is exactly where I saw the banding in my profile generated from a linear scan – in the darker regions.

Now, where did Hunt get his number 733 from? Rearrange the formula, we have:

Y/Yn = ((L* + 16)/116)3

For L* = 10, relative luminance is 0.01126
For L* = 11, relative luminance is 0.01261

a difference of 0.00135. So, to get from a luminance of L*(10) to L*(100), which required 7 bits in non-linear L* space, we would have to go from 0.01126 to 1.0 in linear space, using steps of 0.00135 (otherwise in the darkest areas we would see banding). Number of levels, N, required in linear space:

N = (1.0 - 0.01126)/0.00135 = 732.4 levels.

Therefore 10 bits (1024 possibilities) are required in linear space to avoid banding. My scans were 14 bit, but how many bits does the profiling software use? If less than 10, and if based on a linear IT8 target scan, it would seem to me that banding may sometimes become visible in the darker areas of an image.

I have generated a few dozen test profiles from my six targets (scanned non-linearly), and only on one occasion so far have I noticed banding – and then at an insignificant level – when the profiles were applied to an image. But the first time I applied a profile based on a linear scan to one of my test images, banding was obvious and obtrusive. Just a coincidence, or the result of linear scanning?








Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 10, 2011, 08:40:49 am
Your scanner CCD, like all digital cameras, produces linear data.  That linear data is the starting point, whether you make your gamma adjustment in your scan software, in PS or via a profile.  I’ve never experienced the banding you describe.  Are you sure you’re scanning in 16 bit and starting with a properly exposed scan?

Regarding exposure, so long as you get a proper exposure, it doesn’t matter whether you use auto exposure or manual exposure (using the master analog gain).  Manual exposure is often necessary to get the best exposure.  A good exposure should clip neither the highlights nor the shadows.  As a general rule, follow the “expose to the right” method commonly used for digital cameras.  (Your scanner is just a specialized digital camera.)
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: Ethan_Hansen on May 10, 2011, 02:51:11 pm
[iTherefore 10 bits (1024 possibilities) are required in linear space to avoid banding. My scans were 14 bit, but how many bits does the profiling software use? If less than 10, and if based on a linear IT8 target scan, it would seem to me that banding may sometimes become visible in the darker areas of an image.[/i]

I don't know about Argyll/Coco as far as scanned bit depth goes. This is not only whether the software accepts high-bit input, but whether it makes use of the information. This is a problem with many scanner profilers.

The built-in profiling software for VueScan and SilverFast do not have this limitation, however I have not found their profiles to be of as high quality as, for example, ProfileMaker produced. Profiles are, by their nature, more suited for performing color adjustments that the gross tonal shifts required to shove linear gamma input into gamma 2.2 output. Again, I have never used Argyll for scanner profiling, so I can't say how well it performs here, even if high-bit data are supported.

When you see banding in profiles made from the raw, linear gamma data, it is better to let the scanner software perform the tonal adjustments to move the input from linear to either 1.8 or 2.2 gamma. With the previous generation of Nikon film scanners combined with ProfileMaker 4.something, I had the best results scanning into Wide Gamut RGB and building a profile from there. Newer flavors of Nikon hardware + PMP5.10 allowed using either linear raw data (via VueScan) or NikonScan with CM off, but gamma 2.2 output.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 10, 2011, 03:55:33 pm
It’s possible that Argyll/Coco is creating a faulty profile using linear data, but I suspect that Guy’s banding problem could be the result of a problem somewhere in his workflow; maybe double profiling, incorrect assigning or conversion of the profile or some other cause.  In any event, I doubt that there is any significant difference between (a) first applying a gamma adjustment and then profiling versus (b) profiling directly from the linear data.  I’ve done it both ways.  The results are a little different, but not dramatic.  So, if you’re having a problem using linear data, Guy, go ahead and do your gamma adjustment first, but I’d try keeping CM off.

By the way, does Nikon Scan include an option to use the scanner’s native color space (i.e., no CM) and include a corresponding profile that you can assign in PS?  Just curious.


Profiles are, by their nature, more suited for performing color adjustments that the gross tonal shifts required to shove linear gamma input into gamma 2.2 output.

Interestingly, the Minolta scanning software for my Minolta 5400 includes an option to profile directly from the scanner's linear data. e.g., I can assign the included Minolta linear profile to the 16 bit linear output.  This profile has no problem doing the gamma and color adjustments in one step. I've also used IT-8 targets to make my own profiles directly from the scanner's linear output without any problems. So, at least in my experience, there doesn't seem to be any problem using a profile to perform both the gamma and color adjustments.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: Ethan_Hansen on May 10, 2011, 04:07:45 pm
Dean - I should have clarified the point. It requires additional handling above and beyond the normal number crunching to perform large shifts to tonal values without introducing artifacts. It is possible, and the newer software for producing scanner profiles is adept at this.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 10, 2011, 10:00:17 pm
By the way, does Nikon Scan include an option to use the scanner’s native color space (i.e., no CM) and include a corresponding profile that you can assign in PS?  Just curious.

You have the option to choose no CM, and you have the option to choose Scanner RGB, both come into PS unprofiled, but they have different data and look different.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 10, 2011, 10:09:33 pm
Do you have a profile for Scanner RGB?  I assume Nikon included one with Nikon Scan, and it sounds like the native color space for your scanner. If so, try scanning with Scanner RGB, then when you open the scan in PS assign the Scanner RGB profile.

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 10, 2011, 11:17:31 pm
There does not appear to be a profile for Scanner RGB, but I'll hunt around and see what I can find.

When using Coca, there are 4 options for generating a profile:

• Lab clut
• XYZ clut
• Shaper + matrix
• Gamma + matrix

My limited understanding of the above says that:

• Lab clut is an Lab based look-up table i.e. the profile is based on a non-linear set of values contained in a look-up table.
• XYX clut is a linear version of Lab clut.
• Shaper + matrix sounds like there is an equation involved somewhere, with the parameters contained in a matrix.
• Gamma + matrix sounds very similar to the shaper + matrix.

Anyone know about these various options? The only info I have been able to find is from the Argyll site:

Similar to a display profile, an input profile can be either a shaper/matrix or LUT based profile. Well behaved input devices will probably give the best results with a shaper/matrix profile, and this may also be the best choice if your test chart has a small or unevenly distributed set of test patchs (ie. the IT8.7.2). If a shaper/matrix profile is a poor fit, consider using a LUT type profile.

When creating a LUT type profile, there is the choice of XYZ or L*a*b* PCS (Device independent, Profile Connection Space). Often for input devices, it is better to choose the XYZ PCS, as this may be a better fit given that input devices are usually close to being linearly additive in behaviour.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 11, 2011, 10:15:21 am
There does not appear to be a profile for Scanner RGB, but I'll hunt around and see what I can find.
[/i]

Look in the Nikon program files.  You should find a list of quite a few profiles.  Is there one or more that doesn't appear to match the options given when using the software, such as maybe "NKLS5000_P.icm".  If so, that could be the profile for Scanner RGB.

Anyway, back to creating your own profile.  I've read that when you turn off CM, or select Scanner RGB in CM, that the scan opens in PS with sRGB imbedded, which is a defect in the software.  If so, you need to be sure not to use sRGB and instead you should delete the profile tag. If what I've read is correct, maybe Coca was seeing that incorrect sRGB tag and it caused your faulty profile.

Also, it appears that no CM and Scanner RGB are the same, just that you can control gamma (and other features) with one and not the other. See: http://www.nikonusa.com/pdf/manuals/Scan4/NikonScan-4_CMS_en.pdf

No CM or Scanner RGB is what you probably want to use for creating your profile since they are the native color space of your scanner and have the widest gamut. Of course, you then need to scan your film with the same settings, then when you open the scans in PS be sure not to use the incorrect imbedded sRGB tag, but instead assign the profile you created.

My hunch is that the particularities of Nikon Scan led you to use some incorrect setting when you created your profile with linear data, which caused the banding problem. 
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 12, 2011, 07:29:02 am
If what I've read is correct, maybe Coca was seeing that incorrect sRGB tag and it caused your faulty profile.  
I'm reasonably certain the Coca ignores embedded profiles because, as a test, I generated two profile from a scanned target, one with and one without an embedded profile and the Coca profiles came back the same (as far as I could tell by applying them to an image).

TESTING PROFILES
I've been thoroughly putting Coca through its paces and have found some anomalies. This is what I have done:

A. Scanned a Kodachrome IT8 target at 2000 dpi, 14-bit, with seven different settings…

1. CM off, Gamma 1.0
2. CM off, Gamma 1.5
3. CM off, Gamma 1.8
4. CM off, Gamma 2.2
5. CM on, Adobe RGB (gamma 2.2, embedded profile called Adobe RGB)
6. CM on, Apple RGB (gamma 1.8, embedded profile called Apple RGB)
7. CM on, sRGB (gamma 2.2, embedded profile called sRGB)

… and gave the resulting images names such as Kodachrome IT8 Gamma 1.0, Kodachrome IT8 Adobe RGB, and so on.

B. I saved the targets without profiles.

C. I generated 4 profiles for each target, using these options in Coca: Lab, XYZ, Gamma+Matrix, Shaper + Matrix, and gave the profiles names such as: Kodachrome IT8 Apple RGB XYZ. The name reflects the way the target was scanned, and the way the profile was generated. There are 28 profiles, generated at high quality, and two generated at ultra-high quality to see if there was any difference (there wasn't).

D. Then I chose five of my test slides (slides that are a challenge to scan) and scanned each of those with the seven settings given in A. The images had names such as: Ref 02 Gamma 1.0, Ref 05 Apple RGB, the name reflecting the slide number and the scan method. This gives me 35 images with which to test the profiles.

E. Then starting with Ref 02, I opened the seven scanned versions of Ref 2 in PS – I had seven tabs open, each tab showing the same image but scanned a different way.

F. Then for each tab, I applied the four profiles applicable for that image, compared them, and took notes. So, for example, the first tab might have been Ref 02 Adobe RGB (scanned with Adobe RGB), and then I applied the profiles: Lab, XYZ, Shaper and Gamma to see the effect. Then I moved to the next tab, and repeated, until all seven versions of Ref 2 had been compared.

G. Then I closed all the Ref 2 images, and opened the seven Ref 5 images, and applied four profiles to each.

A pretty thorough test to determine which profile worked the best. And one did work the best – and it wasn't Gamma 1.0.

RESULTS OF TESTS
Something very interesting turned up: with CM off, the profiles resulting from Gamma 1.5 and Gamma 2.2 scans of the IT8 target generated nonsense profiles in Coca. When applied to the appropriate image, nonsense colours appeared. Any suggestions why? Gamma 1.0 worked, Gamma 1.8 worked, Adobe RGB worked … but not Gamma 1.5 and 2.2. Same scanner, same process, same profile generator, same application to an image, but nonsense results.

FILES TO REPLICATE MY TESTS
Just in case anyone wants to try and work out what went wrong I have uploaded all that is needed to replicate my testing. Coca is free, so if you've never used it, here is your chance. In this folder, http://www.mediafire.com/?thogddgfozxi2, can be found:

• Coca installation file (Coca_setup.exe, 10.7 MB)
• 7 IT8 scans + data file required by Coca (Kodachrome IT8 Targets.zip, 14.8 MB)
• 7 Kodachrome scans (Reference Scans.zip, 11.5 MB)
• 30 profiles, 28 at high quality, 2 at ultra high (Kodachrome Coca Profiles.zip, 6.7 MB)

To reduce file size, the IT8 scans have been resampled to 1000 dpi, 8bit (from the original 2000 dpi, 14-bit). This resampling has no effect on generating profiles, as far as I could tell when I re-profiled some of them at the reduced resolution.

BANDING
If you want to see the banding problem, check out the deep blue of the water in the foreground of the Kodachrome scans. Banding is subtle, but definitely present in all images except Gamma 1.8. It was present in most of the other reference images as well, to various extents. The slides I have chosen as reference slides are a serious challenge to scan. They show extreme contrast, or gentle gradations, or saturated colours against other saturated colours. Most scanners do a reasonable job scanning an "easy" slide – and you won't see banding – but I've dived in the deep end.

COCA HELP
If you need help with Coca, the home site is here: http://www.nla.gov.au/preserve/dohm/coca.html. You do need to put a sensible name into the "Internal Profile Description" field (Section 7), because that is what appears as the profile name in PS. I ran Coca from Windows XP. I don't know if it works on later versions.






Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 12, 2011, 02:48:48 pm
Guy, your test results are bizarre. I don’t know the reason, but it’s a red flag that something is amiss.  I wouldn’t trust any of your profiles to be accurate.  The problem could be software, but it’s much more likely that you have a glitch in your workflow. It could be an incorrect setting in Nikon Scan, PS or how you’re processing your scans to create your profiles.

You’ve obviously put a lot of time into your tests.  I wish I had an answer for you, but all I can offer is a suggestion that you try to keep your tests very limited until you solve the mystery.  For example, work solely with your IT-8 target slide, CM off, gamma 2.2, and one profile method.  Create your profile, and then instead of testing with a regular slide, use your IT-8 slide. Open your IT-8 scan in PS and assign the profile you created. The result doesn’t have to be perfect, but it should be very good.  If not, you probably have a problem in your workflow.  In such case, make only one change in Nikon Scan, PS or your workflow, and retest.  Repeat as necessary. (The reason I suggest you use CM off and gamma 2.2 is because that combination now gives nonsense results and thus may make it easier to track down the problem.)

There is one PS setting you may want to change.  It sounds like you don’t have PS set to automatically alert you to profile mismatches or missing profiles.  If so, you may want to change your PS color settings by checking all the boxes under “Color Management Policies”. 
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 12, 2011, 05:04:40 pm
I took your suggestion about limiting the profile generation to one type and applying it to the IT8 target. But I went one step further: I did the lot on the PC this time (because I'm a Mac man, I do everything I can on the Mac, but Coca only runs on a PC). The interesting thing is: Gamma 2.2 worked first time, and worked with all the reference images as well.

Something strange must have happened to Coca just for Gamma 1.5 and Gamma 2.2, because there is nothing wrong with the target scans. Might have been just a blip in the system.

Anyway, just a short summary of what I have found.

COLOUR DIFFERENCES – Negligible
There is no real problem with any of the various profiles I have generated (apart from the gamma 1.5 & 2.2 glitch). The colour differences between all the dozens of tests are not that much. And since I intend editing further anyway, the differences would mostly disappear in the editing.

BANDING
The banding problem appears to be real and occurs here and there. Increasing the gamma when scanning the target, and using XYZ profile seems to reduce the problem.

SHADOW CONTRAST
There is a slight, but definite improvement in the deepest shadow contrast, sometimes matched by better colour rendition in the shadows, as the gamma of the target scan is increased – the higher the gamma, the greater the improvement. I doubt this can be compensated for by the profiling program as effectively as increasing the gamma during the target scan. One possible reason for this is the gamma of Kodachrome, which is approximately 1.5 according to Hunt – set to that figure to improve shadow contrast in dark-surround viewing (the conditions in which slides are normally viewed). This might be fine for viewing, but causes problem when scanning, because it means the darker areas are darker than they were in real life as compared to the lighter areas. This is the prime reason that you can't copy slides with the same type of film as the original (unless the original film had a gamma of 1) – the shadows become even darker. And it explains why when scanning slides the shadows have to be lifted out of their blackness. Lifting them before profiling seems to give better shadow detail – the profile software has less correction to do.

The above statements are open to correction. I have to whittle down all the test results to the most promising profiling method – at this stage XYZ consistently gives the best results – and then regenerate profiles at Gamma = 1.0, 1.5, 1.8 and 2.2 and apply them to all my reference slides.

The aim of all this testing is to get the best possible profile, accompanied by the easiest editing method in PS, to get superb results from Kodachrome. I expect the resulting PDF, tentatively named The Art and Science of Kodachrome Scanning (and which will be made freely available), will be a lengthy document full of test images, graphs, and quotes from authoratative sources. And I'm hoping volunteers from this forum will offer to proof read it and offer comments.

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 12, 2011, 06:54:20 pm
Now, when you open an IT-8 scan in PS and apply the profile you created, do you get a scan without major color casts?  I ask because the IT-8 scans you posted, with CM on and aRGB or sRGB profiles embedded, open in PS, using the respective profile, with very large color casts.  In other words, does your profile provide a significant improvement over the Nikon CM?

"COLOUR DIFFERENCES – Negligible" -- This is also my experience with my scanner and profiles, even among the "canned" profiles that came with my scanner software (gamma adjusted as well as linear), third party profiles created for my film type and model of scanner, and the profiles I created myself for specific film types using both linear and gamma adjusted targets. I've only used Fuji Velvia and Provia, however, never Kodachrome. But, they all usually have some color cast that needs correction in PS or ACR, as well as tone, contrast and other adjustments.

As far as banding and shadow contrast, I've never see any banding with my scanner (using Velvia and Provia) regardless whether I started with linear or gamma adjusted files, nor have I noticed any significant difference in shadow contrast. Perhaps your experience is different because you're scanning Kodachrome. 

Anyway, glad you got the mystery solved.  Good luck with your project.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 13, 2011, 04:07:22 am
Now, when you open an IT-8 scan in PS and apply the profile you created, do you get a scan without major color casts?

There is an obvious change in colour when I apply the IT8 profile to itself for all scans. I tested like this:

1. I opened all seven IT8 scans (Adobe RGB, Apple RGB, Gamma 1.0 and so on) with their original as-scanned profiles. Adobe, Apple, and sRGB looked identical; the other four were different (Gamma 1.0, 1.5, 1.8 & 2.2) but had the same general look, though of different brightnesses.

2. I then applied the appropriate IT8 profile to each. Except for the weird colours of Gamma 1.5 and 2.2, they all looked identical.

3. Then I applied the new "correct" profiles for Gamma 1.5 and 2.2, and all seven images looked identical – except Gamma 1.0 which had slight banding in a few of the darker patches.

Given that all seven had no perceptible difference, I think I'm safe in assuming that my profiling of this scanner is about as good as it is likely to get. But I'm going to tweak a little more just for the learning experience.


Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 13, 2011, 09:56:32 am
Given that all seven had no perceptible difference, I think I'm safe in assuming that my profiling of this scanner is about as good as it is likely to get.

I think that's a safe assumption.  

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 14, 2011, 11:07:45 am
Guy, I looked at your IT8 Adobe RGB scan, using both the original Adobe RGB profile and the XYZ profile you made. Indeed, there is a significant difference between the two profiles, but neither appears to produce a good result.  Both profiles produce major color casts.  My initial reaction is that it would be difficult to say which profile is actually better.

I compared your scan to my linear scan of my Velvia IT8 target with the generic Minolta linear profile. My scan appears to be much, much better than your result.  They aren't even in the same ballpark. The comparison that really matters, however, is how close your IT8 scan matches your IT8 slide.

Of course, you're scanning Kodachrome and I'm using Velvia, which may account for some of the difference.  In any event, I thought that you might be interested in the comparison. 
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 15, 2011, 03:00:57 am
Thanks, Dean, for having another look at my profiles. I have several Velvia targets, and generated profiles from them at the same time as I did the Kodachrome profiles, but haven't tested them yet. The targets themselves are different. Two examples: the vertical cyan column starts with virtually white for the Kodachrome (obvious pale blue for Velvia), and ends up slightly less saturated-looking than the Velvia cyan. And the bottom-left 4x4 set of patches are obviously more saturated for the Velvia than the Kodachrome.

I have uploaded a comparison of the two, called Kodachrome & Velvia IT8 Comparison, to http://www.mediafire.com/?thogddgfozxi2. The particular Velvia is Velvia RVP50. Even though there are differences, that might not explain why my scans and profiles are different. Maybe my scanner, or the profiles, are not as good as they could be.

I'd like to see your scan of the Velvia target and compare them with mine. Could you send me, or upload, an image of the target with embedded profile? You can gmail me at gdburns.

A Question
I've been trying to install profiles onto the PC I've been given. I've tried right-clicking then selecting Install, but Windows comes back with an error message "This is not a valid profile". So I manually put them in this folder: \Windows\system32\spool\drivers\color, and then they can be applied. Any idea why right-click > Install won't work?

Bug in NikonScan?
Finally, when I turn CM on and select Scanner RGB in NikonScan, there is no profile applied when I open the image in Photoshop, both on the Mac and PC. Must be a bug in NikonScan. For all other colour-managed spaces, the image arrives in PS with the appropriate profile embedded.

Edit: I didn't read Nikon's Color Management PDF carefully enough. On page 89 it says of Scanner RGB:

This profile replicates the color space achieved when scanning with Nikon CMS off… In order to produce the effect achieved by turning Nikon CMS off, the monitor profile is not used, nor is an ICC profile included with the image when it is opened in the host application.

So Scanner RGB does not include a profile. But the statement implies that Scanner RGB should give the same colour numbers as when CM is turned off, but that's not what I found. I'll have to rescan using both settings and double check if the scanned image is the same.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 15, 2011, 11:41:49 am
Bug in NikonScan?
Finally, when I turn CM on and select Scanner RGB in NikonScan, there is no profile applied when I open the image in Photoshop, both on the Mac and PC. Must be a bug in NikonScan. For all other colour-managed spaces, the image arrives in PS with the appropriate profile embedded.

Here is what I wrote above about this problem.  "I've read that when you ... select Scanner RGB in CM, that the scan opens in PS with sRGB imbedded, which is a defect in the software.    ...

 See: http://www.nikonusa.com/pdf/manuals/Scan4/NikonScan-4_CMS_en.pdf"

Also note what I wrote above asking if you had a profile for Scanner RGB.

It appears that either there is no profile attached or an incorrect profile.  So, if you want to use CM on and Scanner RGB, and not use a profile you make, you should try to find a Scanner RGB profile in the Nikon Scan program files.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 15, 2011, 11:46:23 am
A Question
I've been trying to install profiles onto the PC I've been given. I've tried right-clicking then selecting Install, but Windows comes back with an error message "This is not a valid profile". So I manually put them in this folder: \Windows\system32\spool\drivers\color, and then they can be applied. Any idea why right-click > Install won't work?

No idea.  I downloaded your XYZ profile from your Adobe RGB  IT8 scan, and that profile installed on my PC (Vista 64 bit) by right clicking and clicking on "install". 
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 15, 2011, 12:18:44 pm
My Velvia 50 IT-8 slide is also different from your Kodachrome IT-8 slide.  So, in my comparison I focused on the grey scales.  

I expect that few if any of the grey scale patches are perfectly grey, with each of the RGB values being equal, but I assume that they should be close. All those patches on my scan have RGB values that are close to the same.  In other words, my grey patches are close to being perfectly grey. Your grey patches, except in the some mid-tones, are far from grey.  

Here are a couple of examples of the RGB numbers from my scan versus your scan with your XYZ profile:  

Patch 5:  Mine: 181 181 183.  Yours: 176 191 172

Patch 20:  Mine: 32 31 32.  Yours: 24 20 12

Here is my IT-8 scan.  I scanned it as 16 bit linear, opened it in PS and assigned the generic Minolta linear profile, converted it to aRGB, changed it to 8 bit, reduced the size and saved as a jpeg. (Note, I think that I may have slightly over exposed the scan, but I don't thing that slight over exposure is significant for the comparison.)
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 16, 2011, 07:49:35 am
My Velvia 50 IT-8 slide is also different from your Kodachrome IT-8 slide.  So, in my comparison I focused on the grey scales.  

I expect that few if any of the grey scale patches are perfectly grey, with each of the RGB values being equal, but I assume that they should be close.  

[Note: Excuse the lengthy posts. I use the information in threads I start as my archive of pertinent info. I lose things too easily on my computer, and this way, other people can comment if something I have said is amiss.]

I don't think the grey patches are truly grey. I looked up the Kodachrome and Velvia text files that contain the colour info for the IT8 targets. These are the numbers for Patch 5, taken from the GS5 (Grey Scale 5) row in the text files:

Kodachrome GS5: 69.79, -8.84, 3.65
Velvia GS5: 65.89, -1.54, 2.07

When I typed those numbers into PS to see the colour, there is an obvious difference – the Velvia is almost grey whereas the Kodachrome has a distinct green tinge. In the IT8 world, it looks like greys are NOT grey, and they differ from target to target.

GS5 COLOUR NUMBERS
The figures you quoted for GS5 colour numbers for my Kodachrome scan and your Velvia scan may not be truly representative of the actual colour numbers of Patch 5. Two reasons:

1. There is quite a bit of variation within each patch. If you move around inside a patch, the colour number change significantly. This is best seen by looking at the histogram of a patch which will show a spread of values.

2. If you downsample the image (the Velvia jpeg was 1000 pixels wide, the Kodachrome was 2700 pixels wide), the variation becomes smaller because the pixels are averaged.

I brought up this point with Andrew, the author of Coca, about a month ago. This is what I asked:

I'm slowly working my way through testing Coca. In one of your emails you quoted from the Argyll site: "The TIFF file can be either 8 or 16 bits per color component, with 16 bit files being slower to process, but yielding more accurate results". I don't understand how that can be the case. When I look at any of the colour patches in Photoshop, there is a wide range of values within each patch. Why would 16-bit resolution give any better results than 8-bit, given the wide range of RGB values for what should be a single value? Maybe I have missed something, but if the average was taken of these numbers (chosen at random, but assume they are supposed to be a single value of 1.23 sampled at 16-bit):

1.23456
1.20897
1.25639

You get an average of 1.2333. If you find the average of the numbers rounded to 2-decimals (1.23, 1.21, 1.26) you will get a result that is no less accurate because the original readings varied so widely.


I received this response: "Indeed, you make sense. I don't know the answer, so I send your question to Graeme, the guy who created Argyll. Let's see what he is going to say." A few days later, Graeme replied through Andrew:

Well, if noise in your system sufficiently dithers the quantization, then no, 16 bit is not likely to make much difference. But this is not necessarily the case with all systems or device values. Some device values on some systems can show systematic biases to 8 bit quantized values (e.g. significant numbers of pixels of values 3, 4 and small numbers of other values). The other thing to keep in mind is that scanin can (and is sometimes) used on synthetic images rather than real world images, where there is no source of natural noise to dither 8 bit quantization. The other aspect is simply - why throw information away unnecessarily, and subject the system to the uncertainly of whether 8 bit quantization will play a role in the end result ? Offering the option of 16 bit processing allows a resolution of this uncertainty. Graeme Gill.

What I take from all the above, given the spread of colour numbers in each patch, is that "16 bit is not likely to make much difference", but you may as well use it because: "why throw information away unnecessarily".

It's seems to me that colour profiling has quite an amount of room for error. For example, in the Velvia GS5 patch in the scan previously uploaded – already averaged by resampling and jpeging – the RGB values ranged from:

R: 179-185
G: 179-187
B: 179-186

Quite a spread – but the Kodachrome spread was even wider.

GRAIN & DUST
And this leads me to another point: what should be done about dust spots and blemishes on the scan? I assume the profiling software averages each patch. If so, unless it clones out the dust, or ignores any values that are obviously in error, then it will be averaging dust, hairs, grain, blotches and actual patch colour. I have always turned on Digital ICE to remove most of the blemishes, but a few still remain so I get rid of them. However, Hutchcolor reckons otherwise (and I reckon the author is wrong: how can a simple average improve the situation when noise is present?):

Q: Upon enlarging the HCT target scan I notice a lot of grain in the patches. Should I attempt to smooth out the grain before creating the profile?

A: NO! Any grain you see is a normal product of film emulsions and/or the scanner, and is actually valuable in helping an 8-bit per channel scanner "see" more that the normal 256 tone levels. Good profiling software will integrate (average) the grainy pixels to produce an RGB value with better than 8 bit precision. If you try to smooth the HCT target scan (or even a live original) for example with a noise filter, you will actually be reducing the precision of the profile.


Maybe there is some validity to what he says, but I would like some proof that grain improves a profile.

KODACHROME BLUE
Hutchcolor may be correct about grain and profiles, but he's wrong about "Kodachrome blue":

On most scanners, Kodachromeฎ transparencies produce a strong blue or blue-magenta cast, because the yellow dye used in Kodachrome film emulsions appears weaker through typical scanner filters than it does to the human eye.

Wrong. RWG Hunt makes it clear (p229, Reproduction of Colour) why all scanners will see Kodachrome with a blue cast (Hunt was Assistant Director of Research at Kodak, Harrow):

When the viewing conditions consist of projection by tungsten light in a  darkened room, the light form the projectors appears yellowish, and therefore to obtain results that appear grey the picture has to be slightly bluish; this is why the curves of Fig. 14.9(a)… are not even approximately coincident, the blue densities being lower than, and the red densities higher than, the green densities, in order to produce the bluish result required.

Silverfast makes even more of a hash of the explanation of "Kodachrome blue" than does Hutchcolor:

Accordingly the human eye will see more of the yellow spectrum contained in a Kodachrome image than the scanner's CCD. Consequently scanning a Kodachrome transparency with an ICC profile based on a Fuji or Ektachrome IT8 calibration, will produce bluish scans. Only with a dedicated Kodachrome calibration target can correct and original "Kodachrome colors" be achieved.

Even people who should know their stuff, appear ill-informed when it comes to scanning Kodachrome. I'm not trying to belittle their comments, I'm just trying to understand the scanning of Kodachrome (and slides in general) and how it is best done, and misinformation like the above doesn't help.




Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 16, 2011, 08:04:49 am
I have just come across an article about scanning and gamma:http://photo.bragit.com/scanning/it8tests.shtml

I haven't digested it fully yet, but the author seems to be of the same mind as I am: that for accurate profiling of deep shadows, an increased gamma is desirable. In his update to the article (linked at the bottom of the above webpage) he says:

Since the referred article was written in December 2004, I now recommend to do everything in gamma 2.0, also when inCamera profiles are used. The reason is that it provides a better base for lifting deep shadows.

Then he provides lots of examples. My testing at gamma 1.0, 1.5, 1.8, and 2.2 seemed to suggest that gamma 2.2 was sometimes better than 1.8, but for various reasons I've settled on 1.8. Maybe the optimum lies at 2.0 as "bragit" suggests. Time for more testing!

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 16, 2011, 11:13:14 am

Kodachrome GS5: 69.79, -8.84, 3.65
Velvia GS5: 65.89, -1.54, 2.07

When I typed those numbers into PS ...


How do you type these numbers into PS?  I'm using PS CS4, 64 bit, and it allows me to enter whole numbers only.

GS5 COLOUR NUMBERS
The figures you quoted for GS5 colour numbers for my Kodachrome scan and your Velvia scan may not be truly representative of the actual colour numbers of Patch 5. Two reasons:

1. There is quite a bit of variation within each patch. If you move around inside a patch, the colour number change significantly. This is best seen by looking at the histogram of a patch which will show a spread of values.

I set my color picker to a large average number of pixels and get very little variation within each patch.  Also, IMO, using the info data is much more useful for this purpose than the histogram. On a pixel level, I agree there is a much wider variation, due to dye clouds, grain and the inherent difference of averaging.  IMO, that pixel level variation is irrelevant.

Regarding dust and grain, I agree with you about dust and using ICE, but I agree with Hutchcolor about grain.



KODACHROME BLUE
Hutchcolor may be correct about grain and profiles, but he's wrong about "Kodachrome blue":

On most scanners, Kodachromeฎ transparencies produce a strong blue or blue-magenta cast, because the yellow dye used in Kodachrome film emulsions appears weaker through typical scanner filters than it does to the human eye.

Wrong. RWG Hunt makes it clear (p229, Reproduction of Colour) why all scanners will see Kodachrome with a blue cast ...

...

Even people who should know their stuff, appear ill-informed when it comes to scanning Kodachrome. I'm not trying to belittle their comments, I'm just trying to understand the scanning of Kodachrome (and slides in general) and how it is best done, and misinformation like the above doesn't help.

I disagree with your criticism of Hutchcolor.  It appears that your criticism is because he said "most scanners", not all scanners.  That difference seems inconsequential.  Moreover, unless someone has tested every single model of scanner ever produced, they'd be safer in saying "most scanners", not all scanners. So, IMO, Hutchcolor's use of the words "most scanners" is a positive sign, and certainly doesn't warrant criticism. 



Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 16, 2011, 11:37:01 am
I have just come across an article about scanning and gamma:http://photo.bragit.com/scanning/it8tests.shtml

Maybe he's correct, but in my experience, as demonstrated by my example scan above, I have no problem using linear data. On the other hand, I have no problem using gamma adjusted data.  So, by all means use gamma adjusted data if it suits you.

So far I've just glimpsed at the linked article.  But one thing jumped out at me.  When he describes setting up his scanner and "Optimizing lamp lightness", he wrote: "The exact lamp setting for my unit is: +8 units overall, plus an additional +6 and +4 units respectively for green and blue."  This approach is contrary to the usual recommendation and contrary to my practice. Is it wise to make color corrections before profiling?  I'll need to read his article more carefully, but a lot of what I've glanced at raises questions.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 17, 2011, 12:09:23 am
How do you type these numbers into PS?  I'm using PS CS4, 64 bit, and it allows me to enter whole numbers only.

I rounded the numbers.

I disagree with your criticism of Hutchcolor.  It appears that your criticism is because he said "most scanners", not all scanners.  That difference seems inconsequential.  Moreover, unless someone has tested every single model of scanner ever produced, they'd be safer in saying "most scanners", not all scanners. So, IMO, Hutchcolor's use of the words "most scanners" is a positive sign, and certainly doesn't warrant criticism.

I should have made myself clearer in my criticism. Hutchcolor's explanation of Kodachrome's blue cast – "because the yellow dye used in Kodachrome film emulsions appears weaker" – is simply wrong. Kodachrome was designed to have a blue cast, and it will scan with a blue cast even if the image is IT8 profiled. Nothing to do with the human eye response being weaker or stronger than scanner sensors. I wasn't criticising his "most scanners" comment, except indirectly by my comment that all scanners will see a Kodachrome blue cast – and they will, unless the scanner manufacturer offers a Kodachrome setting on their scanning software, and that setting accurately compensates for Kodachrome's blue cast. I found, for example, that Vuescan's Kodachrome setting doesn't alter the scan.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 17, 2011, 04:58:21 am
When he describes setting up his scanner and "Optimizing lamp lightness", he wrote: "The exact lamp setting for my unit is: +8 units overall, plus an additional +6 and +4 units respectively for green and blue."  This approach is contrary to the usual recommendation and contrary to my practice. Is it wise to make color corrections before profiling?  I'll need to read his article more carefully, but a lot of what I've glanced at raises questions.

That's what I'm looking for – articles that raise lots of questions. It's only by questioning that you learn. I haven't gone through the entire bragit.com article yet, as I got sidetracked by all the links.

My initial answer to your query: "Is it wise to make color corrections before profiling?" is that it shouldn't matter. The idea of profiling is to correct for errors, so if someone wants to introduce gamma or colour corrections while scanning, it shouldn't make much difference, as long as those same settings are used for all scans. The reason why it may be desirable to introduce corrections while scanning, is to bring the scanned image of the target as close as possible to the real IT8 target. That way, the profiling software has less correction to do. One of the links that bragit.com provides is to computerdarkroom.com, where it states:

Those of you reading this who already own a scanner profiling application… will be wondering if the above procedure for determining the ideal gamma is either necessary or desirable. The simple answer is NO and Yes in that order! The reason it's not necessary is the IT8 calibration will always reproduce grey-scale patch GS11, etc, as per the associated reference data file irrespective of what gamma value you choose to make the original scan with. In effect the IT8 calibration process will produce the optimum Tone Curve.

Sounds reasonable to me. But his explanation of the "Yes" part – is gamma correction desirable, is not at all clear. Something to do with loss of shadow detail in some cases for some scanners.

Getting back to my original post about gamma: I assumed that a setting of Gamma 1 would give similar scans for all scanners. Of course I imagined different colours, different brightnesses, but basically I assumed a similar general look, no matter who made the scanner. It looks like that is a wrong assumption. When I set gamma = 1, the scan is extremely dark – the mean value for the GS11 patch is 48. GS11 should scan in the range 100-115. That's something I didn't realise, and which is not explained anywhere that I could find in the Coca or Argyll documentation. Here's a very clear explanation from computerdarkroom.com (Optimising the Scanner Tone Response Curve):

The purpose of scanning the grey calibration strip [of the] IT8 calibration target… is to find a the ideal Tone Curve or gamma-gradation value for your particular scanner. The simplest approach is to adjust the gamma value so that the grey patch GS11 on the IT8 calibration slide has an average RGB value in the range 100-115. This means scanning the IT8 at different gamma values then selecting the one that is closest to the optimum. You should also find that the average RGB values for patches GS0 (white patch) is in the range 235-250 and GS23 (black patch) is in the range 10-25. In each case it is better to avoid values outside these ranges.

Dean, it looks like your scanner has a different Gamma 1.0 than mine. All this talk from me about needing to vary gamma might come down to the fact that your Gamma 1 may be equivalent to my Gamma 1.8. In your case you see good results at Gamma 1 (GS11 ~100), whereas I need Gamma 1.8 to achieve the same result. If I use Gamma 1, the scan is so far away from what it should be, the profiling introduces small errors at the extremes.

QUES
How can you tell which of our scanners is actually scanning at a true Gamma 1.0? We both have IT8 targets that would be pretty much identical, yet my Gamma 1.0 scan (I'm assuming) is seriously darker than yours. Why? Is my scanner faulty, or is your scanner applying corrections before you are given the ability to adjust?

The only way I can think of to work out which scanner if actually giving true Gamma 1.0 is to go back to the maths:

 Y/Yn = ((L* + 16)/116)3

Where Y/Yn is the relative luminance (Yn is max white), and L* is the brightness value in Lab coordinates – which are available in the data file for each IT8 target. I have just scanned my Velvia target at Gamma 1.0, 2.0 and 2.2 with these results for GS0, GS11 and GS23 (in order for each gamma):

Gamma 1.0 … 235,  40,   1.2
Gamma 2.0 … 244, 100, 15.8
Gamma 2.2 … 245, 109, 20.0

Taking the computerdarkroom.com figures as a guide, that means I should be using Gamma 2.2. It would be interesting to see your readouts for Gamma 1.0.

Now the maths to work out if my Gamma 1.0 is actually gamma = 1.0. Assume that it is. If so, the 0-255 scale is linear would be linear with respect to luminosity and be equivalent to the "Y" values in the formula, with Yn being the maximum white for the IT8 target (GS0). From the IT8 data file, the L* values are:

GS0 = 92.18
GS11 = 41.95

Maximum white in the Velvia IT8 target has an L* value of 92.18. Plugging this into the formula gives: Y/Yn = 0.811. The maximum luminance of the Velvia brightest white is actually only 81.1% of reference white (I'm not sure what that is defined as). But as this is all relative, it doesn't matter anyway.

What is the relative luminance of GS11? It is defined as having an L* value of 41.95, which gives Y/Yn = 0.125.

The luminance ratio of GS11/GS0 is therefore 0.125/0.811 = 0.154. If my scanner is truly scanning at Gamma 1.0, then the RGB value of GS11 (measured in Photoshop as 40 – see above) should be 0.154 times the RGB value of GS0 (measured as 235):

Measured GS11: 40
Theoretical GS11: 0.154 x 235 = 36.

Indicating that my scanner when set to Gamma 1.0 is giving a close match to a linear series of luminance values (I consider within 10% close, given all the inaccuracies involved). A graph of the results for several greyscale patches gave a straight line (see attachment). Dean, it would informative if you could run through the same calculations for your Gamma 1.0 scan.

My understanding of colour theory is only superficial, so there may be errors in the above analysis. Anyone care to comment? What I'd like explained is: why my Gamma 1.0 scan appears much too dark on my monitor. My theory is:

1. It is a true Gamma 1.0 scan, meaning that linear values of luminance are being saved to the image file.

2. In Photoshop, my default colour space is sRGB, and that is the space in which the image is being shown. sRGB has a gamma of 2.2, meaning the input-output relationship is a power-law of the form Out = In2.2 (see attached graph).

3. If the above are true, then my image is going to have a lot of dark areas because each RGB value will be mapped downwards. Only if the profile has a linear characteristic matching the RGB data will the image display properly.

To confirm the above, I generated a linear sRGB profile following these instructions (Making a Linear IC Profile): http://fnordware.blogspot.com/2008/05/making-linear-icc-profile.html. I then applied the profile to my supposed linear scan and … it appeared similar to my IT8 scans with gammas of 1.8 and 2.2. This has suggested to me a test for whether or not an image has linear gamma: Open the image in PS and apply a linear sRGB profile. If the image is way too bright, then you do not have a Gamma 1.0 image.

I have uploaded a zip file called Gamma 1 Test Documents to http://www.mediafire.com/?thogddgfozxi2 in case anyone wants to play around and offer comments. The file contains:

• Gamma 1.0 Plot (image as attached here and Grapher version for opening on Mac);
• Gamma 1.0.tif (a linear IT8 Kodachrome scan);
• Linear sRGB IEC61966.icc ( a linear sRGB profile)
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 17, 2011, 06:49:07 am
The bragit.com articles were interesting as far as altering gamma was concerned, but otherwise were not very informative. You cannot base a conclusion about scanning slides on the results when applied to three slides. I have 20 reference slides, each scanned and edited scores of times and I still haven't come up with a reliable method of scanning and editing to obtain optimum results – but I'm getting close. Another problem with bragit.com is that the author comes up with no overall method that might work. Sometimes he uses gamma 1.0, other times gamma 2.0. A bit of a dog's breakfast, unfortunately.

The author at computerdarkroom.com is more informative, but lacks justification for his conclusions. Why should gamma be changed so that GS11 is in the range 100-115? Where did those numbers come from?

One idea was presented by bragit.com that I hadn't considered so far: the editing colour space. All my editing has been in the IT8 profile colour space, whereas he converts to Adobe RGB, saying that shadows are best lifted in the IT8 space, but colour editing is best done in Adobe RGB.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: dmerger on May 17, 2011, 05:36:38 pm
Here's the linear version of the IT8 slide I posted above. I merely reduced the size and saved as a jpeg.  Edit: Of course, I also had to change it to 8 bit.

Why are you using sRGB as your working color space?  Wouldn't aRGB or, perhaps better yet, ProPhoto be a better choice?

In my opinion, the scanning info on Computer Darkroom is, for the most part, just advertisement for SilverFast.  I'm not inclined to comment further on info from that site.  As for the bragit info generally, maybe what he writes is accurate for his scanner and workflow, but doesn't seem to coincide with my experience with my scanner.

As for when to do your gamma, color and other such adjustments, I've already said about all I can say, which is I do not have problems with linear.  Worth considering, however, is digital cameras use CCDs just like your scanner, and the clear consensus is it is best to process their raw files, which are linear, with something like ACR or Lightroom, both of which I believe use linear data internally.    

Don't be too quick to dismiss the info from HutchColor.  Bear in mind that whether Kodachrome has a blue cast depends on the light source.  Hunt was talking about projecting Kodachrome with a tungsten light source.  HutchColor was talking about scanning.  Maybe HutchColor is wrong, but maybe it's possible to reconcile the two statements. In any event, both agree that Kodachrome has a blue cast when it's scanned. Is the reason why so important for the present purposes?

I guess I'm not sure what you hope to accomplish by creating a Kodachrome profile.  It now seems like you want your profile to accurately reproduce the colors of Kodachrome, including a blue color cast.  (Note, as I mentioned above, that the blue color cast is partly a function of the light source.  In other words, perhaps you could say that Velvia has a yellow color cast if all you did was view it projected with tungsten light.)  I thought you were trying to create a profile that didn't produce a blue color cast, but I'm not so sure any more.  
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 18, 2011, 08:27:21 pm
Hi Guy,

Gamma in scanning was necessary in the old days when scanners only output 8 bits. Internally, the scanner ADCs were putting out more than 8 bits, linear. Gamma served to compress the higher-bit "raw" scan data to fit into only 8 bits of output.

If you read the Hutch Color scanning info carefully you will see that the exercise of applying different gammas was to optimize the 8-bit output.

Most desktop scanners nowadays allow saving the full internal high-bit linear data, making it unnecessary to apply gamma (which loses data). The linear high-bit output of a scanner is the best (most complete) data that you can get.

"Theoretically," the unmolested linear high-bit scans should be best for profiling. An input profile will normally be referenced to the XYZ PCS (profile connection space), which is a linear space. One of the first things that profiling software must do is to linearize the target data to the reference. If the target data is gamma-compressed, it just gives the profiling software more work to do as it will have to calculate a function to reverse the gamma-compression. Plus, many levels have been lost as a result of the gamma-compress-decompress process.

So linear target scans should be at least as good as, if not better than, gamma-compressed scans. Why are the linear profiles the worst in your tests?

I downloaded your reference scans and profiles to see what was going on, but there is a problem - your reference scans are all 8-bit. Of course this will favor scans that have been gamma-compressed before truncation to 8-bits. Linear scans in 8-bit are guaranteed to be posterized. Can you make the 16-bit versions available for download?

Rgds,
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 18, 2011, 10:49:09 pm
Crames – just a quick reply. From memory, the scans from which I generated the profiles were 2000 dpi 16-bit. The scans I uploaded were probably reduced in size to make them smaller. I'll upload the full-resolution scans today.

P.S. I finally finished Hunt. Did you see my reply to the thread where you asked about the effect of dim surrounds when editing in dark surrounds (or vice versa)?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 18, 2011, 11:46:16 pm
Crames – just a quick reply. From memory, the scans from which I generated the profiles were 2000 dpi 16-bit. The scans I uploaded were probably reduced in size to make them smaller. I'll upload the full-resolution scans today.

Thanks. Small is OK but keep the full 16-bit depth.

Another thing to consider is the way Photoshop displays images with 16-bit linear profiles. Photoshop can display linear images with heavy posterization even when the underlying data is smooth, but the posterization will go away once the image is converted to a gamma space.

Quote
P.S. I finally finished Hunt. Did you see my reply to the thread where you asked about the effect of dim surrounds when editing in dark surrounds (or vice versa)?

Thanks for the reply and sorry for the delay - I will try to respond shortly.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 19, 2011, 05:22:52 am
I have uploaded the 16-bit scans to a folder called Kodachrome IT8 Scans 16-bit, 57 MB, which can be accessed here: http://www.mediafire.com/?thogddgfozxi2

I have revisited Gamma = 1 scans and have compared the scanned values of the darker grayscale patches (GS20-23) with what they should have been according to the L* values. The results were a bit of a surprise. The first figure is the value shown by the histogram after the target was scanned; the second figure in brackets is the L* value from the IT8 data file, set to the range 0-255 by the formula RGB = 255L* /903.3 (the formula which applies when L* is less than 8, according to Hunt page 110):

G20 … 1.49  (1.20)
G21 … 1.33  (0.79)
G22 … 1.33  (0.30)
G23 … 1.32  (0.14)

The surprise wasn't the difference between scanned and theoretical – the surprise was the constant high black level for the last three patches. Under a strong light I could see the difference between G21 and G22, but G22 and 23 looked the same. The blacks appeared to be beyond the capability of the scanner. So I thought I'd scan the slide using the POS setting instead of the Kodachrome setting, as that should give a purer result (no Kodachrome conversion):

G20 … 1.07  (1.20)
G21 … 1.00  (0.79)
G22 … 0.98  (0.30)
G23 … 0.99  (0.14)

This is what is happening: I could clearly see the change in colour between the Kodachrome and POS scans: the Kodachrome has a yellow cast compared to the POS. Yellow seems to sweep across the entire slide when I changed between the two. And this yellow cast washes out the densest blacks. This throws up a quandry:

1. The use of ICE to remove dust via infrared detection is a huge plus for the Coolscan under NikonScan. It works superbly for Kodachrome slides, hardly affecting the image in almost all cases, and in the few cases where it does bad things, I have a simple workaround. ICE is a must-have for me.

2. Well then, if the Kodachrome setting causes problems with the densest blacks, why not use ICE under the POS setting? IT8 profiling should take care of the colours. Unfortunately, when scanning under POS, ICE uses a stronger version of dust removal, and this stronger version can't be used with Kodachrome because ICE interprets something in Kodachrome as dust and removes it. ICE+POS (for a Kodachrome slide) means wrong colours by about 5-10 RGB values; whereas ICE for a Kodachrome slide with the Kodachrome setting has no effect on colour.

3. So I can't use the POS setting because ICE (under POS) is too rough with Kodachrome; but the Kodachrome setting slightly decreases the density of the densest blacks, resulting in the last three being blocked.

NikonScan should have incorporated a feature to vary the ICE strength.

And nobody should suggest that I try SilverFast. I have, and its implementation of ICE when driving the Coolscan is pretty abysmal.

If anyone would like to see the result of the four combinations of ICE and POS on a Kodachrome target, check out the zip file called Kodachrome IT8 ICE & POS Comparison (31 MB) available for download from the link given above.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 19, 2011, 08:13:52 am
I have uploaded the 16-bit scans to a folder called Kodachrome IT8 Scans 16-bit, 57 MB, which can be accessed here: http://www.mediafire.com/?thogddgfozxi2

I was hoping to see the ref 05 scans in 16 bit, but the IT8 Scans 16-bit are definitely much smoother.

Can't comment on the ICE/NikonScan issues as I don't have a Coolscan. What is "POS?"

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 19, 2011, 09:00:38 am
POS is Nikon's shorthand for "Positive" – every slide type except Kodachrome.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 19, 2011, 11:01:16 am
I was hoping to see the ref 05 scans in 16 bit, but the IT8 Scans 16-bit are definitely much smoother."

I'll upload some of the reference scans in 16-bit if you want to play around.

Try as I might, I cannot get a Gamma 1.0 scan of a high-contrast slide to look as good in the shadows as a Gamma 1.8 scan. And in my experience, it's the shadow area that is very important for a slide to look good on screen. Whether it's my scanner or Argyll I don't know. Will post the results for all to see as soon as I am confident of the results. There's no particular reason I chose gamma 1.8. It was just one of a range of values I was experimenting with.

I've been wondering if there is an optimum gamma (other than 1), so I've been playing around with some equations to work out how closely a gamma curve matches the L* curve. A close match would mean that when profiling the scan in Lab coordinates, the errors would be minimized. I have attached a plot showing the gamma 2.2 and the L* function, and a third curve showing the errors between the two. By plugging in a range of gamma values, it turns out that the optimum match between gamma and L* occurs around gamma 2.5.

More experimenting coming up.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 21, 2011, 12:42:39 am
I'll upload some of the reference scans in 16-bit if you want to play around.

Try as I might, I cannot get a Gamma 1.0 scan of a high-contrast slide to look as good in the shadows as a Gamma 1.8 scan. And in my experience, it's the shadow area that is very important for a slide to look good on screen. Whether it's my scanner or Argyll I don't know. Will post the results for all to see as soon as I am confident of the results. There's no particular reason I chose gamma 1.8. It was just one of a range of values I was experimenting with.

Please upload them. I am wondering about the defects you are seeing, as I don't have a problem with linear scans.

Quote
I've been wondering if there is an optimum gamma (other than 1), so I've been playing around with some equations to work out how closely a gamma curve matches the L* curve. A close match would mean that when profiling the scan in Lab coordinates, the errors would be minimized. I have attached a plot showing the gamma 2.2 and the L* function, and a third curve showing the errors between the two. By plugging in a range of gamma values, it turns out that the optimum match between gamma and L* occurs around gamma 2.5.

There is validity to that approach, but I'm not sure how it applies for different kinds of Argyl/Coca generated profiles. It might not help a measurement-based 3D look-up table type of profile, but it could help a matrix-shaper profile. See the discussion in the Hardeberg thesis: Acquisition and reproduction of colour images (http://pastel.enst.fr/26/00/Jon_HARDEBERG.pdf), Chapter 3 (3.2.3.5), where he reduced perceptual error by using an exponent of 1/3 on the scanner values.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 21, 2011, 02:18:46 am
Guy,

Since you are experimenting, here is a profile you might want to try:

https://sites.google.com/site/cliffpicsmisc/home/profiles/KChromeCoolscanT3250.icm?attredirects=0&d=1

It was made with Coca using your target scan (16-bit gamma 1), but using a Q60 reference file that I modified to have a tungsten 3250K white instead of the usual D50.

To use it, assign it to a Kodachrome scan, then convert to a working space of your choice using the ABSOLUTE COLORIMETRIC INTENT. This should give you the Kodachrome scan as though illuminated with tungsten projector light.

Of course, you should view the image in a dark surround to complete the simulation. If you project the image with a video projector, I wonder how close it will look to the actual projected slide?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 22, 2011, 07:16:56 am
There is validity to that approach, but I'm not sure how it applies for different kinds of Argyl/Coca generated profiles. It might not help a measurement-based 3D look-up table type of profile, but it could help a matrix-shaper profile. See the discussion in the Hardeberg thesis: Acquisition and reproduction of colour images (http://pastel.enst.fr/26/00/Jon_HARDEBERG.pdf), Chapter 3 (3.2.3.5), where he reduced perceptual error by using an exponent of 1/3 on the scanner values.

Cliff: if you keep giving me articles to read, I'll never finish testing. The Hardeberg thesis is very informative. He modified the linear RGB values from the scanner by applying an exponent of 0.33, trying to approximate the relationship between L* and Y/Yn which has the same exponent. But a closer correlation comes at ~0.4 (gamma = 2.5) if the curves I have generated are correct (see attachment). Maybe there are good reasons why he chose 3.0 instead of 2.4.

For my own reference I have summarised Hardeberg's Chapter 3, in which he explains how he went about generating a profile from an IT8 target. Page references are to Hardeberg, but in what follows there are no quotations.

Summary
Hardeberg found the most accurate method of generating a profile by the methods he tested was to:

1. Linearise the RGB scanner values;
2. Apply gamma correction of 1/3 to the RGB values; and
3. Transform via a third-order polynomial to Lab space. [p51]

That mouthful means:

1 – Before the real processing begins, it is best to ensure the RGB data from the scanner is linear in case of non-linearities caused by stray light, sensor "dark" current, gamma correction of unknown amount applied by the scanning software, and other non-linearities. However, for the two devices tested by Hardeberg, no linearisation was required by one, and only "a correction for the black offset" was required by the other. It seems doubtful to me that any software could attempt to linearise raw RGB output without damaging the data – unless for a very simple case like "black offset". In practice, linearising the RGB data is of no concern (and is not feasible for most users anyway).

It seems to me the main reason Hardeberg includes this step is to make sure that gamma correction hasn't already been applied. i.e. you don't want to use gamma corrected RGB values as input for steps 2 and 3, because his step 2 applies gamma correction. So with his method of profiling, the data must start as linear as possible, otherwise you'll end up with gamma correction upon gamma correction.

I wonder if this is where the idea started that profilers should have linear input data? Yes, profilers do require linear RGB data – if your next step is to apply gamma of 1/3 (as does Hardeberg), to ensure the most accurate profiles in Lab space. But if the profiling software does not apply gamma correction to the RGB data, then Hardeberg's results clearly show that for the most accurate profiles, the RGB data should be gamma corrected before it enters the profiler.

2&3 – If you are transforming from one data set to another, you will get the most accurate results if the two sets start off as close as possible. That means, if your RGB data is linear, you should use the XYZ colour space for your transformation when profiling, because the Y component of XYZ is linear. The problem is that, as Hardeberg explains, the XYZ colour space is not linear with respect to colour, but only with respect to luminance (if I understand him correctly). So although it might appear to be best to use linear RGB values as input to an XYZ profiler, because of the way humans see colour, such profiling (according to Herdeberg's results) won't be as good as transforming to an Lab space – which is reasonably linear with respect to colour.

But… the Lab colour space is not linear with respect to absolute luminance, so it is not linear with respect to scanner RGB values. And transforming between the two will result in errors. Let's put some values into this discussion, in particular, some of the Grayscale values GS0-GS23 (first figure is relative luminance as measured by a linear RGB scanner, the second the L* value of the IT8 target):

73 –>    88
65 –>    84
59 –>    81
  3 –>    20
  1.2 –> 10
  0.3 –> 2.8

The transformer has to work out an equation that relates the two sets of figures. If it is expecting a cubic relationship, or models the relationship cubicly, the fit will be quite good. The fit will be even better if the profiler is presented with this set of numbers (I just made them up) which are derived from the first set by gamma correcting:

89 –>    88
83 –>    84
81 –>    81
21 –>    20
  9.8 –> 10
  2.4 –> 2.8

Transforming the second set will result in better accuracy than the first. This is what Hardeberg has shown.

So, if you know your profiler works in Lab space and applies gamma correction before profiling – then it is best to present it with linear data. However, if it works in Lab space and does not apply gamma correction, then the profile will be more accurate if you massage the RGB figures so that they are close to the Lab numbers to start with. But how does a user know how the internals of their profiler works?

This is all in theory. I haven't seen much colour difference between any of the dozens of targets I scanned, then profiled using the four settings in Coca. But I have seen luminance differences in the deepest blacks of a profiled slide. And that just happens to be one of the areas in which I want the best rendition of my slides - the deep, moody blacks.

Gamma correction has been shown by Hardeberg to theoretically give better results for his test conditions (though there is not much difference between Methods 1 and 3, see Results below). And I've seen an improvement in some of my slides when I gamma correct a target, but I'm not convinced yet how real the improvement is.


Profile Methods tested by Hardeberg
To generate profiles, Hardeberg used different IT8 charts, with and without preliminary gamma correction of scanner RGB values, and three polynomial orders (linear, square and cubic) for the RGB/XYZ and RGB/Lab transform. He tried three general methods:

1. Transforms (first, second and third order) to XYZ space
2. Transforms (first, second and third order) to Lab space
3. Gamma correction of RGB data, followed by polynomial transforms to Lab space.

The main drawback with method 1 is that the XYZ colour space is very poorly correlated to visual colour differences. A transform directly to Lab values (method 2), instead of XYZ values, should give better results since distance in CIELAB space corresponds quite well to perceptual colour differences.

Method 3 involved an exponent of 1/3 (identical to a gamma correction of 1/3) being applied to the scanner RGB values before the transformation to Lab space. The use of cubic root was motivated from considering the Lab transformations, which involves cubic-root functions on the linear XYZ tristimulus values. [p44-46]

Results
Table 3.1 shows that method 3 is the best solution for generating a profile with minimum errors, as measured by the mean colour differences. Figures below show first, second and third order results:

Method 1…   5.1, 2.1, 1.6
Method 2… 19.4, 7.3, 4.6
Method 3…   5.5, 1.4, 1.0

From the testing results, transforming directly from linear RGB values to Lab is not a good solution, whereas applying a gamma correction of 1/3 gives very good results, especially when third-order polynomials are used. [p46-47].
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 22, 2011, 11:54:09 am
Cliff: if you keep giving me articles to read, I'll never finish testing.

I thought you might like that one! ;D

Quote
The Hardeberg thesis is very informative. He modified the linear RGB values from the scanner by applying an exponent of 0.33, trying to approximate the relationship between L* and Y/Yn which has the same exponent. But a closer correlation comes at ~0.4 (gamma = 2.5) if the curves I have generated are correct (see attachment). Maybe there are good reasons why he chose 3.0 instead of 2.4.

Sorry, another article but it's short. Bruce Lindbloom did an analysis to find the best gamma to approximate L* http://www.brucelindbloom.com/index.html?CompandCalcHelp.html#BestGammaForLab (http://www.brucelindbloom.com/index.html?CompandCalcHelp.html#BestGammaForLab) and concluded:

Quote
If you go to the trouble of mathematically finding the "best" gamma, you will find that its value depends on your definition of "best."

    If best means minimizing the largest error, then the best gamma is 2.1723.
    If best means minimizing the RMS (root mean square) error, then the best gamma is 2.3243.

and settled on 2.2 as a good compromise.

Quote
For my own reference I have summarised Hardeberg's Chapter 3, in which he explains how he went about generating a profile from an IT8 target. Page references are to Hardeberg, but in what follows there are no quotations.

An excellent summary, Guy.

I wonder how much gamma actually effects the accuracy of the profiles generated by Coca and others. It would help to have a way to evaluate the accuracy of profiles made with all the different methods. There was a handy profile checker included with Little CMS http://www.littlecms.com/ (http://www.littlecms.com/) which has been spun off into LPROF http://lprof.sourceforge.net/ (http://lprof.sourceforge.net/). It gives dE and other color errors for each target patch. Argyl has a command line profcheck command http://www.argyllcms.com/doc/profcheck.html (http://www.argyllcms.com/doc/profcheck.html).

In my (limited) experience, most profilers will give low errors of dE 1 or less throughout most of the scanner range. But, no surprise, the largest errors, particularly with Kodachrome are in the highest densities. The Dmax and lowest gray-scale patches are not well resolved by the desktop scanners we are working with. Varying the gamma, it seems to me, is not going to help the profiling software to linearized these darkest tones. If anything, gamma is going to push them even closer together, making them even less distinguishable? Maybe this results in more random dithering of the dark tones when passed through the profile, the random noise appearing smoother?

Anyway, one profiling package I used, SCARSE http://www.scarse.org/ (http://www.scarse.org/) seemed to give smoother dark tones from Kodachrome. It's author Andrei Frolov told me that he put extra effort into keeping shadow detail intact, and tested extensively with Kodachrome while developing the last version 0.4.

So all profiling software is not created equal. When it comes to Kodachrome, the extra-high densities and non-neutral gray-scale put extra demands on the profilers.

Still itching to get a look at your 16-bit reference scans, since I still haven't actually seen the problem.



Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: joofa on May 22, 2011, 01:27:50 pm
the XYZ colour space is very poorly correlated to visual colour differences

I think early colorimeters were quick to deduce this conclusion  - a practise that is continuing to this day.

Joofa
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 22, 2011, 10:47:14 pm
Cliff: I'll answer your last few posts in a day or two. But I'll have to read that new article first.

I haven't uploaded my reference scans yet. I want to make absolutely sure that the problem in the shadows is real and is not some artifact of my monitor, eyeballs, room lighting, light entering the scanner or other variables. So I intend rescanning the IT8 target and reference slides at gammas 1.0, 1.5, 2.0, 2.2, 2.4 – then I'll upload.

Another link to varying gamma while scanning a target: http://lprof.sourceforge.net/help/lprof-help.html. Apart from the hyperbole "huge loss of details in shadows", it seems as if the author knows his stuff.

Gamma: On most scanners you can select the gamma to be used for scanning the image. In general you should use a gamma between 2.2 and 3.0. A Gamma 2.2 has the additional benefit of being close to the sRGB gamma, and this means the uncorrected Image will "look nice" on an "average" monitor. It is also near to perceptual gamma. Gamma 2.4 has the additional benefit of being closest to perceptual space, and this is a very good reason to use this value. Less that 2.2 (and of course the infamous 1.0) can generate huge loss of detail in shadows, only to give a slight bettering of highlights. Don't use this unless your are using 16 bits per sample, and even in such case, don't do it unless you know what are you doing! Gammas around 2.4 are best for flat bed scanners and film scanners with limited dynamic range.  With high dynamic range film scanners values closer to 3.0 may be best.  Hutch Color, for example, recommends a gamma of 2.8 for high dynamic range scanners. But for flat bed scanners more than 2.4 (up to 3.0) looses some highlight detail with no gains in shadow detail.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 23, 2011, 12:15:00 am
Cliff: I like the Bruce Lindbloom site. Wonderful stuff. I can't image any colour site more authoratative that his. I was wondering how Bruce RGB got its name (it's one of the options in my NikonScan software), and I've also been wondering why there was a discontinuity in the L* function at L* = 8 which I noticed when I was plotting derivatives. As Bruce explains at length, http://www.brucelindbloom.com/index.html?LContinuity.html, it's because the figure of 903.3 (which I took from Hunt) should actually be  24389/27.

I'm also wondering (I do a lot of wondering) whether sRGB is a better fit to the L* function than a pure gamma, but Bruce's calculator doesn't allow me to plug in sRGB.

Talking about sRGB, since all my Kodachrome scans will only be viewed digitally (no printing envisaged) via BluRay at home or the local cinema, I'm assuming the best colour space for editing would be sRGB – so as not to run into the problem of out-of-gamut colours.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 23, 2011, 01:50:07 pm
Another link to varying gamma while scanning a target: http://lprof.sourceforge.net/help/lprof-help.html. Apart from the hyperbole "huge loss of details in shadows", it seems as if the author knows his stuff.

Gamma: On most scanners you can select the gamma to be used for scanning the image. In general you should use a gamma between 2.2 and 3.0. A Gamma 2.2 has the additional benefit of being close to the sRGB gamma, and this means the uncorrected Image will "look nice" on an "average" monitor. It is also near to perceptual gamma. Gamma 2.4 has the additional benefit of being closest to perceptual space, and this is a very good reason to use this value. Less that 2.2 (and of course the infamous 1.0) can generate huge loss of detail in shadows, only to give a slight bettering of highlights. Don't use this unless your are using 16 bits per sample, and even in such case, don't do it unless you know what are you doing! Gammas around 2.4 are best for flat bed scanners and film scanners with limited dynamic range.  With high dynamic range film scanners values closer to 3.0 may be best.  Hutch Color, for example, recommends a gamma of 2.8 for high dynamic range scanners. But for flat bed scanners more than 2.4 (up to 3.0) looses some highlight detail with no gains in shadow detail.

Like some of the previous references, this advice is geared towards optimizing 8-bit scans. Note the relaxation of the gamma requirement when using 16-bits.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 23, 2011, 02:16:44 pm
Cliff: I like the Bruce Lindbloom site. Wonderful stuff. I can't image any colour site more authoratative that his. I was wondering how Bruce RGB got its name (it's one of the options in my NikonScan software), and I've also been wondering why there was a discontinuity in the L* function at L* = 8 which I noticed when I was plotting derivatives. As Bruce explains at length, http://www.brucelindbloom.com/index.html?LContinuity.html, it's because the figure of 903.3 (which I took from Hunt) should actually be  24389/27.

Lindbloom's color space is BetaRGB. BruceRGB is a different space, created by Bruce Frasier (http://www.brucefraserlegacy.com/).

Quote
I'm also wondering (I do a lot of wondering) whether sRGB is a better fit to the L* function than a pure gamma, but Bruce's calculator doesn't allow me to plug in sRGB.

It's true that both L* and sRGB use a 2-piece TRC, with a linear segment for the darkest tones, but I don't know the answer to your question.

Quote
Talking about sRGB, since all my Kodachrome scans will only be viewed digitally (no printing envisaged) via BluRay at home or the local cinema, I'm assuming the best colour space for editing would be sRGB – so as not to run into the problem of out-of-gamut colours.

I believe that Blu Ray uses "Rec. 709", which has the same white point and primaries as sRGB, but possibly a different tone curve (TRC).
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 23, 2011, 09:44:48 pm
Lindbloom's color space is BetaRGB. BruceRGB is a different space, created by Bruce Frasier (http://www.brucefraserlegacy.com/).

There are too many Bruces in the colour world. When I read somewhere that Lindbloom had created a colour space, I assumed it was Bruce RGB. Thanks for the correction.

Attached is a comparison of sRGB and L*, using the exact specifications for each. Of all the gamma-type curves, sRGB has the best approximation to L* in the shadows.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 24, 2011, 04:07:41 am
There are too many Bruces in the colour world.

 :D

Quote
Attached is a comparison of sRGB and L*, using the exact specifications for each. Of all the gamma-type curves, sRGB has the best approximation of L* in the shadows.

I think you might be on to something.

I made some test profiles using LPROF to see how a gamma 2.2 and an sRGB TRC would affect the errors compared to linear.

To apply the TRCs, I first made 2 variations of the standard sRGB profile, using the Photoshop Color Settings Custom Profile tool. The first is a profile with the sRGB white point and primaries, but a gamma 1.0. The second was the same but with a gamma 2.2 (which Photoshop automatically names "Simplified sRGB"). I opened the linear scan and assigned the sRGB gamma 1 profile, then Convert to Profile to the sRGB gamma 2.2 profile. I suppose that this resulted in only the gamma 2.2 TRC being applied, since the white point and primaries are the same. For the sRGB TRC I did the same, but converted to the standard sRGB profile.

Linear
MEAN_DE   0.479439
MAX_DE   7.25239
MIN_DE   0.0111697
STD_DE   0.791264
Lab dE:
GS18   0.85846
GS19   2.15729
GS20   2.73401
GS21   4.51088
GS22   6.33844
DMAX   7.25239

Gamma 2.2
MEAN_DE   0.475655
MAX_DE   6.95842
MIN_DE   0.00871406
STD_DE   0.762017
Lab dE:
GS18   0.89821
GS19   2.14864
GS20   2.50815
GS21   4.37009
GS22   6.36931
DMAX   6.95842

sRGB TRC
MEAN_DE   0.3986
MAX_DE   2.47964
MIN_DE   0.00954745
STD_DE   0.370155
Lab dE:
GS18   0.71167
GS19   1.86969
GS20   2.0874
GS21   1.0714
GS22   1.12813
DMAX   1.61505

As you can see, preconditioning the target scan with the sRGB TRC significantly improves the error, especially in the very darkest patches. Also, I would say that the application of gamma 2.2, alone, does not make much of a difference. The the linear segment is the key to the improvement.

There is a small problem though - in order to use the profile you first have to apply the sRGB TRC to the scan. Maybe Argyl, with it's almost infinite options, has a way to incorporate the preconditioning within the profile itself.

Definitely worth pursuing IMHO.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 24, 2011, 07:37:48 am
There is a small problem though - in order to use the profile you first have to apply the sRGB TRC to the scan. Maybe Argyl, with it's almost infinite options, has a way to incorporate the preconditioning within the profile itself.

Wouldn't scanning into sRGB space achieve the same thing?

QUES 1: Where did all those multi-decimal numbers in your post come from? Data analysis from within LPROF?

It's good to see the improvement where I was hoping it would be – in the shadows – but I now think the reason my shadows had poor detail has less to do with the profiling and more to do with the density range of my scanner or the quality of my target. For Gamma 1, there is barely a difference between GS20-23 (i.e. those patches are virtually "blocked"), but there is a reasonable difference when scanned at Gamma 1.8 and above. I can't explain that, because I assumed that for the higher gammas the scanner would derive them from the Gamma 1 figures. These are the mean RGB figures, measured in a small area of each patch in Photoshop > Histogram, Gamma 1 first, then Gamma 1.8:

GS19 … 2.25,  18.19
GS20 … 1.43,  14.82
GS21 … 1.32,  13.44
GS22 … 1.33,  12.24
GS23 … 1.33,  12.42

What intrigues me is – for my scanner, the Gamma 1.8 figures are not derived directly from Gamma 1.0. If they were, GS22 would be higher than GS21. This leads me to ask two more questions:

QUES 2: What scanner do you use?

QUES 3: What are the RGB values of GS20-23 patches for one of your IT8 scans using Gamma 1.0?

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 24, 2011, 08:31:51 am
Wouldn't scanning into sRGB space achieve the same thing?

It would depend on the scanner software. It's possible that the scanner software would apply more than just the sRGB TRC, and also do a matrix transformation to the sRGB primaries as well as a chromatic adaptation to the sRGB white. I tried to avoid that extra manipulation and apply only the TRC.

Quote
QUES 1: Where did all those multi-decimal numbers in your post come from? Data analysis from within LPROF?

That's from the error report that LPROF conveniently generates after making a profile.

Quote
It's good to see the improvement where I was hoping it would be – in the shadows – but I now think the reason my shadows had poor detail has less to do with the profiling and more to do with the density range of my scanner or the quality of my target. For Gamma 1, there is barely a difference between GS20-23 (i.e. those patches are virtually "blocked"), but there is a reasonable difference when scanned at Gamma 1.8 and above. I can't explain that, because I assumed that for the higher gammas the scanner would derive them from the Gamma 1 figures. These are the mean RGB figures, measured in a small area of each patch in Photoshop > Histogram, Gamma 1 first, then Gamma 1.8:

GS19 … 2.25,  18.19
GS20 … 1.43,  14.82
GS21 … 1.32,  13.44
GS22 … 1.33,  12.24
GS23 … 1.33,  12.42

What intrigues me is – for my scanner, the Gamma 1.8 figures are not derived directly from Gamma 1.0. If they were, GS22 would be higher than GS21.

I think you are limiting your measurements to 8 bits because of the way Photoshop displays the histogram. If you Select a region of the target and Filter, Average, you can read the average in the Info window set to 16-bit for more precision. Then you can see the variation in the linear scan values at the lowest levels. Also, see the LPROF measurement reports below which use the full bit depth.

So I still think it is true to say the the Linear scan is the Mother of All Scans - all of the others are derived from it. Unfortunately profiling software is not necessarily able to take advantage of additional information that might be available in a linear scan, and have been optimized instead for gamma adjusted scans which must be the more common type.

Quote
This leads me to ask two more questions:

QUES 2: What scanner do you use?

I have a Polaroid Sprintscan 4000, which gives only 12-bits. But note that I'm not using my scans for this, I'm using the target scans you provided.

Quote
QUES 3: What are the RGB values of GS20-23 patches for one of your IT8 scans using Gamma 1.0?

I would have to dig that out for my scanner - it would be an interesting comparison. Here is what LPROF reports for your 16-bit linear and your 16-bit Gamma 1.8.

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 25, 2011, 09:19:19 am
Ok, for comparison here is data from my Sprintscan.

There is a problem with flare in the darkest patches, a lot of it due to the bright line separating GS22 and Dmax. This is after cleaning the mirror and lens.

Your Nikon also shows flare affecting those patches, but with less structure than the Polaroid. The advantage of the 2 extra bits in the Nikon gives you is also visible as better smoothness.

The flare is causing the DMAX patch to scan lighter in some of the channels than GS22. Interestingly, the Nikon has a bump in the red channel at GS22 that is not present in the Polaroid.

So the Kodachrome target is tougher to scan than it should be because of the lighter areas surrounding the darker GS gray scale patches. I think it's fair to assume that in real scans the darkest patches will be surrounded with dark patches of similar level, so that there would be less flare in those parts of the image. Or should I say that anomalies in the darkest tones caused by the profile will be more visible due to a lack of simultaneous-contrast effects - that puts more demands on the accuracy of the profile than we are getting because of flare in scanning the target.

An approach to improve the target scan and therefore, hopefully, the profile in the darkest tones would be to reduce the flare in the target. One way would be to overlay a mask on the target slide to block the lighter structures surrounding the patches. Seems extremely fiddly to make such a tiny mask, though. A more practical approach would be to measure the darkest patches by hand and editing the measurement file with flare-free estimates.

Time for more experiments - you better get right on it, Guy!

There are a million worms in this can you have opened.....

attached:

Polaroid Sprintscan GS and DMAX measurement snippet
Polaroid SS dark patch image, lightened
Nikon dark patches, lightened
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 25, 2011, 12:14:26 pm
Thanks Cliff for all the data. I'll pour over it in the next few days. I've just about got this profiling to a level as good as it's going to get for my setup (Nikon Coolscan V ED, Kodachrome and Argyll via Coca). It's now fairly clear that blacks are a challenging area when profiling. There are several reasons for this in my setup:

• Light entering the scanner via the slide hole contributes a small amount of light to the deepest blacks. One measurement indicated about 0.2 RGB units (not yet double checked). I now cover the hole with black cloth when scanning, just in case.

• I have to use the Kodachrome setting because of ICE (dust removal). I'd rather not use it because the profiling removes Nikon's attempt at colour correction anyway. The problem is that the Kodachrome setting throws a light yellow wash over the entire slide, bleaching GS21, 22, and 23 to similar L* values (not yet double checked). I suspect that this yellow wash is applied after gamma is applied in NikonScan. This may explain some of the benefit of applying gamma – the yellow wash becomes a much smaller percentage of the higher gamma-corrected values of GS21-23. Instead of being similar, they are spread apart, allowing Argyll the space to match them to the target values.

• Human visual sensitivity (the L* curve) for shadows is higher than for highlights. Blacks have to be scanned much more finely.

• Kodachrome (and other slide types?) throws an additional burden on the scanner because the Density/Exposure curve exhibits a gamma of ~1.5, throwing the shadows deeper into darkness. Although this doesn't directly affect profiling accuracy, this increased gamma means the histogram of a typical slide will be weighted towards the black end (quite heavily in many cases) – and these shadow areas will require their levels stretched apart (increased contrast) to bring them into the light. Stretching the shadows without introducing errors, requires scanner and profile to have better accuracy that the L* curve alone would indicate.

• For GS20-23 scanned with low gammas (1.0 -?), my scanning/profiling process is not accurately transforming the scanner RGB values into IT8 L* values. I suspect these is an optimum value of gamma for my setup of around 1.8. To test this, I'll have to scan the target at various gammas to see which gives the closest match.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 25, 2011, 12:37:20 pm
That white streak between GS22 and 23 has been annoying me for a while. To fix the problem of the streak casting a flare on GS22 and 23, would there be any problem with working out the average L* in a representative small area in the centre of each, and then filling the whole of both patches with that average value?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 25, 2011, 09:13:18 pm
• For GS20-23 scanned with low gammas (1.0 -?), my scanning/profiling process is not accurately transforming the scanner RGB values into IT8 L* values. I suspect these is an optimum value of gamma for my setup of around 1.8. To test this, I'll have to scan the target at various gammas to see which gives the closest match.

It was your target scan that was used in the above post (http://www.luminous-landscape.com/forum/index.php?topic=54040.msg444279#msg444279) that showed error results for those patches. With the sRGB TRC, respectable DE errors of 2 or less are achievable in those patches, with your equipment.

Are you still using the 8-bit target scans to make profiles?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 25, 2011, 09:16:55 pm
That white streak between GS22 and 23 has been annoying me for a while. To fix the problem of the streak casting a flare on GS22 and 23, would there be any problem with working out the average L* in a representative small area in the centre of each, and then filling the whole of both patches with that average value?

That could work. Maybe use a value with an amount of flare subtracted from the average.

Or paint over the white streak with some ink.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 25, 2011, 11:25:31 pm
It was your target scan that was used in the above post (http://www.luminous-landscape.com/forum/index.php?topic=54040.msg444279#msg444279) that showed error results for those patches. With the sRGB TRC, respectable DE errors of 2 or less are achievable in those patches, with your equipment. Are you still using the 8-bit target scans to make profiles?

Always been 16-bit except when initially testing. I was being kind to Argyll when I mentioned a problem in my "scanning/profiling process". My first draft of the post you quote from was:

For GS20-23 scanned with low gammas (1.0 -?), Argyll does not accurately transform the scanner RGB values into IT8 L* values. I suspect these is an optimum value of gamma for my setup of around 1.8 to get around this Argyll problem. To test this, I'll have to scan the target at various gammas to see which gives the closest match.

Then I thought: Well, I can't be sure it's Argyll; the error might be occuring somewhere else in the entire scanning/profiling process. So I changed the wording.

And a note to myself and anyone else reading all previous posts: figures given for average RGB values may be in error by as much as 0.5 RGB units because of the incorrect way Photoshop calculates Histogram > Average – it uses 8-bits instead of the maximum available.

Now for a rant. Don't you just love the way PS reports 16-bit L* values as 0-32768 when L* should range from 0-100? Can you imagine someone wanting a more accurate percentage reading of some measurement, say, 67%, so they are told a 16-bit accurate figure is 21953%. Ridiculous. And another thing about 16-bit L* and Photoshop: the values are actually 15-bit, not 16-bit.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 26, 2011, 12:43:46 am
Then I thought: Well, I can't be sure it's Argyll; the error might be occuring somewhere else in the entire scanning/profiling process. So I changed the wording.

Or the problem might be the in the choice of Argyll options that the Coca front end is using.

I mentioned SCARSE before. My correspondence with its author some years ago gives some insight into the choices that profiler writers have to make. He found problems with the curve fitting of the tranfer curve for Kodachrome, because the code assumed that the gray-scale should be neutral, as it is in most "normal" slide films. Of course, if you force it to be neutral it won't match the targets and Lab delta E goes out the window. He had to alter the code to do a robust non-linear fit to all the data to accommodate Kodachome's non-neutrality. I can imagine similar problems with other profilers, but with the authors never taking Kodachrome's perculiarities into account.

Moral of the story, profilers that work well with normal slide films are not necessarily going to work well with Kodachrome. Testing is needed. I wonder how well the commercial profilers do? I have inCamera, which makes extremely accurate profiles, but the original version I have has 8-bit limitations that make it unsuitable for my uses with Kodachrome. (I understand that the newer version doesn't have the 8-bit limitations.) LPROF seems to do a good job, especially with the sRGB TRC. I believe there is a Mac binary version available.

Quote
And a note to myself and anyone else reading all previous posts: figures given for average RGB values may be in error by as much as 0.5 RGB units because of the incorrect way Photoshop calculates Histogram > Average – it uses 8-bits instead of the maximum available.

Now for a rant. Don't you just love the way PS reports 16-bit L* values as 0-32768 when L* should range from 0-100? Can you imagine someone wanting a more accurate percentage reading of some measurement, say, 67%, so they are told a 16-bit accurate figure is 21953%. Ridiculous. And another thing about 16-bit L* and Photoshop: the values are actually 15-bit, not 16-bit.

A lot of "gotchas" to be found along the way, to be sure.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 26, 2011, 01:18:33 am
Stray Light Test
An IT8 target was scanned with the slide hole covered and uncovered, with sRGB and Gamma 1.0, to test the effect on the scan of stray light. The scans for each gamma were placed into the one image, resulting in two layers (covered and uncovered) for each of two images (sRGB and Gamma 1.0).

Luminance was measured by selecting a small area in the middle of GS19-22, then applying Filter > Blur > Average for each layer (layer must be selected and visible when applying filter, but not necessarily on top) to ensure exactly the same area was covered. Then using Info > Eye Dropper > Actual RGB > 16-bit, the 16-bit values of the averaged area were entered into the formula: (0.2126R + 0.7152G + 0.0722B)/327.68 to convert the RGB values to relative luminance as a percentage. The coefficients used are those for sRGB as I did not know the values for the Coolscan colour space.

The results are shown in the attachment. For each GS patch, the luminance value for the "uncovered" scanner is given, followed by the change in luminance when the slide hole was covered. Blocking the slide hole has a measureable effect on lightening the densest black patches. Moral: cover the slide hole.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 26, 2011, 03:29:13 am
Yellow Wash Test
Targets were scanned at Gamma 1.0 and Gamma 1.8 with ICE off, scanned with either the POS or Kodachrome setting to see if the yellow wash of Kodachrome bleaches out the highest densities.

Measuring luminance was done by selecting a small area in the middle of GS19-22, then Filter > Blur > Average for each layer (layer must be selected and visible when applying filter, but not necessarily on top) to ensure exactly the same area was covered. Then using Info > Eye Dropper > Actual RGB > 16-bit, the 16-bit values of the averaged area were entered into the formula: (0.2126R + 0.7152G + 0.0722B) / 327.68 to convert the RGB values to relative luminance as a percentage. The coefficients are those for sRGB as I did not know the values for the Coolscan colour space.

See attachment for results. There is an approximately linear relationship of ~10:1 between the values for the Positive setting for the two gammas. A similar relationship exists for the Kodachrome setting. This proves there is no bleaching of high density patches caused by the yellow wash of the Kodachrome setting, as I mistakenly assumed when measuring with Photoshop's dopey Histogram > Average. i.e. both sets of figures should be suitable for profiling as they show a good spread between each patch.

So it's not stray light or yellow-wash that are causing inaccurate profiling of the high density GS patches. The finger is turning towards Argyll or the Coca interface. But it's not pointing directly just yet.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 26, 2011, 10:03:30 am
sRGB warning!

Converting to sRGB damages to the darkest patches, and throws a number of color patches out of gamut.

Compare converting to ProphotoRGB and adjust the white point in Levels to show details in GS20+.

Also, if you examine individual color channels after converting to sRGB, you can see which colors are going out of gamut - they are the patches that go black in one or more channels. The black channels are channels that would have to have negative values to represent the color in sRGB, but have been clipped to zero. AdobeRGB is almost as bad.

It seems that it would be best to convert from the scanner profile to ProphotoRGB for all editing/correction (all the while in 16-bit) and delay conversion to sRGB until the very end, after all adjustments to lighten the dark tones have been made.

attached:
1 crop converted from scanner profile to sRGB plus levels adjustment
2 crop converted from scanner profile to ProphotoRGB plus levels adjustment
3 red channel after conversion from scanner to sRGB
4 green channel after conversion to sRGB


Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 26, 2011, 10:04:22 am
attached:
5 blue channel after conversion to sRGB
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 26, 2011, 09:25:06 pm
You beat me to this one. Yesterday when I was trying to compare L values of the the darkest patches after profiling, I was having difficulty converting to other colour spaces and retaining the same L values. I was going to ask you about it. Thanks for the images. Will check them out in detail later on.

Q1: When I finally decide on a scanning method, would you be able to generate what you think is the best possible profile from my IT8 scan using LPROF, and then generate error reports for your profile and my profile? I want to compare LPROF and Coca/Argyll. I would upload the scan at 2000 dpi, 16-bit Tiff with embedded profile.

Q2. Any idea why the IT8 designers put that white line between GS22 in GS23? It's in the worst possible place – bright white next to the darkest grays.

Q3. I've asked the lads over at the Photoshop forum about this next one, but nothing useful came of it. After a bit of a lead in about scanning IT8 targets, I said:

"I've been working for years with my screen set to a certain profile (My Books) set to colour temperature D50 and gamma 1.8 from memory, that quite accurately mimics what I see from a Xerox iGen printer with what I see on screen. Now, however, the only destination for my images will be my monitor at home, friend's High-definition TVs, or the digital projector at the local cinema, which most likely all have an image space of Rec. 709 or its virtual equivalent, sRGB, both of which use a gamma of 2.2 (or thereabouts) and D65. So I changed my monitor setting to sRGB with the result that the images that I have edited in PS (with monitor set to My Books) now look different. And I can't figure out how to make images in PS with monitor set to sRGB, look the same as when monitor is set to My Books."

Is it possible to mimic a monitor space from within another monitor space? I've tried soft-proofing, applying and converting to profiles, nothing seems to give the same look within sRGB as when looking at the image in My Books, which give a yellow glow to the images, a bit like when viewing a slide.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 26, 2011, 11:19:36 pm
Q1: When I finally decide on a scanning method, would you be able to generate what you think is the best possible profile from my IT8 scan using LPROF, and then generate error reports for your profile and my profile? I want to compare LPROF and Coca/Argyll. I would upload the scan at 2000 dpi, 16-bit Tiff with embedded profile.

Yes. Why not scan at the 4000dpi maximum?

There are some other profilers that I can try as well, such as SCARSE, Argyll (full-blown command line), inCamera. If there are others following this thread, maybe they can jump in with some others. Let's make it a shoot-out.

There's also Silverfast and Vuescan. Did you see Mark Segal's review of the Plustek 7600 (http://www.luminous-landscape.com/images-105/Plustek-7600.pdf) here on LuLa, where he carefully evaluates Silverfast profiles for Kodachrome and Fujichrome on several scanners, including the Nikon 5000?

Quote
Q2. Any idea why the IT8 designers put that white line between GS22 in GS23? It's in the worst possible place – bright white next to the darkest grays.

Don't know. I agree, it's a big problem.

Quote
Q3. I've asked the lads over at the Photoshop forum about this next one, but nothing useful came of it. After a bit of a lead in about scanning IT8 targets, I said:

"I've been working for years with my screen set to a certain profile (My Books) set to colour temperature D50 and gamma 1.8 from memory, that quite accurately mimics what I see from a Xerox iGen printer with what I see on screen. Now, however, the only destination for my images will be my monitor at home, friend's High-definition TVs, or the digital projector at the local cinema, which most likely all have an image space of Rec. 709 or its virtual equivalent, sRGB, both of which use a gamma of 2.2 (or thereabouts) and D65. So I changed my monitor setting to sRGB with the result that the images that I have edited in PS (with monitor set to My Books) now look different. And I can't figure out how to make images in PS with monitor set to sRGB, look the same as when monitor is set to My Books."

Is it possible to mimic a monitor space from within another monitor space? I've tried soft-proofing, applying and converting to profiles, nothing seems to give the same look within sRGB as when looking at the image in My Books, which give a yellow glow to the images, a bit like when viewing a slide.

You might want to spin that question off into a separate thread. Did you try converting the My Books profile to sRGB using Absolute intent? That should preserve the relatively yellow D50 white within the D65 target space.

Earlier (http://www.luminous-landscape.com/forum/index.php?topic=54040.msg443645#msg443645) I provided a profile for your scanner that has a tungsten white point. The idea is similar - when you convert using Absolute intent the tungsten yellowness is carried over to the target color space. It could turn your digital projector into a tungsten-bulb slide projector. Did you try it?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 27, 2011, 02:05:49 am
Yes. Why not scan at the 4000dpi maximum?
Coca doesn't accept 4000 dpi files, at least on my setup. Plus they would be 100 MB files. Time consuming to upload, but possible.


Did you see Mark Segal's review of the Plustek 7600 (http://www.luminous-landscape.com/images-105/Plustek-7600.pdf) here on LuLa, where he carefully evaluates Silverfast profiles for Kodachrome and Fujichrome on several scanners, including the Nikon 5000?

I just had a quick look. My own experience with the 7600's smaller brother, the 7400 (I think), was not something you'd write home to mum about. I'm going to borrow it again when I finish with the Kodachrome, and put it through it paces. I will say though, that the 7400 looks and feels cheap and plasticy next to the Nikon.


[re white line] Don't know. I agree, it's a big problem.
I was thinking of drawing over it with a felt pen, but the line is only 0.1 mm wide. Could be tricky.


I provided a profile for your scanner that has a tungsten white point. The idea is similar - when you convert using Absolute intent the tungsten yellowness is carried over to the target color space. It could turn your digital projector into a tungsten-bulb slide projector. Did you try it?
Tried it, and had a horrible feeling it is an accurate representation of how slides look compared to digital – washed out, fuzzy, and dim. What I'm going to do is to get a scanned image on screen, and project the same image onto a mini slide-screen I've built which sits on the desk next to my monitor. Then I'll take a photo of both, side by side, and post.

I have already done a similar test with a mate's 46" Bravia screen showing a scanned slide, and next to it was the same slide projected at the same size. 3.5 stops difference in brightness (Bravia's favour), with most slides looking better on the Bravia, though a few projected images had the edge. And this was before profiling, and with minimal editing. I'll be repeating the tests soon, and expect that digital versions of the slides will look so-significantly better that I may never show slides again.


Shoot Out
Might be useful, but from what I've learned so far, there is so much optimising that has to be done, you may never be able to fairly compare various devices – unless each has been exhaustively optimised. I was hoping to put together a PDF, The Art and Science of Scanning Kodachrome, that would work for any setup, but I'm not sure that will be possible now. I'll still put it together, but the title might have to change. Something like Kodachrome and Black Magic – How Scanning and Satanic Practices Intertwine. Forget the science, it mostly black magic in my experience.

Another horrible Feeling
After all this time spent profiling, I find I'm ending up with mediocre images that require significant editing in PS – though I do have a technique that is semi-automatic and only takes 5-10 minutes per slide. I look at the editing required and I think: Why bother profiling? So that's another set of tests: to compare the same editing approach to profiled and unprofiled images. The former had better give better results, or I'll start practising satanic rites on my IT8 targets – starting with surgery to remove that white streak.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 27, 2011, 04:17:59 am
Optimum Scan Gamma Tests
I scanned a Kodachrome IT8 target at Gamma 1.0, 1.2, 1.4, 1.6, 1.8, 2.0 and sRGB (Nikon), then generated a profile using Coca. The profiles were then applied to the respective scans and the L* values measured for these patches: GS0, GS5, GS10, GS15, GS19, GS20, GS21 and GS22. The measurement was done on a small section within each patch (~25% of the area), each section being identically placed and sized for each scan.

The difference between the actual value of each patch (taken from the IT8 data file) and the measured value was calculated and then graphed: GS values horizontally and absolute L* difference vertically. See attachment.

Optimum Gamma
The optimum scan gamma for my setup when using a Coolscan V ED and profiled with Coca is about gamma 1.6.

The graph clearly shows that the accuracy of Argyll's fitting of the RGB data to the L* data depends on the shape of the data it is presented with. The closeness of the modeling alters as the input numbers alter their shape. The wild card in the whole process is that the GS23 patch is being affected by the white streak on its GS22 side – the streak throws flare across both patches causing them to be lighter than they should be, with greater effect on GS23 because of the direction the scanner scans (from left to right for the Coolscan). The flare would have a strong impact on curve fitting at high densities, because GS23 is scanned as being lighter than it should be – and lighter than GS22 (it should be darker). This lightening, together with inaccuracies because of the curve fitting process, throws the profiling into maximum error in the darkest sections – just where minimum error is required to accurately reproduce shadow detail.

Next step - faking a GS23 patch
1. I need to determine the maximum density that the Coolscan can measure by scanning a dense black from another slide, to make sure it is the same (or darker) than GS23.

2. When I have established that the density of GS23 is within the Coolscan's capabilities, I can alter the Lab value of the scanned GS23 patch in Photoshop so that it approximates the real figure.

3. Then get rid of some of the flare on the GS22 side of the white streak (again in PS), and reprofile. This should allow the profile to more accurately model the densest blacks because there won't be the unexpected density reversal of GS22 and GS23.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 27, 2011, 08:02:54 am
Tried it, and had a horrible feeling it is an accurate representation of how slides look compared to digital – washed out, fuzzy, and dim. What I'm going to do is to get a scanned image on screen, and project the same image onto a mini slide-screen I've built which sits on the desk next to my monitor. Then I'll take a photo of both, side by side, and post.

Did you notice that (for any of the profiles) the image is dimmer with Absolute intent, compared to Relative? Absolute gives the actual Lab numbers of the target. Relative involves remapping the gray-scale to the white and black of the Profile Connection Space. So the Relative intent, which is the one normally used in practice, results in a stretching of the gray-scale, causing the Lab numbers to depart from the target values. Seems to have the largest affect in the lowest densities, but the high-densities are also affected. This is according to ICC recommendations. Another potential pitfall in the measurement process.

Quote
Shoot Out
Might be useful, but from what I've learned so far, there is so much optimising that has to be done, you may never be able to fairly compare various devices – unless each has been exhaustively optimised. I was hoping to put together a PDF, The Art and Science of Scanning Kodachrome, that would work for any setup, but I'm not sure that will be possible now. I'll still put it together, but the title might have to change. Something like Kodachrome and Black Magic – How Scanning and Satanic Practices Intertwine. Forget the science, it mostly black magic in my experience.

Not a shootout among devices, just among the profilers, using the same target scans.

Quote
Another horrible Feeling
After all this time spent profiling, I find I'm ending up with mediocre images that require significant editing in PS – though I do have a technique that is semi-automatic and only takes 5-10 minutes per slide. I look at the editing required and I think: Why bother profiling? So that's another set of tests: to compare the same editing approach to profiled and unprofiled images. The former had better give better results, or I'll start practising satanic rites on my IT8 targets – starting with surgery to remove that white streak.

Hopefully all the black magic will at least result in better shadow detail, but that remains to be seen!
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 27, 2011, 08:07:03 am
Next step - faking a GS23 patch
1. I need to determine the maximum density that the Coolscan can measure by scanning a dense black from another slide, to make sure it is the same (or darker) than GS23.

2. When I have established that the density of GS23 is within the Coolscan's capabilities, I can alter the Lab value of the scanned GS23 patch in Photoshop so that it approximates the real figure.

3. Then get rid of some of the flare on the GS22 side of the white streak (again in PS), and reprofile. This should allow the profile to more accurately model the densest blacks because there won't be the unexpected density reversal of GS22 and GS23.

I agree. That looks like a good plan.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 28, 2011, 01:58:54 am
I had some time to kill so I tried the measurement modification trick to see how it can affect profile accuracy.

I decided to use the Argyll command line tools, in order to have the possibility of modifying the measurements before generating the profile, and generating error reports. Coca doesn't allow these things.

1. First I generated an Argyll TI3 measurement file from the target scan and Q60 reference file:
scanin -v "kodachrome it8 gamma 1.0 .tif" it8.cht K3199910.Q60

2. I took the measurements into Excel and played around with linearizing the RGB numbers of the GS patches against the linear XYZ data of the Q60 reference. If you examine the first Excel graph below you can see that it is necessary to modify more than just the GS23 (DMAX). I ended up fitting a linear curve to GS00 through GS12, and used the slopes of the curves to generate new RGB numbers for GS12 through GS23 from the target XYZs, as follows:

Red = X * 1.1648
Green = Y * 1.0487
Blue = Z * 1.5226

Note that the Argyll scales the RGB measurements to the range 0..100.

The second Excel chart below shows the linearized curves.

3. The linearized RGB numbers were inserted into the TI3 measurement file.

4. A high quality profile was generated:
colprof -v -D"KodaChrome GB G1 Nikon MOD" -qh -ax "kodachrome it8 gamma 1.0 MOD"

5. Error report generated:
profcheck -v2 "kodachrome it8 gamma 1.0 MOD.ti3" "kodachrome it8 gamma 1.0 MOD.icm" > kcit8g1MOD_err.txt

The errors for all patches are very good - peak err = 3.093032, avg err = 0.217543. The gray-scale errors are much lower with the modified measurements. Of course, these are the errors of matching phony data, but the hope is that the phony data looks more like the shadow regions of real slides, with less flare than the Q60 target.

Here are the Lab DE errors of the GS patches:
[Lab Error DE] Patch No.
[0.151678] GS00:
[0.060774] GS01:
[0.007665] GS02:
[0.024161] GS03:
[0.010669] GS04:
[0.009631] GS05:
[0.008370] GS06:
[0.003975] GS07:
[0.049782] GS08:
[0.013283] GS09:
[0.030466] GS10:
[0.022639] GS11:
[0.305744] GS12:
[0.042647] GS13:
[0.283512] GS14:
[0.379665] GS15:
[0.492750] GS16:
[1.368748] GS17:
[0.874377] GS18:
[0.885278] GS19:
[0.889087] GS20:
[0.729789] GS21:
[0.438823] GS22:
[0.562788] GS23:

The proof will be in the viewing - here is the MOD profile (https://sites.google.com/site/cliffpicsmisc/home/profiles/kodachromeit8gamma1.0MOD.icm?attredirects=0&d=1) for you to test with your images.

Here is the  modified IT8 measurement file (https://sites.google.com/site/cliffpicsmisc/home/profiles/kodachromeit8gamma1.0MOD.ti3?attredirects=0&d=1).

Here is the error file (https://sites.google.com/site/cliffpicsmisc/home/profiles/kcit8g1MOD_err.txt?attredirects=0&d=1).
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 28, 2011, 09:58:30 am
Guy,

Despite the favorable error numbers, I am not seeing any visible improvement. The gray-scale looks just as smooth with any profile, as long as the scan is 16-bit.

I have only your 16-bit IT8 target scan to look at, and you must admit it's slightly boring. It would be nice to test profiles with your Example 5 scan, or other image in 14/16-bit, as it looks like there might be a huge amount of recoverable shadow detail in the very-lowest bit-depths that could be affected by these profiles. I tried the MOD profile on my 12-bit Sprintscan scans, but can't make out a difference within their 12-bit range.

At this point it's still not clear to me what visible improvement you're shooting for or is even possible.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 29, 2011, 12:05:48 am
I have discovered yet another anomoly in PS (the reason I posted, then deleted, then reposted). When viewing images at 50% in PS on my system, under certain circumstances the blacks in images look significantly different, and the Lab values are reported incorrectly. I have just averaged a small square in GS20 (Gamma 1.0) and measured it at 1635. I get a different figure when I zoom out. Anyway, now that I am aware of it, I can proceed.

I have uploaded three of my recent IT8 scans at gammas 1.0, 1.2, 1.6 with the relevant profile applied. On my system, the only change I can see as I tab from one to the other, is in the GS17-23 area. All other colours look identical, not all for these three images, but for all my IT8 scans. Do you see any difference in GS17-23 as you tab from one to the next? I certainly do on my system. As I tab from Gamma 1.6 -> 1.4 -> 1.0 the blacks become lighter and a haze is cast over them – the same effect I see in the darkest area of some slides.

If GS20-23 look different on your monitor, we've got a problem – we're not seeing the same thing. Download the files (IT8 Gamma 1.0 to 1.6.zip) from: http://www.mediafire.com/?121gpv6pt7rjj6j

I'll upload some of my reference scans today – crops to areas with lots of blacks.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 29, 2011, 03:36:29 am
Do you see any difference in GS17-23 as you tab from one to the next? I certainly do on my system. As I tab from Gamma 1.6 -> 1.4 -> 1.0 the blacks become lighter and a haze is cast over them – the same effect I see in the darkest area of some slides.

Yes, I see the blacks become lighter just as you have described.

Strangely enough, after applying 1.5 on the middle slider in Levels to lighten those patches, the effect is reversed - the blacks become darker as I tab from Gamma 1.6 -> 1.4 -> 1.0. I also see that the higher scan gammas show more noise.

But if the 1.5 gamma adjustment is done with Image/Adjustment/Exposure, the lightness is more similar among the scans and doesn't change very much while tabbing - but I can still see the increased noise at Gamma 1.2 and 1.6.

It also seems to matter whether the image has been converted to another color space or not.

Weird.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 29, 2011, 07:46:08 am
I have uploaded 4 sets of reference images to: http://www.mediafire.com/?35wnw6cxd7vm1

Each folder contains four versions of a 16-bit image: G1.0, G 1.5, G1.8, all cropped the same at 4000 dpi; and a 1000 dpi full version so you can see the whole picture. The Gamma 1.8 versions have several curves attached, so you can see my efforts at editing. There is also an explanatory PDF.

Happy shadow looking!

QUES
I want to check if sRGB is the same as Nikon sRGB (a profile I can scan into if I decide to do so). In sRGB I generated 6 LAB patches:

• 1,1,1
• 2,2,2
• 3,3,3
• 4,4,4
• 8,8,8
• 12,12,12

I converted the sRGB to Nikon sRGB and then compared the RGB values. They were the same to within about 2 or 3 RGB units (16-bit). That means Nikon sRGB and sRGB are identical, doesn't it?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 29, 2011, 12:00:59 pm
Thanks Cliff, for the Lab script. Now that I can generate low L* values, I can insert a fake GS23. I'm wary of playing around with the target scan, but I really think that GS23 and to a lesser extent GS22, have been compromised by the white strip. I want to try and determine a suitable value for GS23 by extrapolating from the other patches. Here is a list of IT8 "L" values in pairs, the first is the IT8 value, the second is the measured value from the unprofiled scanned slide. The first figure in each case is for GS11, then GS15, and GS18-23.

Gamma 1.0
39.86, 42.79
20.67, 21.17
10.45, 10.1
7.29, 7.18
4.27, 5.10
2.80, 4.34
1.07, 3.75
0.51, 3.82

Gamma 1.6
39.86, 59.84
20.67, 40.86
10.45, 29.57
7.29, 26.21
4.27, 23.21
2.80, 21.84
1.07, 20.17
0.51, 20.41

I don't have software to correlate numbers, so I use this calculator: http://www.arachnoid.com/polysolve/index.html. When the Gamma 1 numbers are pasted into that calculator with linear fit, you can clearly see that the lowest two points, GS22 and GS23, are too high. Correlation for the linear fit is 0.989. Pretty good. So it looks like I should alter both GS22 and 23 downwards.

However, when the Gamma 1.6 figures are inserted, only the lowest point, GS23, is off by any significant amount. Correlation is extremely good at 0.999, suggesting that only very minor correction is needed.

The graphs seem to indicate for Gamma 1.0, both GS 22 and 23 should be lowered – as I would expect because of that white stripe. But for Gamma 1.6, the graph indicates that only GS23 needs adjusting. My guess is that lowering both to suit Gamma 1.0 may improve the profile for Gamma 1.0, but would it be as good an improvement as a slight lowering of only GS23 to bring it into line for Gamma 1.6?

I'm betting on the latter, as it only involves one slight modification to one patch. And I'm betting that as long as GS23 is lower than GS22, it won't make much difference what value it is. I don't think any of this will make much difference anyway, but it's a good way of learning about profiling.

If you think it is worthwhile trying both methods for the profile shootout, I'll give it a go. Looking at the graphs, how much lowering do you reckon? My idea is to remove GS23 from the graph, then insert its "X" value into the calculator (0.51), and see what "Y" value pops out – which will become the new L value for my fake GS23.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 29, 2011, 03:21:38 pm
I have uploaded 4 sets of reference images to: http://www.mediafire.com/?35wnw6cxd7vm1
...
Happy shadow looking!

Thanks! A lot of stuff to look at. It's a holiday weekend here so I won't be able to jump on it right away.

Looking quickly, #5 seems to have a herringbone noise pattern in it for some reason.

Quote
QUES
That means Nikon sRGB and sRGB are identical, doesn't it?

I'm not sure. I read on the Web that the Nikon profile might have a different sized TRC lookup table or something. Not sure of the significance. The ICC has a Profile Inspector (http://www.color.org/profileinspector.xalter) that lets you see all the parts inside profiles if you want to compare them.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 29, 2011, 04:15:09 pm
I'm betting on the latter, as it only involves one slight modification to one patch. And I'm betting that as long as GS23 is lower than GS22, it won't make much difference what value it is. I don't think any of this will make much difference anyway, but it's a good way of learning about profiling.

If you think it is worthwhile trying both methods for the profile shootout, I'll give it a go. Looking at the graphs, how much lowering do you reckon? My idea is to remove GS23 from the graph, then insert its "X" value into the calculator (0.51), and see what "Y" value pops out – which will become the new L value for my fake GS23.

Well, it can't hurt to try....

My experiment with Argyll described above resulted in a profile ("MOD") that, despite having smaller errors,  introduces a bit of a red cast to the shadows. I think this is the opposite of what is wanted - if anything Kodachrome should have less red (higher red densities) in the shadows, if we're capturing exactly what's on the film. So far, the tint of the shadows is the main thing I am seeing that varies among the profiles. So not only do you want to eliminate the upward hook from flare at the bottom of the range, but also avoid a color cast by having a certain ratio among the channels in that range.

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 29, 2011, 10:23:52 pm
Guy, your reference scans and pdf have been very interesting. And by the way I like your writing style - are you a professional?

I think the quality problems you describe are mainly caused by editing the scans while still assigned to the scanner profile space. You should convert to a standard color work space such as Prophoto RGB before doing any editing.

The problem is that, until you convert to another space, the profile is in effect floating and hasn't permanently corrected the scanned colors. By editing in this state, you are changing the numbers that are sent as input to the profile. The result is that the changed RGBs get indexed into inappropriate parts of the look-up tables and whatnot within the profile.

By converting to a standard work space, the original scan values are corrected by the profile as they should be. After that any edits will be based on the corrected colors.

It's the difference between:

scan - edit - apply profile

and

scan - apply profile - edit

So we have been evaluating profiles quite differently! I always convert to a working space (either Prophoto RGB or a linear variant of it) before anything else.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 29, 2011, 11:33:52 pm
So we have been evaluating profiles quite differently! I always convert to a working space (either Prophoto RGB or a linear variant of it) before anything else.

We must be inhabiting the same ether space and unconsciously communicating. Last night I was thinking about what happens when you change colour numbers in the profile space and thought: colours must be moving away from what the profile is trying to correct, but couldn't quite convince myself that it was true. Now I know it is. Yet another trap for a novice Kodachrome scanner – don't edit in the profile space, even when testing. It will be interesting to see if editing in another space makes much difference to the shadows.


Today's Testing – Checking the Target itself
I want to find out how internally consistent my GS patches are. I'm going to do that by:

1. Generating a selection the size of a GS patch.
2. Move the selection over GS0, average it, and measure the L*.
3. Step Back, scale the selection to 80%, average, and measure L*
4. Repeat for 60%, 40%, 20%.
5. Move to GS1 … GS23 and repeat.

By doing the above, I'll know how internally consistent each patch is and whether it is being affected by boundary effects. I have a suspicion that the darker patches (say GS18 and up) are being compromised in their L* values (by lighter areas that surround them) either when manufactured or when scanned. A similar effect to what was discussed in another thread (when I asked about Hunt's "ideal" DH characteristic for a slide) in which I drew a graph of the effect of ambient light on a slide when projected – a tiny amount of ambient light has a large effect on the dark areas, but hardly any effect on the brighter areas.

QUES
Any idea how profilers average a patch? How much of the patch do they average? Do they discard values exceeding certain limits? Do they correct the average if the histogram shows a bias (it should be bell-shaped, I assume)?

It seems to me that to obtain an optimum profile, you should precondition the RGB data and the darker IT8 patches.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 30, 2011, 12:38:37 am
We must be inhabiting the same ether space and unconsciously communicating. Last night I was thinking about what happens when you change colour numbers in the profile space and thought: colours must be moving away from what the profile is trying to correct, but couldn't quite convince myself that it was true. Now I know it is. Yet another trap for a novice Kodachrome scanner – don't edit in the profile space, even when testing. It will be interesting to see if editing in another space makes much difference to the shadows.

I processed your Reference 18 Gamma 1 with the linear XYZ profile, converted to ProPhoto RGB, then a simple Exposure adjustment to brighten, adjust contrast, and cut down a little flare. Then Levels to make small adjustments to the Red and Blue channels (for simple alignment of the characteristic curves, as I was advocating in the Kodachrome threads a few months ago). I think this Gamma 1 version compares favorably with your edited version. Check out the color detail in the face. PSD file with Layers. (https://sites.google.com/site/cliffpicsmisc/home/guyburns/Ref18Gamma1.0_Prophoto_Exposure%26_Levels.psd?attredirects=0&d=1)

Quote
QUES
Any idea how profilers average a patch? How much of the patch do they average? Do they discard values exceeding certain limits? Do they correct the average if the histogram shows a bias (it should be bell-shaped, I assume)?

It seems to me that to obtain an optimum profile, you should precondition the RGB data and the darker IT8 patches.

This is from the Argyll documentation of the scanin measurement utility (http://www.argyllcms.com/doc/scanin.html), which of course is also used in Coca:
Quote
Normally scanin computes an average of the pixel values within a sample square, using a "robust" mean, that discards pixel values that are too far from the average ("outlier" pixel values). This is done in an attempt to discard value that are due to scanning artefacts such as dust, scratches etc. You can force scanin to return the true mean values for the sample squares that includes all the pixel values, by using the -m flag.

Scanin uses the information contained in a cht_format (http://www.argyllcms.com/doc/cht_format.html) file that specifies the locations of the patches to be read, and what fraction of the patch area to read.

The Scanin command seems to give reasonable measurements. It's completely automatic and produces a measurement file that can be altered with a text editor, so it's not necessary to modify the target scan itself. Then just run the colprof command to make the profile and you're done. Since only 2 or 3 commands are needed to build profiles, it takes only a short time to install and learn to use Argyll.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 30, 2011, 04:53:54 am
Since only 2 or 3 commands are needed to build profiles, it takes only a short time to install and learn to use Argyll.

The only problem with that idea is, you are obviously very experienced with this sort of thing and I'm not. I took one look at this instruction … "Making the tools accessible: You should also configure your %PATH% environment variable to give access to the executables from your command line environment" … My goodness. I'd rather learn Ancient Greek than enter a modern programming environment – although I was pretty good at entering a stack of punch cards encoded in FORTRAN into a PDP 11. But that was too many years ago now to contemplate.

Just the intensity of Argyll's documentation puts me off. Pity, but that's the way it is. I'm stuck with Coca.

Some interesting things came of reading about Scanin:

Resolution
It is designed to cope with a variety of resolutions, and will cope with some degree of noise in the scan (due to screening artefacts on the original, or film grain), but it isn't really designed to accept very high resolution input. For anything over 600DPI, you should consider down sampling the scan using a filtering downsample, before submitting the file to scanin.

I'll have to ask the author of Coca whether he scales the image before sending it to Scanin. I've been using 2000 dpi because 4000 dpi didn't work. Maybe even that's overkill.


Preconditioned or not?
Normally scanin has reasonably robust feature recognition, but the default assumption is that the input chart has an approximately even visual distribution of patch values, and has been scanned and converted to a typical gamma 2.2 corrected image, meaning that the average patch pixel value is expected to be about 50%. If this is not the case (for instance if the input chart has been scanned with linear light or "raw" encoding), then it may enhance the image recognition to provide the approximate gamma encoding of the image.

Another question for the author of Coca – has he left the gamma at it's default of 2.2? Maybe this explains why gammas higher than one seem to work better for me. It's becoming very confusing. From the above quote it appears there is an assumed preconditioning to gamma 2.2, yet the author of Argyll said in an email to me there was no preconditioning:

Argyll offers a number of models for input profiles (gamma + matrix, shaper + matrix, cLut). None of them use polynomial models to create curves or tables. I suspect that the described "pre-conditioning" approach really doesn't apply to models similar to a gamma+shaper model, since any pre-conditioning gamma will be cancelled out in a simple mathematical fashion. The Argyll shaper+gamma model uses a power curve plus higher-order shaping curves, so once again I'd be surprised if applying some extra gamma would change the result much, since it will be simply counteracted by the gamma curve parameter in the model.

For cLut type profiles, by default Argyll automatically creates a device curve that maximizes the linearity of the fit of the data points. I would guess that this has a similar effect to the "pre-conditioning" idea, but is tailored to the actual data, rather than being assumed. Note that applying an assumed "pre-conditioning" gamma may have a very detrimental effect for some device characteristics, if they don't meet the assumptions. The device values for instance might be linear light, or they could be gamma corrected. The profile has to account for both cases and everything in between.

In optimizing the model fit, Argyll generally uses the CIE94 delta E formula, since it has a better correspondence to visual errors than straight Lab delta E. Naturally a delta in XYZ space is not used, as it has poor visual correlation.


Preconditioning. Guy Burns reckons 1.6 is optimum for his setup; Hutchcolor mentions 3.0, 2.8; Cliff reckons 1.0; LPROF is somewhere in between; others say set GS11 to 100-115. What all this means, of course, is that they all work acceptably. Send a profiler an image with any gamma you like and you'll get a decent profile most of the time. Not necessarily the optimum, but decent.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 30, 2011, 07:05:00 am
Just the intensity of Argyll's documentation puts me off. Pity, but that's the way it is. I'm stuck with Coca.

I hear you. No matter, Coca is good. If you want to try something in Argyll, I can run it for you.

Quote
Resolution
It is designed to cope with a variety of resolutions, and will cope with some degree of noise in the scan (due to screening artefacts on the original, or film grain), but it isn't really designed to accept very high resolution input. For anything over 600DPI, you should consider down sampling the scan using a filtering downsample, before submitting the file to scanin.

In testing scanin I used my own 4000 ppi scans with no down-sampling and had no problems. The measured values in the famous GS20+ region are close to what I get by with careful, small, selection to avoid artifacts.

Have you checked to see whether gamma affects the averaging, if you're not scaling in a linear space?

Quote
Preconditioned or not?
Normally scanin has reasonably robust feature recognition, but the default assumption is that the input chart has an approximately even visual distribution of patch values, and has been scanned and converted to a typical gamma 2.2 corrected image, meaning that the average patch pixel value is expected to be about 50%. If this is not the case (for instance if the input chart has been scanned with linear light or "raw" encoding), then it may enhance the image recognition to provide the approximate gamma encoding of the image.

I haven't seen a problem with the feature recognition in linear gamma or any other.

Quote
Preconditioning. Guy Burns reckons 1.6 is optimum for his setup...

Perhaps only when the scanner profile is used as the editing space?

attached: here is the diagnostic image that scanin generated while reading your Gamma 1 scan, showing the patch recognition and reading areas.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 30, 2011, 10:14:57 am
Cliff – I'll go through your last few posts over the next day or two and pick out the gems. I've just about come to the end of my testing, other than seeing the effect of editing in a space other than the profile space. I don't reckon I'll see any improvement, because I based my choice of gamma 1.6 (and earlier, 1.8 ) by just looking at the raw, profiled, unedited scans. After I made the choice of what I thought was the best raw scan, only then did I try editing. I did try editing a couple of gamma 1.0 scans a week or so ago, but couldn't get them to look as good as a gamma 1.8 (this was before I decided 1.6 was even better).


L* Errors Caused By The Surround
Attached is a graph of L* errors for a variety of GS patches, measured with respect to each patch. This graph is not comparing L* with the IT8 value; the graph shows the variation in average L* in each patch as the linear size of the selection changes: 100%, 80%, 64%, 51% and 41% (funny numbers because I scaled by 0.8 each time). As expected, the 100% patch is most effected by the surround, and the effect is most pronounced for darker patches. Starting from a 100% selection (in reality about 95%), as the selection was made smaller for each patch, the errors became smaller. Below ~40%, there was very little change in the average value, so I chose 40% as my reference.

Summary
To avoid L* errors caused by the surround being of a different luminance than the patch itself, the optimum size of the selection should be around ~40%, resulting in a maximum error of less than 0.1 L* units. Errors remain low up to an 80% selection, but anything above that should be avoided, particularly for the darker patches, because the errors become significant.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 30, 2011, 11:12:59 am
I don't reckon I'll see any improvement, because I based my choice of gamma 1.6 (and earlier, 1.8 ) by just looking at the raw, profiled, unedited scans. After I made the choice of what I thought was the best raw scan, only then did I try editing.

Hmm. Without cranking up the lightness to evaluate the dark tones?

Let me know what you think of the gamma 1 version I did yesterday of Reference Scan 18. Crops attached.

Quote
L* Errors Caused By The Surround
Attached is a graph of L* errors for a variety of GS patches, measured with respect to each patch. This graph is not comparing L* with the IT8 value; the graph shows the variation in average L* in each patch as the linear size of the selection changes: 100%, 80%, 64%, 51% and 41% (funny numbers because I scaled by 0.8 each time). As expected, the 100% patch is most effected by the surround, and the effect is most pronounced for darker patches. Starting from a 100% selection (in reality about 95%), as the selection was made smaller for each patch, the errors became smaller. Below ~40%, there was very little change in the average value, so I chose 40% as my reference.

Summary
To avoid L* errors caused by the surround being of a different luminance than the patch itself, the optimum size of the selection should be around ~40%, resulting in a maximum error of less than 0.1 L* units. Errors remain low up to an 80% selection, but anything above that should be avoided, particularly for the darker patches, because the errors become significant.

That's interesting. I will play around with scanin later to see if its "robust mean" is similarly affected.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 31, 2011, 08:00:05 am
Hmm. Without cranking up the lightness to evaluate the dark tones?
When initially comparing the results of IT8 profiling, and in all subsequent comparisons, I simply opened my reference files (scanned at the various gammas), applied the relevant profile, and had a look at the image, whether it be light, dark, or in between. No cranking up the lightness, no bringing down the highlights, I evaluated all images unedited. Ref 18 Gamma 1.0 (for example) immediately struck me as having something wrong with it. When a similar effect was observed in some of the other Gamma 1.0 references, I decided that Gamma 1.0 was not optimum. Editing may have been able to bring it up to look similar to the others, but I haven't tested that yet because at this stage I want to compare the effect of IT8 profiling alone to determine the best starting point. Comparing editing at various gammas is a test yet to come, after I work out how to optimally scan using my setup. But what I have done, is to play around with the gamma I consider to be the best starting point, to see what improvements can be had. When I work out how to edit in the most effective, efficient manner, then I'll test that editing method with all gammas, subsuming the IT8 profiling and editing into one overall valid comparison. If Gamma 1.0 comes out on top, I'll be a Gamma 1.0 man. I'm not in love with a certain gamma – I want the best end result.


Problems with an Analytical Approach
One of the problems with the purely analytical approach of evaluating profiles by their calculated errors, is the possibility of internal errors being locked into the loop and not revealing themselves. Generating a profile is a closed loop: you assume you have an accurate IT8 target which is then scanned, you generate a profile, and the profiler compares the resulting colours with the IT8 colours and generates a profile. A closed loop works well if everything is reasonably accurate. Errors between target and scan (after profiling) will be reported as being low. If however, one of the patches has been incorrectly measured by Kodak, say, or has "surround" problems, then a low error may still be reported because there is nothing to tell the profiler that something is wrong. The profiler may be generating a very accurate profile for the target – it corrects for the inaccuracy, not aware of the inaccuracy – but the profile may be inaccurate for real-life slides. I'm reminded of a comment by the author of an Olympus book on SLR cameras from the 1980s in which he explains the use of 18% gray cards, and finishes with: "I can guarantee that if you go around photographing gray cards, you'll have perfect exposure every time". In the case of IT8 targets, the sentiment could be paraphrased: "If you profile IT8 targets, and generate error reports, you'll always get near perfect results." It's closed loop. Unless there is something wrong with the profiler, you must get good results.


Low Errors, Poor Profile
Let's say that Kodak reckons GS23 has an L* value of 0.5, when in fact the actual patch is 3.0. Let's assume GS23 scans at 3.0 (good scanner), and that the profiler does a good job at profiling. When the errors are calculated, they will therefore be reported as being low. But come to the scan of a real-life slide, which may also contain actual L* = 3.0 values, every time the profile sees L* = 3.0 in the real image it says: "That should really be 0.5". So it alters the colour by darkening, causing a visible problem in real-image shadows, but no obvious problem in the target (the 3.0 patch will be corrected to 0.5 to make the target look like it should).

The above problem happens in practice with my Kodachrome IT8 target (at least that's my explanation. Here is what I think is going on:

1. Kodak manufactures a master IT8 slide using a certain process that will be repeated in the manufacture of subsequent slides.

2. Each patch is measured with a light instrument, and an IT8 data file is created. Now, how is the L* measurement made? I don't know, so I'll take a guess. Does Kodak have a physical mask which sits over the slide and only allows D50 light to come through one entire patch at a time? Maybe, but I think unlikely (remember, I'm just guessing). I reckon they would throw a small circle of light through each patch, measure the colour of the circle of light, and make sure the circle doesn't approach too closely to the sides of the patch. Thus, Kodak measures the colour of only a small portion in the centre of each patch, ignoring the "surround" problem.

3. But, in reality, the surround problem comes back when the slide is scanned and subsequently profiled. For two reasons: (a) the scanner throws flare onto a dark area if it abuts a bright area; (b) the Kodachrome target appears to have been manufactured with a surround problem inbuilt. By looking closely at the top and bottom of GS15-23, through a 15x microscope eyepiece on a bright slide viewer, I think I can see a lighter stripe, top and bottom. If Kodak measured colour with a circle of light centred on the patch, they avoided this lighter area. If a profiler averages the entire patch, it does not avoid this area, and it's average will be different from Kodak's.

4. GS23, the blackest, is most affected: a bright white on the left, and three light grays on the other sides. With my scanner, GS23 scans higher than it should by 3 or 4 L* units – and worse, it scans lighter than GS22 (which has the same bright white on the right, and two light grays, top and bottom). Being lighter, GS22 is less affected. GS21 has light grays top and bottom only, and because it is lighter than GS22, is less affected again. GS21-15 are similarly affected top and bottom, but because they are progressively lighter, the "surround" problem is not as prominent.

5. This surround problem is most prominent for GS15-23. It should not cause problems in other areas, but it is probably best that only the central area of each patch be averaged by a profiler.


Explanation in Figures
We might have a situation where Kodak provides an L* number measured in a small central area, but the scanner/profiler combination profiles a different area, which, because of flare and the surround effect (inbuilt and from the scanner), causes the two values to be reconsilable for the target itself, but not reconsilable when applied to other images.

I'd better explain with figures (made up, just for explanation), which refer to three patches, the darkest two of which show an unexpected (but real) reversal in density, as do GS22 and GS23, caused by flare and surround problems. The first figure is an assumed scanned value of L*, the second is the target value. The figure in brackets is what an ideal profiler would do to the scanned value.

10.1  ->  9.3  (sent darker by 0.8 )
 3.6  ->  1.1  (sent darker by 2.5)
 3.7  ->  0.5  (sent darker by 3.2)

The IT8 profile, when applied to the scanned target, works perfectly. The image on screen looks just like the slide. The profiler has accommodated the flare, the scanner errors, the surround problem. Congratulations all around.

But something comes along to spoil the fun: a real-life slide. It also has L* areas of 10.1, 3.7 and 3.6, but they have not been compromised by flare or surround problems, just slight scanning errors. i.e. they scan at close to these values. Let's assume they scan at exactly these values for the sake of explanation. They are a part of a forest scene in shadow, and 10.1 is brighter than 3.7, which is ever so slightly brighter than 3.6. What does the profile do to them? Well, it transforms them by its inbuilt routines, the same as in the example above:

10.1  ->  9.3
3.7  ->  0.5
3.6  ->  1.1

The shadows of the real scene now have problems. The 10.1 shadow is close to what it should be, but the other two shadow details have been reversed in their brightness and made blacker. The profile, by correcting the IT8 target for flare, surround, and scanner errors, will apply those corrections to all other slides even when the first two problems don't exist.


Summary
Colour correction by profiling, in the presence of flare and surround problems, introduces errors into the shadows, errors that are not present in an uncorrected scan. The errors can be minimized by averaging only the central area of each patch (or by replacing the whole patch with a colour value taken from the centre area), and by faking GS23 to be more in line with what it should be (0.51).

Why is the problem not so apparent at higher scan gammas? Something for me to think about.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on May 31, 2011, 09:53:15 am
When initially comparing the results of IT8 profiling, and in all subsequent comparisons, I simply opened my reference files (scanned at the various gammas), applied the relevant profile, and had a look at the image, whether it be light, dark, or in between. No cranking up the lightness, no bringing down the highlights, I evaluated all images unedited. Ref 18 Gamma 1.0 (for example) immediately struck me as having something wrong with it. When a similar effect was observed in some of the other Gamma 1.0 references, I decided that Gamma 1.0 was not optimum.

If this is why you reject Gamma 1.0, I think you should take another look at it. This issue continues to pop up, and I think it would be good for the conversation to get past it once and for all.

My guess is that you are seeing an anomaly in the way Photoshop displays images in a linear gamma space. Photoshop can make linear gamma images appear extremely posterized. It is worst at 50% zoom or less. It should be better at 100% zoom. If I am right, the posterized look will disappear as soon as you convert to a workspace with a gamma greater than 1, or sometimes just "Layer/Flatten Image" will remove it.

So the display of gamma 1.0 images in Photoshop can be misleading, because images that are perfectly smooth can appear heavily posterized. I think it has to do with shortcuts in the way Photoshop quickly sends data to the display.

I raise this point again because, if this is the reason for your rejecting gamma 1.0 profiles out-of-hand, you are not giving them a fair evaluation. I have not been able to demonstrate to myself that gamma >1 profiles are so obviously superior.

My position on gamma 1.0 profiles is that they are at least as good as higher gamma profiles. I am not saying they are the only way to go. I did a lot of comparisons on real images yesterday that confirms my feeling that gamma 1.0 profiles are not worst, and may actually be slightly better in terms of visibility of noise in the darkest tones, an advantage visible only after extreme brightening of the dark tones.

I have been evaluating the profiles in the context of actual image editing. I will describe my method later, and I also want to address some of the many interesting points you made in your post. For now I will say that my preliminary conclusion is that is possible to make perfectly-equivalent images from Argyll/Coca profiles of any gamma (but only if they are of the S+M type)

Here are some images - does the first one show what you are seeing?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on May 31, 2011, 12:05:55 pm
If this is why you reject Gamma 1.0, I think you should take another look at it. This issue continues to pop up, and I think it would be good for the conversation to get past it once and for all…

Here are some images - does the first one show what you are seeing?
No, that's not what I am seeing. What I see is an obvious, but subtle, change in contrast and colour in the shadows that zooming-in doesn't fix, and that changing colour space/editing doesn't appear to be able to improve, but I haven't fully tested the latter across a range of images.

I'm not trying to convince you that Gamma 1.0 is inferior. It's simply as I see it on my setup. We'll have to agree to disagree: with my setup, Gamma 1.0 appears to give inferior results compared with higher gammas. On your system – different monitor, different platform – it may differ. I have a mate in a photographic club who is very interested in all of this. He runs a PC, so I'll check it out on his system, and get his opinion without prompting him which image I think is superior.

We'll move on – but I will still call it as I see it on my setup. And if I come to see it differently for whatever reason, I'll be saying so. I'm not anti Gamma 1.0, I'm pro best-image.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 01, 2011, 09:33:57 am
Problems with an Analytical Approach
One of the problems with the purely analytical approach of evaluating profiles by their calculated errors, is the possibility of internal errors being locked into the loop and not revealing themselves. Generating a profile is a closed loop: you assume you have an accurate IT8 target which is then scanned, you generate a profile, and the profiler compares the resulting colours with the IT8 colours and generates a profile. A closed loop works well if everything is reasonably accurate. Errors between target and scan (after profiling) will be reported as being low. If however, one of the patches has been incorrectly measured by Kodak, say, or has "surround" problems, then a low error may still be reported because there is nothing to tell the profiler that something is wrong. The profiler may be generating a very accurate profile for the target – it corrects for the inaccuracy, not aware of the inaccuracy – but the profile may be inaccurate for real-life slides. I'm reminded of a comment by the author of an Olympus book on SLR cameras from the 1980s in which he explains the use of 18% gray cards, and finishes with: "I can guarantee that if you go around photographing gray cards, you'll have perfect exposure every time". In the case of IT8 targets, the sentiment could be paraphrased: "If you profile IT8 targets, and generate error reports, you'll always get near perfect results." It's closed loop. Unless there is something wrong with the profiler, you must get good results.

The analytical approach in the present case seems useful in that it confirms the problem with increased errors in the darkest GS patches.

Quote
Low Errors, Poor Profile
Let's say that Kodak reckons GS23 has an L* value of 0.5, when in fact the actual patch is 3.0. Let's assume GS23 scans at 3.0 (good scanner), and that the profiler does a good job at profiling. When the errors are calculated, they will therefore be reported as being low. But come to the scan of a real-life slide, which may also contain actual L* = 3.0 values, every time the profile sees L* = 3.0 in the real image it says: "That should really be 0.5". So it alters the colour by darkening, causing a visible problem in real-image shadows, but no obvious problem in the target (the 3.0 patch will be corrected to 0.5 to make the target look like it should).

It's a real problem if we can't trust the Kodak target values. Apparently Kodak measured the target with some highly-specialized equipment, as the Q60 data are calculated from spectral measurements. The spectral data is available from Kodak in a QSP (http://ftp://ftp.kodak.com/GASTDS/Q60DATA/K3-Data/K3199910.QSP) file.

Another option is the Silverfast Kodachrome targets. They have hand- or batch-measured targets. The supplied Q60 measurements (http://www.silverfast.com/it8calibration/en.html) are quite different from the Kodak K3 one, for example in one of the Q60s, the Silverfast GS0-GS23 ranges from L* 2.02 to 77.89, while the Kodak ranges from 0.51 to 88.28. The Silverfast targets have a lower dynamic range and perhaps would generate less flare. I don't know if profiling with Silverfast targets would be better, but it would almost certainly be different. They have that same white line between GS22 and GS23, however.

Quote
The above problem happens in practice with my Kodachrome IT8 target (at least that's my explanation. Here is what I think is going on:

1. Kodak manufactures a master IT8 slide using a certain process that will be repeated in the manufacture of subsequent slides.

2. Each patch is measured with a light instrument, and an IT8 data file is created. Now, how is the L* measurement made? I don't know, so I'll take a guess. Does Kodak have a physical mask which sits over the slide and only allows D50 light to come through one entire patch at a time? Maybe, but I think unlikely (remember, I'm just guessing). I reckon they would throw a small circle of light through each patch, measure the colour of the circle of light, and make sure the circle doesn't approach too closely to the sides of the patch. Thus, Kodak measures the colour of only a small portion in the centre of each patch, ignoring the "surround" problem.

You are probably right about this, although they might have used both a mask and a restricted illumination circle in some kind of microscope arrangement.

Quote
4. GS23, the blackest, is most affected: a bright white on the left, and three light grays on the other sides. With my scanner, GS23 scans higher than it should by 3 or 4 L* units – and worse, it scans lighter than GS22 (which has the same bright white on the right, and two light grays, top and bottom). Being lighter, GS22 is less affected. GS21 has light grays top and bottom only, and because it is lighter than GS22, is less affected again. GS21-15 are similarly affected top and bottom, but because they are progressively lighter, the "surround" problem is not as prominent.

5. This surround problem is most prominent for GS15-23. It should not cause problems in other areas, but it is probably best that only the central area of each patch be averaged by a profiler.

I think a practical approach to the GS23 problem is to scan an unexposed frame, which would provide DMAX with the least-possible flare, and substitute that measurement for the GS23 patch. I am going to try to dig out an unexposed frame and try this on my scanner. Then GS22 and others could be asjusted from GS23 with some interpolation based on the real-world DMAX.

Quote
Explanation in Figures
We might have a situation where Kodak provides an L* number measured in a small central area, but the scanner/profiler combination profiles a different area, which, because of flare and the surround effect (inbuilt and from the scanner), causes the two values to be reconsilable for the target itself, but not reconsilable when applied to other images.

I'd better explain with figures (made up, just for explanation), which refer to three patches, the darkest two of which show an unexpected (but real) reversal in density, as do GS22 and GS23, caused by flare and surround problems. The first figure is an assumed scanned value of L*, the second is the target value. The figure in brackets is what an ideal profiler would do to the scanned value.

10.1  ->  9.3  (sent darker by 0.8 )
 3.6  ->  1.1  (sent darker by 2.5)
 3.7  ->  0.5  (sent darker by 3.2)

The IT8 profile, when applied to the scanned target, works perfectly. The image on screen looks just like the slide. The profiler has accommodated the flare, the scanner errors, the surround problem. Congratulations all around.

But something comes along to spoil the fun: a real-life slide. It also has L* areas of 10.1, 3.7 and 3.6, but they have not been compromised by flare or surround problems, just slight scanning errors. i.e. they scan at close to these values. Let's assume they scan at exactly these values for the sake of explanation. They are a part of a forest scene in shadow, and 10.1 is brighter than 3.7, which is ever so slightly brighter than 3.6. What does the profile do to them? Well, it transforms them by its inbuilt routines, the same as in the example above:

10.1  ->  9.3
3.7  ->  0.5
3.6  ->  1.1

The shadows of the real scene now have problems. The 10.1 shadow is close to what it should be, but the other two shadow details have been reversed in their brightness and made blacker. The profile, by correcting the IT8 target for flare, surround, and scanner errors, will apply those corrections to all other slides even when the first two problems don't exist.

I think this is a good explanation of the problem. The input data is not monotonic - due to flare, you could get the same Red value from either patch GS20 or GS23, for example. The profiler can't allow this, so has to do some smoothing in the curve fitting process to produce a 1:1 mapping of the scanner RGBs to XYZ or Lab. How well the profiler does this smoothing is the key. It might well be that introducing the gamma preconditioning to the data gives the Argyll algorithms an easier job of curve-fitting.

Quote
Summary
Colour correction by profiling, in the presence of flare and surround problems, introduces errors into the shadows, errors that are not present in an uncorrected scan. The errors can be minimized by averaging only the central area of each patch (or by replacing the whole patch with a colour value taken from the centre area), and by faking GS23 to be more in line with what it should be (0.51).

I did some tests with Argyll by changing the size of the measurement area on the patches. I did this by editing the BOX_SHRINK parameter in the it8.cht file. Coca uses the same it8.cht file, which you can find and change it in the Coca/Argyll/ref directory if you're feeling adventurous.

SAMPLE_ID   XYZ_X   XYZ_Y   XYZ_Z   RGB_R   RGB_G   RGB_B

BOX_SHRINK 3.5
GS20   0.50000   0.47000   0.34000   0.79935   0.50474   0.56849   
GS21   0.33000   0.31000   0.22000   0.74026   0.42002   0.42761   
GS22   0.13000   0.12000   0.10000   0.77738   0.34446   0.30561   
GS23   0.06000   0.06000   0.08000   0.79995   0.37243   0.32470

BOX_SHRINK 7.5
GS20   0.50000   0.47000   0.34000   0.78572   0.49316   0.55793      
GS21   0.33000   0.31000   0.22000   0.72382   0.41017   0.41799      
GS22   0.13000   0.12000   0.10000   0.73651   0.32489   0.28964      
GS23   0.06000   0.06000   0.08000   0.75358   0.34378   0.29991

BOX_SHRINK 12.0
GS20   0.50000   0.47000   0.34000   0.77368   0.48801   0.55766   
GS21   0.33000   0.31000   0.22000   0.71336   0.40633   0.41275   
GS22   0.13000   0.12000   0.10000   0.71942   0.31878   0.28413   
GS23   0.06000   0.06000   0.08000   0.73492   0.33332   0.29179   
You can see that as BOX_SHRINK (amount to exclude around the edges) increases, the average RGB numbers get smaller. BOX_SHRINK is in units relative to the patch size, for your targets the measurement areas are:

Full patch 46 X 92 pixels
Shrink 3.5 34 X 83 pixels (Coca default)
Shrink 7.5 20 X 68 pixels
Shrink 12 4 X 51 pixels

Unfortunately, although it seems some flare appears to be excluded, reducing the size of the measurement area this way still doesn't prevent GS23 from being lighter than GS22.

Scanning an unexposed slide to determine GS23/DMAX could be very helpful. Do you have one?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 02, 2011, 12:37:57 am
I did some tests with Argyll by changing the size of the measurement area on the patches. I did this by editing the BOX_SHRINK parameter in the it8.cht file. Coca uses the same it8.cht file, which you can find and change it in the Coca/Argyll/ref directory if you're feeling adventurous… Unfortunately, although it seems some flare appears to be excluded, reducing the size of the measurement area this way still doesn't prevent GS23 from being lighter than GS22.
If I can find the file, I'll certainly change it. Where do you find the patch size that Coca uses? Or is it the same as what I would measure in Photoshop? My graphs in a previous post, indicate you have to go down to 40% patch size before the RGB values stop decreasing. That means a box-shrink of about 28 to fully remove the effect of the gray surround and flare. But it doesn't work for GS23. The only way to fix GS23 would be to replace it with a new artificial patch of suitable value.

What I have done is to manually replace the whole of GS15-22 with the average of a centre 40% patch from each, and completely replace GS23. When I finish testing I'll upload the "improved" IT8 target. Initial testing has shown that it has no effect on anything other than the last few GS patches – looking good!

What would be the best way to generate a fake GS23? I used your Lab generator to generate the actual IT8 value. A better way, I assume, would be be work out the ratio between GS22 and GS23 and do it that way. Is the following technique valid?

Given the L* values of:
GS22 (IT8) = 1.07
GS23 (IT8) = 0.51

These are very close to 2:1, i.e. one exposure stop.

1. From a Gamma 1.0 scan, select 40% of GS22, and average it.
2. Open Colour Picker and set the foreground colour to the average of the 40% patch. Step Backward to undo the averaging on GS22.
3. Make a 100% selection of GS23 and Edit > Fill with the foreground colour.
4. Apply an Exposure correction of -1.0

Result: an artificial GS23 that should correlate well with its closest neighbour.

Quote
Scanning an unexposed slide to determine GS23/DMAX could be very helpful. Do you have one?
I'll have a look through my slides. Must be a piece of unexposed Kodachrome somewhere. If not, I know someone in Hobart who has a roll of unexposed Kodachrome. I'll ask him to snip a piece off the end. That would be suitable, wouldn't it?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 02, 2011, 02:27:54 am
If I can find the file, I'll certainly change it. Where do you find the patch size that Coca uses? Or is it the same as what I would measure in Photoshop? My graphs in a previous post, indicate you have to go down to 40% patch size before the RGB values stop decreasing. That means a box-shrink of about 28 to fully remove the effect of the gray surround and flare.

I measured it from a diagnostic image that Argyll makes. (I posted an example (http://www.luminous-landscape.com/forum/index.php?topic=54040.msg445535#msg445535) before.) A BOX_SHRINK of 7.5 will give you a 20 x 68 pixel measuring area from the full 46 x 92 patch, about 32%.

Quote
But it doesn't work for GS23. The only way to fix GS23 would be to replace it with a new artificial patch of suitable value.

I'm trying to replace the GS23 patch with DMAX scanned from unexposed KC. I scanned some unexposed K200 and K64. Haven't found any unexposed K25 yet, but I have a two frames that are severely underexposed (uxK25a and b in the following table). Here are the Polaroid scanner 16-bit RGB numbers I measured in PS:

K64       53   10   20
K200      20   2   8
uxK25a   53   16   44
uxK25b   44   17   36
average   42.5   11.25   27

GS23   98   58   78 (from Q60 scan)

I'll have to study this a little more - not sure which to use. I'm trying to splice a patch that has essentially no flare into a series of patches that do have flare.

As you discuss below, I guess you can say that GS23 should be little less than half the GS22 RGBs.  Red highest and Green lowest, I think it's important to preserve the ratio of R:G:B to prevent casts in the shadows.

Notice that despite having the same dyes, the different types of Kodachrome have a different DMAX. You can see this also in curves in the Kodak publications. I wonder which emulsion the Q60 are on? I might take mine out of its mount to take a look. It puzzles me that the Kodak Q60 is designed to be used with all Kodachrome types, but it's DMAX is only going to match one of them.

Quote
What I have done is to manually replace the whole of GS15-22 with the average of a centre 40% patch from each, and completely replace GS23. When I finish testing I'll upload the "improved" IT8 target. Initial testing has shown that it has no effect on anything other than the last few GS patches – looking good!

What would be the best way to generate a fake GS23? I used your Lab generator to generate the actual IT8 value. A better way, I assume, would be be work out the ratio between GS22 and GS23 and do it that way. Is the following technique valid?

Given the L* values of:
GS22 (IT8) = 1.07
GS23 (IT8) = 0.51

These are very close to 2:1, i.e. one exposure stop.

1. From a Gamma 1.0 scan, select 40% of GS22, and average it.
2. Open Colour Picker and set the foreground colour to the average of the 40% patch. Step Backward to undo the averaging on GS22.
3. Make a 100% selection of GS23 and Edit > Fill with the foreground colour.
4. Apply an Exposure correction of -1.0

Result: an artificial GS23 that should correlate well with its closest neighbour.

Looks like a good plan. The only problem I see is working in Lab - the L* values are easy to interpolate but what do you do about the a* and b* values?

Quote

I'll have a look through my slides. Must be a piece of unexposed Kodachrome somewhere. If not, I know someone in Hobart who has a roll of unexposed Kodachrome. I'll ask him to snip a piece off the end. That would be suitable, wouldn't it?

Unexposed but developed I hope!
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 02, 2011, 07:16:06 am
The only problem I see is working in Lab - the L* values are easy to interpolate but what do you do about the a* and b* values?

Well, since "a" and "b" and I don't get on, I'm going to ignore them! More thought is needed, but I think that as GS23 is at the end of the range, it may as well be faked as the exact IT8 value. Since all it's GS neighbours scan higher than they should anyway (because of flare and surround), a lower GS23 than what otherwise would appear on the slide, will have the effect of pulling the others into line when the profiler tries to regress them.

Some initial results from testing my modified IT8 profiles (I modified GS15-22 to reduce flare and surround effects, and faked GS23).

Conditions
1. Scanned the IT8 target at gamma 1.0, 16-bit, 4000dpi.
2. Modified GS15-22 as previously described, and replaced GS23 with its expected value.
3. Applied a linear sRGB profile. Saved it as sRGB 1.0.
4. Converted sRGB 1.0 to a gamma 1.6 sRGB profile. Saved it as sRGB 1.6.
5. Converted sRGB 1.0 to a true sRGB profile. Saved it as sRGB 2.2.
6. Downsampled the three versions to 1000 dpi and profiled each, using Lab and S+M.
7. Applied the new profiles to the relevant image.
8. For each image, compared Lab and S+M and chose the best, then compared the three images against each other and against my earlier optimum of Gamma 1.8 (with original profile).


Results
Very very close, using Ref 18 as the test image. Much closer than previously. Gammas 1.0 and 1.6 have a very slight wash over them in the darkest areas compared to sRGB and 1.8 sRGB, the latter having the densest blacks (very slightly denser than sRGB). However, the dense blacks of 1.8 come about by complete saturation in the red channel, whereas the sRGB's blacks are about 1 RGB unit away from saturation. 1.8 also has more contrast in the dark areas, but in some way it appears unnatural, as if the profile has gone too far in pushing blacks downwards. And since I don't like saturation, this relegates 1.8 to being behind sRGB.

In addition, there is a reversal of effect for S+M and Lab profiles. For Gamma 1.0, S+M gave the best result with slightly less haze than Lab. For G1.6 and sRGB, the reverse applies – Lab gave the slightly better result.

Using Reference 18 and applying to it an Lab profile generated from a modified IT8 target, sRGB comes out on top. But there is not much in it (just the slightest extra haze in the darkest areas of the other contenders), and the result may change when I apply the profiles to my other reference images.


Effect of Modified IT8 target on L*
Unless I've made an error somewhere, the results of using a modified IT8 target show an improvement over an unmodified target. Attached are the L* values for the darker GS patches for Gamma 1.0, 1.6 and 2.2 (sRGB). The figures confirm what I see under close visual inspection of the dark areas of Ref 18: that for the modified IT8 target, preconditioning to sRGB gives the best results; and for the unmodified target, Gamma 1.6 gives the best result. But there is not much difference, and all the gammas give very acceptable results, with any difference probably being swamped in the editing.


Gamut Warnings for Gamma 1.0, 1.6, and sRGB
When using unmodified profiles, gamma warnings are displayed for the three images, with approximately equal distribution across the three. These warnings disappear when converted to sRGB. For modified profiles, the warnings show a slight increase for Gamma 1.6 and sRGB as compared to Gamma 1.0, but again they disappear when converted.


QUES: Does this mean that sRGB has a wider gamut than the generated profile?

Also, when viewing Ref 18, I noticed that the funny-looking areas that occur at low magnification, exactly matched the out-of-gamut areas. I thought it was a PS problem, but it might be because the colours were out of gamut.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 02, 2011, 07:37:59 am
Regarding Box Shrink – I found it so I'll give it a go. Windows said it couldn't open the text file, so I opened it with Notepad. Since I don't like playing around with such things unless I am reasonably sure of what I am doing:

Q1: Will Coca still recognise the data file if I save it from Notepad?

Q2: Is this IT8.cht file newly created each time Coca is run, or is it a permanent fixture which Coca originally installs and then just looks at?

Q3: What's the mathematical relationship between Box Shrink and the size of the box?

Q4: If I change the image resolution, I assume the Box size changes, so I could end up in strife if I have a Box Shrink that is too big. Where is the Box size stored so that I can check the size?

Thinking about it – and there'll be more questions for sure – it might be easier if I just replace the sus patches with a 40% average. That, I can do, and I know it works.



Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 02, 2011, 08:57:45 am
Regarding Box Shrink – I found it so I'll give it a go. Windows said it couldn't open the text file, so I opened it with Notepad. Since I don't like playing around with such things unless I am reasonably sure of what I am doing:

Q1: Will Coca still recognise the data file if I save it from Notepad?

Yes. With Notepad, if you "Save" after changing the BOX_SHRINK number, it should be fine.

Beware that if you "Save AS" instead of "Save", Notepad will fight you and try to append a ".txt" suffix to the file name, which is no good. If you do a "Save AS", to save a backup version for example, change the "save as type" to "all files" and it will not append the ".txt" at the end of the name.

Quote
Q2: Is this IT8.cht file newly created each time Coca is run, or is it a permanent fixture which Coca originally installs and then just looks at?

It's permanent, and will only change if you edit it.

Quote
Q3: What's the mathematical relationship between Box Shrink and the size of the box?

I was afraid you were going to ask that. It's not so clear to me. There are other lines in the IT8.CHT file that define the BOX size. The gory details are in the Argyll documentation for the CHT File Format (http://www.argyllcms.com/doc/cht_format.html):

Quote
The physical units used for boxes and edge lists are arbitrary units (i.e. pixels as generated by scanin -g, but could be mm, inches etc. if created  some other way), the only requirement is that the sample box definitions need to agree with the X/YLIST definitions. Typically if a scanned chart is used to build the reference, the units will be pixels of the scanned chart.

The BOXES keyword introduces the list of diagnostic and sample boxes. The integer following this keyword must be the total number of diagnostic and sample boxes, but not including any fiucual marks. The lines following the BOXES keyword must then contain the fiducial mark, diagnostic or sample box definitions. Each box definition line consists of 11 space separated parameters, and can generate a grid of sample or diagnostic boxes:

    kl lxs lxe lys lye w h xo yo xi yi

-SNIP-

 w, h are the width and height of each box in the array.

-SNIP-

The keyword BOX_SHRINK marks the definition of how much each sample box should be shrunk on each edge before sampling the pixel values. This allows the sample boxes to be defined at their edges, while allowing a safety margin for the pixels actually sampled. The units are the same arbitrary units used for the sample box definitions.

Here is the beginning of the IT8.CHT where this stuff is specified:

BOXES 290
  F _ _ 1 1  616.0 1.5  615.5 358  1 358.5
  D ALL ALL _ _ 615 409 1 1 0 0
  D MARK MARK _ _ 14 14 1 1 0 0
  Y 01 22 A L 25.625 25.625 26.625 26.625 25.625 25.625
  X GS00 GS23 _ _ 25.625 51.25 0.0 358.75 25.625 0.0

BOX_SHRINK 3.5

I think the x and y dimensions of the GS patches are defined on the line beginning with "X", as 25.625 x 51.25 units. The number for BOX_SHRINK would be relative to these. I guess it's somehow scaled to the actual pixel dimensions of the scan. Somehow a BOX_SHRINK of 3.5 arbitrary units results in a 46 x 92 pixel patch being shrunk to about 34 x 83, if I've measure correctly.

I'm sorry but my eyes have completely glazed over at this point. What was that you said about the Argyll documentation? ???

Quote

Q4: If I change the image resolution, I assume the Box size changes, so I could end up in strife if I have a Box Shrink that is too big.

Later I will resize a target scan and check the scanin diagnostic image to see what happens.

Quote
Where is the Box size stored so that I can check the size?

See above, I think it's in the line starting with "X".

Quote
Thinking about it – and there'll be more questions for sure – it might be easier if I just replace the sus patches with a 40% average. That, I can do, and I know it works.

It would be useful to compare doing it both ways. If the automated results are good enough, it could save a lot of trouble for anyone who wants to duplicate your results.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 02, 2011, 09:04:50 am
Wow, you've been busy! It will take me a while to digest this. Can you package up some of your new profiles so that I can take a look?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 02, 2011, 05:32:45 pm
You wouldn't think something as simple as scaling could be made so difficult. If what you said about Box Shrink (BS) and the patch sizes after shrinking is correct, BS and the size of the box after shrinking (F, as a fraction of the horizontal original size) are related by:

BS = 12.81 (1-FH)

I want to scale the boxes to 40%, hence BSH = 12.8 (1-0.4) = 7.7.

The problem is, the vertical scaling is different, given by:

BS = 25.63 (1-FV)

The difference is caused by having a constant amount subtracted from different-sized edges. A problem arises when trying to make the box small enough (40%) to get away from the "surround" flare at top and bottom, the major component of the flare. You end up with a tall, narrow box given by this relationship:

FH = 2FV - 1.0

It is not possible using Box Shrink to scale vertically to 0.4, because the lower limit (when the box turns into a vertical line with no horizontal dimension) is 0.5. So, because of the way Argyll implements scaling, a figure higher than 50% has to be chosen, thus incurring additional L* errors. Therefore let's choose FH = 0.4, to give FH = 0.7. My hand drawn graph of a few posts back shows that for GS22 (GS23 will be faked) for a 70% patch, the error will be between these limits:

64% Patch … 0.1
80% patch … 0.2

So the optimum Box Shrink would appear to be 25.63 (1 - 0.7 ) = 7.7. This keeps the sample box away from the top and bottom surround, but still has a good horizontal spread to sample as many points as possible. But it won't be as good as using a true 40% box. On the other hand, the scaling will apply to all patches (and not just a few of the GS patches as I did manually), so overall the problem of flare and contamination may improve.

Note: the dimensions of each patch (25.625 x 51.25), appear to be the number of pixels when the image is scanned at 600 dpi, the figure that the Argyll documentation recommended. At 2000 dpi, I measured the dimensions as 85 x 182.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 02, 2011, 07:58:20 pm
For a 64% vertical patch, the horizontal scale will be 0.64-0.5 = 0.14. Too narrow. For 80%, it will be 0.3, which should be okay. So the optimum Box Shrink would appear to be 25.63 (1 - 0.8 ) = 5.1, say 5. This minimizes the L* error caused by the surround, and maximises the horizontal box dimension. But it won't be as good as using a true 40% box. On the other hand, the scaling will apply to all patches (and not just a few of the GS patches as I did manually), so overall the problem of flare and contamination may improve.

I ran your "Kodachrome IT8 Gamma 1.0.tif" scan thru scanin with BOX_SHRINK set to 5.1. Here are the Argyll IT8 measurement file (https://sites.google.com/site/cliffpicsmisc/home/profiles/kc_argyll_G1_meas_shrink5.1.it8?attredirects=0&d=1) and the diagnostic image (https://sites.google.com/site/cliffpicsmisc/home/profiles/diag.tif?attredirects=0&d=1) so that you can see the measurement areas. A little off center, I wonder if there be a way to fix that?

Maybe a change in the box dimensions can accomplish the vertical scaling you need?

There is a huge mailing list archive for Argyll (http://www.freelists.org/archive/argyllcms) that might have relevant info.
 
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 02, 2011, 11:03:03 pm
I made an error in the formula relating FH and FH, now corrected, which means the optimum BS is 7.7 (not 5.5). Not much difference.

GS Boxes off centre? How come all the others aren't off centre? Maybe it's best to keep BS at 3.5 if Argyll does things like shift boxes around.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 03, 2011, 03:39:37 am
I made an error in the formula relating FH and FH, now corrected, which means the optimum BS is 7.7 (not 5.5). Not much difference.

GS Boxes off centre? How come all the others aren't off centre? Maybe it's best to keep BS at 3.5 if Argyll does things like shift boxes around.

I updated the files. Looks better.

diagnostic image 7.7 (https://sites.google.com/site/cliffpicsmisc/home/profiles/diag7.7.tif?attredirects=0&d=1)
measurements 7.7 (https://sites.google.com/site/cliffpicsmisc/home/profiles/kc_argyll_G1_meas_shrink7.7.it8?attredirects=0&d=1)
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 03, 2011, 06:30:35 pm
There are a couple of interesting threads on the Argyll Mailing List about making scanner profiles.

Mentioned are two options for the colprof command that might be worth trying: -u and -r.

About the -u option:
http://www.freelists.org/post/argyllcms/Number-of-patches-well-behaved-printer,22

Quote
The next release will by default add some extrapolation patches up to the
device min/max values along the neutral axis when -u is used with input
profiles, to overcome the sometimes unexpected default extrapolation behaviour. You can
always override this with extra patches though, if you don't like what it does.

cheers,
        Graeme Gill.

from colprof doc (http://www.argyllcms.com/doc/colprof.html):

Quote
-u: cLUT style input profiles will normally be created such that the white point of the test chart, will be mapped to perfect white when used with any of the non-absolute colorimetric intents. This is the expected behaviour for input profiles. If such a profile is then used with a sample that has a lighter color than the original test chart, the profile will clip the value, since it cannot be represented in the lut table. Using the -u flag causes the lut based input profile to be constructed so that the lut table contains absolute color values, and the white of the test chart will map to its absolute value, and any values whiter than that, will not be clipped by the profile, with values outside the range of the test chart being extrapolated. The profile effectively operates in an absolute intent mode,  irrespective of what intent is selected when it is used. This flag can be useful when an input profile is needed for using a scanner as a "poor mans" colorimeter, or if the white point of the test chart doesn't represent the white points of media that will be used in practice, and that white point adjustment will be done individually in some downstream application.

-un: By default a cLUT input profile with the -u flag set will extrapolate values beyond the test chart white and black points, and to improve the plausibility of the extrapolation, a special matrix model will be created that is used to add a perfect device white and perfect device black test point to the set of test patches.  Selecting -un disables the addition of these extra extrapolated white and black patches.

I'm curious to see how -u will extrapolate down into the dark regions.

About -r option:
http://www.freelists.org/post/argyllcms/Verifying-profile-quality-of-LUTbased-scanner-and-printer-profiles,1

Quote
[argyllcms] Re: Verifying profile quality of LUT-based scanner and printer profiles
With the currently available release of Argyll, it is probably advisable
to specify a fairly high level of smoothing, by using -r 1.0 or so.
The next version will have better defaults in this regard, and
shouldn't usually need a -r parameter. This should result in a smoother
profile with a higher self fit dE.


    But is there any way to verify the 'smoothness' of the profile? In particular, I'm thinking about the discontinuities that might exist in the LUT tables.


The interpolation algorithm doesn't really allow discontinuities,
but it can have "overshoot" or "ringing".

The -r parameter specifies the average deviation of device+instrument readings from the perfect, noiseless values as a percentage. Knowing the uncertainty in the reproduction and test patch reading can allow the profiling process to be optimized in determining the behaviour of the underlying system. The lower the uncertainty, the more each individual test reading can be relied on to infer the underlying systems color behaviour at that point in the device space. Conversely, the higher the uncertainty, the less the individual readings can be relied upon, and the more the collective response will have to be used. In effect, the higher the uncertainty, the more the input test patch values will be smoothed in determining the devices response. If the perfect, noiseless test patch values had a uniformly distributed error of +/- 1.0% added to them, then this would be an average deviation of 0.5%. If the perfect, noiseless test patch values had a normally distributed  error with a standard deviation of 1% added to them, then this would correspond to an average deviation of 0.564%. For a lower quality instrument (less than say a Gretag Spectrolino or Xrite DTP41), or a more variable device (such as a xerographic print engine, rather than a good quality inkjet), then you might be advised to increase the -r parameter above its default value (double or perhaps 4x would be good starting values.)

Smoothing by -r might help get rid of the reversal at GS21-23?

I'm going to give these options a try this weekend.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 03, 2011, 09:32:43 pm
Mentioned are two options for the colprof command that might be worth trying: -u and -r.
Thanks for the interesting info. I have purposely avoided the Argyll mailing list, assuming it would be way beyond my understanding. But filtered through you, there might be something of interest. Are you able to tell what defaults Coca uses for -u and -r when it calls on Argyll? Can they be changed by me by altering a file, similar to the way I can alter Box Shrink? Do you know whether Coca uses absolute intent? I was hoping the profiled highlights would not be showing any significant errors, but from what you've quoted, highlights on a real slide that are brighter than GS0 may be clipped. I don't want that. Maybe I should be faking GS0 as well.

I read somewhere in a Hutchcolor document that he recommended manually inserting a pure black patch to improve the profile. I'll see if I can find it again.

Quote
Smoothing by -r might help get rid of the reversal at GS21-23?

The only way I'm comfortable with of getting rid of the GS22/23 reversal is by faking GS23 at its target value, or a value can can be demonstrated to be reasonable. Unless my scanner is reading the value of GS23 incorrectly, it seems to me that GS23 has been compromised beyond being useful.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 04, 2011, 12:30:51 am
Just going through all your recent posts to respond.

1. http://www.luminous-landscape.com/forum/index.php?topic=54040.msg445565#msg445565
Re your editing of Ref 18 at gamma 1.0 and gamma 1.8: A difficult slide to digitise accurately. The original is nowhere near as dark as it scans, but because it is underexposed, Kodachrome sent the lower exposures (most of the slide) into darkness – Hunt's 1.5 gamma thing – and the detail is hard to retrieve. The original slide shows obvious detail in the shadow on the tree; the pack is not solid black, it has a definite gradation to dark gray on the top half, and the red of the pack is not mottled with black as in the gamma 1.0 version, but more like the gamma 1.8 version. On a bright lightbox under a loupe this is a very pleasant-looking, moody slide. When projected, it loses a bit; and when scanned and presented on a monitor, it needs a lot of work before it appears as it should.


2.
Quote
Apparently Kodak measured the target with some highly-specialized equipment, as the Q60 data are calculated from spectral measurements. The spectral data is available from Kodak in a QSP file.
Thanks for that suggestion. I already had a copy of the QSP file, but I didn't take much notice of it. I've just had a detailed look. So, the numbers are spectral data! I could sit down and plot some spectrums, like the ones that appear in Hunt's book. I might just do that, for interest.


3.
Quote
Another option is the Silverfast Kodachrome targets.
When I purchased my non-Kodachrome targets from a particular supplier (I probably shouldn't include the name], I asked about Kodachrome. This is the reply:

There are currently two manufacturers offering 35mm Kodachrome targets. Last time I checked none of the targets managed to be within the allowed tolerances of the IT8 standard. However, the fault of the Kodak target was small and shouldn't affect scanning quality. The Kodak target faults were much lower than those of the other manufacturer. Also, the Kodak chart has a decent color gamut unlike the target from the second manufacturer, which should avoid problems, especially with some of the better profilers I know. The rather old age of the reference file I got with my Kodak Kodachrome test charts didn't cause any problem and my measurement of that target was within the expected tolerances of the Kodak reference file. Another impressive example of the great aging perfomance of Kodachrome film. Modern dye films are much worse when it comes to aging.


4.
Quote
I updated the files. Looks better.
diagnostic image 7.7
measurements 7.7
The numbers in the measurement files: I assume the XYZ values come from the IT8 data file, and the RGB values (scaled 0-100) come from the 16-bit scan. Is that correct?

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 04, 2011, 04:36:37 am
Are you able to tell what defaults Coca uses for -u and -r when it calls on Argyll? Can they be changed by me by altering a file, similar to the way I can alter Box Shrink?

I don't think that Coca uses either option, which means no -u extrapolation and the value for -r  defaults to 0.5 (percent). It would be great if CoCa had an editable text file that would allow you to add Argyll options that are not available on the CoCa interface, but I looked and it doesn't.

There is another program like CoCa called Argyll CMS GUI (http://www.digifab.com/ArgyllCMSGUI/). In contrast to CoCa it enables most, if not all Argyll options. There is a Mac version available. This might be the answer to your needs to have access to more of the Argyll command options. I tried the Windows version, but could not get it to work. Maybe you will have better luck with the Mac version.

Quote
Do you know whether Coca uses absolute intent? I was hoping the profiled highlights would not be showing any significant errors, but from what you've quoted, highlights on a real slide that are brighter than GS0 may be clipped. I don't want that. Maybe I should be faking GS0 as well.

The Coca profiles contain the choice of Absolute or Relative intent, to be chosen when you Convert to Profile in the workflow. (Note that if -u were available in Coca, it would result in a profile that allowed only Absolute intent)

The K3 Q60 has a relatively dark DMIN GS0 patch at L* 88. It might be possible to get higher L* on some Kodachrome scans, but I don't know. It depends on how consistent Kodachrome DMIN is from type to type and roll to roll. Have you seen any of your scans blow out the whites? I think that if you keep your scanner exposure setting the same when scanning slides as when you scan the target, you might be ok for the most part. On the other hand, your ref 03 scan seems close to the edge of clipping, so you might want to build a little margin for error into the profile.

Quote

I read somewhere in a Hutchcolor document that he recommended manually inserting a pure black patch to improve the profile. I'll see if I can find it again.

The only way I'm comfortable with of getting rid of the GS22/23 reversal is by faking GS23 at its target value, or a value can can be demonstrated to be reasonable. Unless my scanner is reading the value of GS23 incorrectly, it seems to me that GS23 has been compromised beyond being useful.

The Hutch Color RGB Scanning Guide (http://www.hutchcolor.com/PDF/Scanning_Guide.pdf) - there's a wealth of info there about dealing with scanner flare, modifying patches, etc. I need to read it again.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 04, 2011, 12:51:00 pm
Re your editing of Ref 18 at gamma 1.0 and gamma 1.8: A difficult slide to digitise accurately. The original is nowhere near as dark as it scans, but because it is underexposed, Kodachrome sent the lower exposures (most of the slide) into darkness – Hunt's 1.5 gamma thing – and the detail is hard to retrieve. The original slide shows obvious detail in the shadow on the tree; the pack is not solid black, it has a definite gradation to dark gray on the top half, and the red of the pack is not mottled with black as in the gamma 1.0 version, but more like the gamma 1.8 version. On a bright lightbox under a loupe this is a very pleasant-looking, moody slide. When projected, it loses a bit; and when scanned and presented on a monitor, it needs a lot of work before it appears as it should.

I see what you are saying. I think it boils down to differences in black point and contrast. I feel that editing can make them equivalent. Which is the better starting point for editing? Which is easier to edit?

To me, looking at the individual color channels, the Blue channels look the same, the gamma 1 Green channel looks slightly more detailed, and the gamma 1 Red channel definitely looks less noisy and more detailed. Like I mentioned before, I think the >1 gammas can be noisier. Is the extra red noise visible? Maybe not. I attached a screen cap. of the red channel comparison.

It's a very nice image, and satisfying to see how much quality you have brought out of such a dark scan.

Quote
4. The numbers in the measurement files: I assume the XYZ values come from the IT8 data file, and the RGB values (scaled 0-100) come from the 16-bit scan. Is that correct?

Correct.

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 04, 2011, 07:29:33 pm
The Hutch Color RGB Scanning Guide (http://www.hutchcolor.com/PDF/Scanning_Guide.pdf) - there's a wealth of info there about dealing with scanner flare, modifying patches, etc. I need to read it again.

I tried some of the flare-reducing techniques that Hutcheson describes and am encouraged by the results, but have to play around with it more. It seems that the deep shadows can be made very neutral, which would make them easier to edit, compared to the standard-preparation profiles that almost all go red in the dark tones.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 05, 2011, 08:44:33 pm
The Hutch Color Extended Range (XR) technique has been very useful.

In a nutshell, it amounts to subtracting a constant from all pixel RGB values of the profiling target scan. When your target scan has flare, (that is, more flare than normal scans) the profiler compensates by reducing the response to make the target scan match the Q60 reference. When the profile with this built-in flare compensation is used on an image with less flare, the darker tones can be clipped, making ugly artifacts. Hutcheson's technique is to subtract a constant value from the RGB values of the target scan before feeding it to the profiler. The profiler responds to the reduced RGB values by making a profile that does not reduce or clip the dark tones of a low-flare image.

In practice, if you subtract a lot from the RGBs, the profile can actually introduce a boost in the dark tones. If instead of subtracting, you add a value to all RGBs, the profile will cut or clip more. So the control over the making of the profile is, if you want to boost the dark tones, subtract a value, if you want to lower the dark tones, add a value.

In Hutcheson's XR method, he describes subtracting a single constant integer from all RGB values by using the black point slider of Levels. The black point slider in Levels does not offer much precision - it limits you to subtracting 8-bit values. If your target scan image is 16-bit (as it should be), Levels does not allow you to subtract or add small 16-bit integers. The Exposure tool has an Offset slider that offers the increased precision needed for 16-bit scans.

I found that even better is to adjust each R,G,B channel individually with its own value. Adjusting the R,G,Bs individually allows you to control the tint in the dark tones to achieve neutrality. Unfortunately Exposure does not offer a way to offset the 16-bit RGB channels individually - as far as I know there is no facility in Photoshop that can.

To address this I made a simple PS compatible plug-in RGB Offset 16 (https://sites.google.com/site/cliffpicsmisc/home/plugins/RGBoffset16.8bf?attredirects=0&d=1) that allows adding and subtracting from 16-bit RGBs. Sorry, Windows only. Copy to the Plug-Ins folder. It will appear in the Filter menu under "Color Tools."

My testing so far has been with 16-bit linear files, and with the commercial profile software that I've owned for several years, inCamera by PictoColor (http://www.pictocolor.com/incamera.htm). It is much faster to generate a profile than Argyll, and the profiles are accurate. inCamera responds very predictably to target scans with the added and subtracted offsets. After several iterations I was able to come up with profiles, both for my SS4000 and for the GB Coolscan, that do not clip in the shadows, do not have excessively-elevated black points, and are neutral at the black point. I think they are easy to edit with, compared to the usual Kodachrome profiles which tend to give the dark tones a reddish tint, and have apparently crossed-curves where the red channel crosses over the green and blue channels at some point, resulting in a color balance that differs along the tone scale.

I just started testing the same XR procedure with Argyll/CoCa, but I'm not yet finding the degree of control I have with inCamera. It might be a matter of finding the right profile type: shaper + matrix vs Lab clut vs XYZ, etc. It's time consuming going through the various profile types and testing a range of offsets for each. Not to mention, eventually testing each at various gammas. :(

(But I'm thinking XR might not work well with gammas other than 1.)

Edit: Still seems to be crossed-curve in the red channel. Need to work on it some more...
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 08, 2011, 12:30:51 am
Quote
There is another program like CoCa called Argyll CMS GUI. In contrast to CoCa it enables most, if not all Argyll options.
No good for me. It requires OSX 10.5 running on an Intel Mac. I'm still on OSX 10.4.


Quote
I feel that editing can make them equivalent. Which is the better starting point for editing? Which is easier to edit?
After several days of comparing editing techniques, I'm coming to the conclusion that the overall problem of getting the best result from a scanned slide lies not with the selection of the best profiling technique, but in the editing itself. Most of my reference scans require serious editing, even given perfect profiling, and I'm finding that sometimes I cannot obtain similarly good results if I select a different scan gamma to start with, or if I change the colour space. i.e. if after editing I choose a particular image as the optimum (to be used as a reference), most of the time I cannot reach that optimum with a different scan technique or in a different colour space.


Quote
Like I mentioned before, I think the >1 gammas can be noisier. Is the extra red noise visible?
I've seen lots of noise if I try and bring the shadow detail out, and I'll take your results as definite – that Gamma 1.0 has the least noise, as would be expected. Although, I have found that extra noise in the darkest shadows, as long as it is not obviously coloured, improves the image because the noise appears to be detail within the black. i.e. solid blacks are not desirable in most images. If they are broken up with noise, it can give the impression of detail, even though it is false detail.


Quote
It seems that the deep shadows can be made very neutral, which would make them easier to edit, compared to the standard-preparation profiles that almost all go red in the dark tones.
I have a technique that easily removes the colour cast in deep shadows, but it would be easier to edit if it wasn't there.


Quote
I found that even better is to adjust each R,G,B channel individually with its own value. Adjusting the R,G,Bs individually allows you to control the tint in the dark tones to achieve neutrality.
Do you think that will work for a variety of slide images? The shadows on my slides show a range of colour casts because of the lighting conditions: red (sunsets or bright red shirts), green (forest), blue (sky). They have to edited away for a more natural look, and such editing may swamp the small improvements available by adjusting the profile. It's good to have the most accurate profile, so I'll give your new plugin a go to see the effect.


Quote
When the profile with this built-in flare compensation is used on an image with less flare, the darker tones can be clipped, making ugly artifacts.
I'm seeing some of this clipping, I think. It is difficult to edit away, and if the original slide had a lot of shadow detail, it makes for an unsatisfactory image. I am still finding that editing gamma 1.8 scans sometimes gives the best end-result. Not all the time. Gamma 1.0 comes out on top here and there, and so does sRGB. Which is rather annoying. I was hoping for a single method to give optimum results for all slides.


Clever Kodak?
What are the chances that Kodak purposely designed those flaws into GS18-23 so that when profiled, such a target gave the most pleasing shadow result? Monitors, for example, only have 256 levels. If the end result is to be displayed on a digital screen (in my case, eventually, BluRay via a monitor or projector), there must come a point when you have to say: for the best image when viewed, those extra 16-bit levels have to be compressed in a certain way for optimal results on an 8-bit display. I need convincing that all my attempts at better profiling have not resulted in a degraded 8-bit image when viewed – because blacks on the digital image are now closer to the slide-blacks. Deeper blacks are difficult to reproduce digitally (obtaining rich blacks is the Holy Grail of all digital projectors), and having an excess of them, as does Kodachrome, can only cause problems.

Given a perfect profile applied to a perfect scan of a Kodachrome slide, you will still end up with a very poor digital image because Kodachrome has been optimised for projection, not digitising. Kodachrome scans will always require significant editing because of the way they are, and I'm hoping that such editing doesn't render profiling unnecessary. It's been a good learning experience, but I hope it hasn't all been wasted.

What's happening in my testing (a couple of hours a day for six months), is that I'm moving away from the "science" of scanning Kodachrome, and into the "art" aspect. And that may pose more serious difficulties than any of the science.

P.S. Thanks for the emailed profiles. Will test and get back to you.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 08, 2011, 03:41:41 am
I have a present for you, Cliff: Gamma 1.0 scans of two separate unexposed Kodachrome 64 slides: http://www.mediafire.com/?d38ugsiwkc8a7cm
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 08, 2011, 10:21:34 pm
After several days of comparing editing techniques, I'm coming to the conclusion that the overall problem of getting the best result from a scanned slide lies not with the selection of the best profiling technique, but in the editing itself. Most of my reference scans require serious editing, even given perfect profiling, and I'm finding that sometimes I cannot obtain similarly good results if I select a different scan gamma to start with, or if I change the colour space. i.e. if after editing I choose a particular image as the optimum (to be used as a reference), most of the time I cannot reach that optimum with a different scan technique or in a different colour space.

I agree that the profile only does a little of the work, the editing builds upon the starting point that the profile gives you and is what makes the image.

I think that the different effects of the CoCa/Argyll profiles on the crucial shadow region can be summarized as:

1. The higher gamma profiles give a lower black point. With the lower black point, the higher gamma profiles can sometimes over-emphasize or clip the blacks in slides with heavy shadows. The gamma 1 profiles have higher black points that give a milky and low-contrast appearance to the shadows, but can be corrected with a black point adjustment.

2. The higher gamma profiles of type Lab clut and XYZ have the most neutral shadows. Of the gamma 1 profiles, only the Lab clut profile is neutral. I think the most accurate is the higher gamma ( ~1.8 ) XYZ profile which has both maximum neutrality and the lowest black point. It's not necessary to jump through hoops, modifying the target scan, etc. to get a profile with neutral shadows. The CoCa Lab clut profiles and the higher-gamma XYZ profiles will give you neutrality.

Quote
The shadows on my slides show a range of colour casts because of the lighting conditions: red (sunsets or bright red shirts), green (forest), blue (sky). They have to edited away for a more natural look, and such editing may swamp the small improvements available by adjusting the profile. It's good to have the most accurate profile, so I'll give your new plugin a go to see the effect.

The profile should give a neutral starting point when the slide is exposed in photographic daylight, (D55). In other lighting there will color casts that can be tricky to handle, because of the way the color channels are shifted. (This is true for transparencies in general, not just Kodachrome.) See section 14.17 in Hunt. Fig. 14.11(d) shows what can happen when you attempt to correct the color of the lighting with a white balance adjustment.

Quote
I'm seeing some of this clipping, I think. It is difficult to edit away, and if the original slide had a lot of shadow detail, it makes for an unsatisfactory image. I am still finding that editing gamma 1.8 scans sometimes gives the best end-result. Not all the time. Gamma 1.0 comes out on top here and there, and so does sRGB. Which is rather annoying. I was hoping for a single method to give optimum results for all slides.

If there is clipping, then a gamma 1 Lab clut profile, with it's higher black point, might be better to start with. Then you can drop the black point (Levels, Curves, Exposure/Offset, etc), stopping before clipping occurs?

Quote
Clever Kodak?
What are the chances that Kodak purposely designed those flaws into GS18-23 so that when profiled, such a target gave the most pleasing shadow result? Monitors, for example, only have 256 levels. If the end result is to be displayed on a digital screen (in my case, eventually, BluRay via a monitor or projector), there must come a point when you have to say: for the best image when viewed, those extra 16-bit levels have to be compressed in a certain way for optimal results on an 8-bit display. I need convincing that all my attempts at better profiling have not resulted in a degraded 8-bit image when viewed – because blacks on the digital image are now closer to the slide-blacks. Deeper blacks are difficult to reproduce digitally (obtaining rich blacks is the Holy Grail of all digital projectors), and having an excess of them, as does Kodachrome, can only cause problems.

Maybe Kodak was expecting the target to be drum-scanned so the flaws wouldn't occur? I think the flaws are mainly due to limitations of the desktop scanners we are using.

When it comes time to project your images, are you going to profile your projector? There will probably have to be some gamut mapping and black point compensation. More profiling for you to do - good luck with that!

Quote
Given a perfect profile applied to a perfect scan of a Kodachrome slide, you will still end up with a very poor digital image because Kodachrome has been optimised for projection, not digitising. Kodachrome scans will always require significant editing because of the way they are, and I'm hoping that such editing doesn't render profiling unnecessary. It's been a good learning experience, but I hope it hasn't all been wasted.

Good point, and probably why Kodachrome didn't survive the digital era. I'm wondering if some of the Hutcheson techniques can be used to enhance the profile to do more of the work for Kodachrome, such as neutralizing the whole grayscale and reducing overall contrast.

Quote
What's happening in my testing (a couple of hours a day for six months), is that I'm moving away from the "science" of scanning Kodachrome, and into the "art" aspect. And that may pose more serious difficulties than any of the science.

If you're game, a discussion on editing Kodachrome scans would be great.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 08, 2011, 10:25:25 pm
I have a present for you, Cliff: Gamma 1.0 scans of two separate unexposed Kodachrome 64 slides: http://www.mediafire.com/?d38ugsiwkc8a7cm

The average RGBs of those scans are:
#01 141, 63, 53
#02 141, 62, 52

Lower than DMIN or anything on the Q60 target scans by almost half. Your reference scan 18 has shadow areas this dark or even slightly darker (left sides of the backpack and tree trunk.)

Goes to show how much lower the flare is in actual slides.

Edit: typos
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 10, 2011, 02:32:17 am
If you're game, a discussion on editing Kodachrome scans would be great.
Good idea. I'll start it soon. But this thread has a little distance to run yet.

Thanks for the summary of the various gammas and their effects. I won't comment on them here, but I intend taking note of what you said and do a little further testing on the various gammas and their effect on black point.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 13, 2011, 01:21:27 am
Something is bothering me about most of these profiles.

Going back to the Kodachrome characteristic curves (attached below), a neutral subject should have RGB numbers with Blue highest, Green lower or equal to Blue, and Red lowest. In other words a profiled scan should reproduct the Kodachrome blue cast. Looking at the shadows of Ref 18, for example, the raw scanner RGBs have this reversed, with Red being the strongest. When a profile is then assigned and converted to a working space, it should result in a correction that brings the RGBs more in line with the characteristic curves, changing the raw red cast in the shadows to blue (or possibly neutral).

However, what is happening is that most of the test profiles are not restoring a blue cast to the shadows (although they reproduce the blue cast farther up in the tone range). The exceptions are the CoCa/Argyll gamma 1.8 XYZ profile, the Coca/Argyll gamma 1.0 Lab clut profile, and possibly the inCamera profiles.

The following table shows the 16-bit RGB numbers of a small patch in the deep shadows of Ref. 18, after assigning the various profiles and converting to Prophoto RGB.

My thinking is that the profiles that restore the blue cast to the shadows might be easier to edit, since it should then be possible to align and neutralize the entire gray-scale using simple gamma adjustments. In contrast, an image with a red shadows and blue mid-tones will need more complicated adjustments to fix it.

Gamma 1.8
R
G
B

Gamma 1.0
R
G
B
raw RGB
1437
993
891





Argyll XYZ
77
72
182


1274
1008
867
Argyll Lab
167
60
632


1156
1278
1417
SCARSE.4
1570
1348
1102


1633
1452
1141
LPROF
1399
1010
673


1477
1032
661
inCamera3.1
311
217
316


673
714
652
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 13, 2011, 01:35:39 am
Guy - two more profiles for your collection, made with SCARSE for your Coolscan V:

Kodachrome Gamma 1 SCARSE (https://sites.google.com/site/cliffpicsmisc/home/profiles/KC_GB_G1_Scarse.icm?attredirects=0&d=1)
Kodachrome Gamma 1.8 SCARSE
 (https://sites.google.com/site/cliffpicsmisc/home/profiles/KC_GB_G1.8_Scarse.icm?attredirects=0&d=1)
They seem to be pretty smooth in the darker tones, so might offer an advantage when editing, despite a bit of a red cast in the deep shadows.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 16, 2011, 08:07:52 am
Cliff,

Thanks for the extra profiles. Something more for me to play around with. The red cast in the raw scans of deep shadows is an interesting phenomenon. It could be a part of the scene itself, or it could be a problem with the scanner. How is it that a Kodachrome slide taken in near perfect blackness (my two "black" slides of a previous post) show a significant red cast when scanned? As you stated:

Quote
The average RGBs of those scans are:
#01 141, 63, 53
#02 141, 62, 52

The only explanation must be that the Coolscan is throwing a red cast, which profiling should remove. These dense blacks are a very touchy area.


Removing the non-linearity of the D-H Curve
My next series of tests is to try and remove the non-linear response of the Kodachrome D-H curve, by applying a correction curve in Photoshop's Curves. This would have to be applied after profiling (and the scan would have to be a linear scan for this to work), and before editing. Here I am not trying to correct colour, I am trying to remove the non-linear density characteristic of Kodachrome. This is my thinking:

1. The D (density) axis corresponds to what the scanner sees on the slide. For the red channel of Kodachrome 25, the density range, for example, ranges from about 0.2 to 3.8. These values could be scaled and converted to linear relative-luminance, and would become the horizontal axis (the "input" axis) for Photoshop's Curves, ranging from 0-255.

2. For any value of D on the slide, the actual scene "brightness" for that particular D could be derived from the D-H curve. I am assuming that "brightness" and luminance are directly related to exposure. The problem I have is that there is no upper limit to real-life exposure, whereas there is a limit to Photoshop's 8-bit "brightness" levels: 255. How do you relate the two? My solution is to set the minimum exposure from the D-H curve as being equivalent to Photoshop's brightness of 0, and the maximum exposure from the D-H curve as being equivalent to Photoshop's 255.

3. It is a bit confusing to explain, but when I linearised D and H for the Kodachrome 64 red-channel under the assumptions given above, I came up with the figures listed below [removed 20 June as there were errors in the figures]…

Neglecting the complications of dim and dark surrounds and so on, and assuming that the scanned image will be a faithful reproduction of the scene if the relative luminances between scene and image are linear, then the above figures should allow a curve to be set up in Curves to correct for Kodachrome's non-linear density characteristic.

Can you see any problems with this approach?

Another problem: I could go through all the bother of correcting for the D-H characteristic, only to find at projection time (dark surround) that I should have kept the original Kodachrome D-H characteristic as optimum for dark surrounds – assuming that as far as dark surrounds and optimum gamma are concerned, there is no difference between projecting digitally and projecting by a slide projector.

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: Bart_van_der_Wolf on June 16, 2011, 03:51:43 pm
The only explanation must be that the Coolscan is throwing a red cast, which profiling should remove. These dense blacks are a very touchy area.

Hi Guy,

When you look at the Characteristic Curve for slide films in general, and Kodachrome is no exception, then you'll see that the D-max for R/G/B is different, quite a bit different. Check the specifications for the type of Kodchrome you are looking at. Therefore one should stay away from the D-max for calibration purposes, except for perhaps a clipping level or a smoother TRC adjustment.

Cheers,
Bart
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 17, 2011, 01:29:00 am
The red cast in the raw scans of deep shadows is an interesting phenomenon. It could be a part of the scene itself, or it could be a problem with the scanner. How is it that a Kodachrome slide taken in near perfect blackness (my two "black" slides of a previous post) show a significant red cast when scanned?
-snipped-
The only explanation must be that the Coolscan is throwing a red cast, which profiling should remove. These dense blacks are a very touchy area.

I get similar results with my SprintScan. the red channel for unexposed Kodachrome is much higher than the other channels.

The Argyll Lab and the inCamera profiles for my scanner both reduce the excess red and increases the blue, but to different extents:

        Raw         Argyll Lab    inCamera
K64   53 10 20   0  9  141    6  8  8  
K200  20  2  8    0  4  145    2  2  5

Quote
Removing the non-linearity of the D-H Curve
My next series of tests is to try and remove the non-linear response of the Kodachrome D-H curve, by applying a correction curve in Photoshop's Curves. This would have to be applied after profiling (and the scan would have to be a linear scan for this to work), and before editing. Here I am not trying to correct colour, I am trying to remove the non-linear density characteristic of Kodachrome. This is my thinking:

1. The D (density) axis corresponds to what the scanner sees on the slide. For the red channel of Kodachrome 25, the density range, for example, ranges from about 0.2 to 3.8. These values could be scaled and converted to linear relative-luminance, and would become the horizontal axis (the "input" axis) for Photoshop's Curves, ranging from 0-255.

Did you convert from Density to Transmittance by the formula Transmittance = 10^(-Density), then scale Tranmittance by the maximum RGB value? Likewise convert LogE to linear Exposure? The curves will have a completely different shape in a linear-linear plot compared to the log-log characteristic curve.

Quote
2. For any value of D on the slide, the actual scene "brightness" for that particular D could be derived from the D-H curve. I am assuming that "brightness" and luminance are directly related to exposure. The problem I have is that there is no upper limit to real-life exposure, whereas there is a limit to Photoshop's 8-bit "brightness" levels: 255. How do you relate the two? My solution is to set the minimum exposure from the D-H curve as being equivalent to Photoshop's brightness of 0, and the maximum exposure from the D-H curve as being equivalent to Photoshop's 255.

I think you can just scale Transmittance x 255.

Quote
Neglecting the complications of dim and dark surrounds and so on, and assuming that the scanned image will be a faithful reproduction of the scene if the relative luminances between scene and image are linear, then the above figures should allow a curve to be set up in Curves to correct for Kodachrome's non-linear density characteristic.

Can you see any problems with this approach?

Another problem: I could go through all the bother of correcting for the D-H characteristic, only to find at projection time (dark surround) that I should have kept the original Kodachrome D-H characteristic as optimum for dark surrounds – assuming that as far as dark surrounds and optimum gamma are concerned, there is no difference between projecting digitally and projecting by a slide projector.

Now I see why you want to do this - in order to have the option to make your own compensation for surround, etc. specific to your projector and viewing conditions. I would think that a global gamma adjustment would do the trick, while preserving Kodachrome's toe and shoulder, which might be worth preserving.

If you're going to remove Kodachrome's non-linearity, you might as well do it for each channel, in order to get rid of the blue cast at the same time.

Here's a text file of the K25 curves (https://sites.google.com/site/cliffpicsmisc/home/guyburns/k25curves.txt?attredirects=0&d=1) digitized from a graph in a Kodak publication. The columns are LogE, Red Density, Green Density, and Blue Density. LogE is in 0.05 density increments. It should save you a lot of trouble reading numbers off the graph.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 17, 2011, 08:35:43 am
Quote
Did you convert from Density to Transmittance by the formula Transmittance = 10^(-Density), then scale Tranmittance by the maximum RGB value? Likewise convert LogE to linear Exposure? The curves will have a completely different shape in a linear-linear plot compared to the log-log characteristic curve.

Guy, I think you have done this. Sorry!

Does Curves have enough resolution to linearize the dark end of the tone scale?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 18, 2011, 10:00:07 pm
Does Curves have enough resolution to linearize the dark end of the tone scale?
Not if you want to do it accurately – but it may not need to be done accurately. No point worrying too much about that aspect until I prove that the idea works.

If the idea works and I need to generate accurate Curves in Photoshop, would you be able to write a script that accepts Curves "input" and "output" values and passes those values to Curves so that it can draw smooth curves through the values? Could a polynomial equation be sent to Curves?
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 19, 2011, 12:34:27 pm
If the idea works and I need to generate accurate Curves in Photoshop, would you be able to write a script that accepts Curves "input" and "output" values and passes those values to Curves so that it can draw smooth curves through the values? Could a polynomial equation be sent to Curves?

I read that Curves uses cubic splines to draw between points. Curves gives you this automatically, unless you are using the Pencil.

Have you tried using the Pencil in Curves? You can set 256 points individually, and have the option to apply smoothing. With the Pencil, the curve will be saved as an .amp (arbitrary map) file.

Alternatively, you can create a Photoshop amp format file and read it in to Curves. Amp files are very simple, a list of the 256 output numbers, but to make one you will probably have to use a hex editor, unless someone knows of an easier way to make a binary file. It will be possible to make a plugin to do it, if necessary.

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 19, 2011, 10:07:25 pm
I've given up on the idea of correcting the D-H curves. This is the second time I've tried it (my first attempt was a month ago) and both times no improvement came. Both times I tried two methods: an absolute approach where I tried to correct the D-H curves point-for-point; and a relative approach where I left the blue channel unchanged and applied alterations only to the green and red. I think the absolute approach failed because I didn't know how to handle the end points – for example, the D-H curves show a minimum density of ~0.21 (0.62 relative luminance), yet my scanned slides go much lighter than that. And if you don't know how the end points of the original luminance of the scene translate to density on the slide, I think you're stuck. I was more certain of getting useful results from the relative method, but the end points were again a problem.

Below are the Input/Output figures I tried to put into Curves. The first column is the input, the second is Green output, and the third is Red output. Because PS doesn't allow an input of less than 4 into Curves, I combined the first four settings into one for green and red: 4,7, then rounded the others. Note how the differences disappear at Input = 157 (equal to density of 0.21, the lightest shown on the D-H curves). So the corrections only apply to half the brightness range of the slide. Didn't seem right when I calculated the figures, and it certainly didn't remove the blue cast.

In      G      R
0.1   0.3   0.5
0.2   0.3   0.4
0.4   0.5   0.6
1.5   1.8   2.3
6.0   7.1   9.0
24.4   27.6   34.2
102   102   125
157   157   157

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 21, 2011, 01:24:52 am
– for example, the D-H curves show a minimum density of ~0.21 (0.62 relative luminance), yet my scanned slides go much lighter than that.

You shouldn't get lighter than dmin if you Convert-to-Profile from the scanner profile using Absolute. Are you using the same exposure that you used when scanning the profile target?

Quote
Below are the Input/Output figures I tried to put into Curves. The first column is the input, the second is Green output, and the third is Red output. Because PS doesn't allow an input of less than 4 into Curves, I combined the first four settings into one for green and red: 4,7, then rounded the others. Note how the differences disappear at Input = 157 (equal to density of 0.21, the lightest shown on the D-H curves). So the corrections only apply to half the brightness range of the slide. Didn't seem right when I calculated the figures, and it certainly didn't remove the blue cast.

I'm going to give it a try. Maybe there is a way to do it without using Curves.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 26, 2011, 01:14:32 pm
Guy, are you still with me?

After some difficulty trying to invert them, it seemed worthwhile to check if the profiles are giving us anything that resembles the published characteristic curves. On my last couple of rolls of K64 I shot a few frames with a ColorChecker and a Kodak Q-13 gray-scale. This let me generate some plots after applying the various profiles. The procedure was to assign the profile, then convert to a modified gamma 1.0 Prophoto RGB working space with Absolute intent. The plots are displayed in a way that allows comparison to the published Kodachrome characteristic curves that were recently posted.


(https://sites.google.com/site/cliffpicsmisc/home/profiles/KC-Curves_Compare_Profiles_LogLog.png?attredirects=0&d=1)

The first plot is the raw scan without profile.

The second plot shows the gray-scale after applying the Argyll Lab CLUT profile, which is the "best behaved" of all the Kodachrome profiles I have tested for my Polaroid SS4000 scanner.

The third plot shows a result that is typical for most of the other profiles. Note how the red curve crosses over red and blue, giving a changing color-cast along the tonal range.

The big surprise is the fourth plot. For the heck of it, I wanted to see how bad it would be to use an Ektachrome profile. It's better than any of the Kodachrome profiles! Particularly in the upper-left part of the curves, which are the highest densities. This profile makes it very easy to edit, needing only white and black point setting and some gentle curves (and/or gamma adjustments) to achieve a neutral gray-scale along the whole range of tones.

The problem with using the Ektachrome profile is that, although the gray tones can easily be made neutral, the non-gray colors can be inaccurate. The solution would be to make a Kodachrome profile from an Ektachrome profile, a "Pseudo-Chrome" profile using the method described by Hutcheson (http://www.hutchcolor.com/PDF/Kodachrome_profiles.pdf).
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on June 27, 2011, 12:25:09 am
Cliff, thanks for the four curves. I'll play around with Ektachrome and the Hutch method to see what comes of it. And maybe I should choose Lab clut as my preferred profile, instead of XYZ (see Knockout Rounds, below).

You shouldn't get lighter than dmin if you Convert-to-Profile from the scanner profile using Absolute. Are you using the same exposure that you used when scanning the profile target?

For a Gamma 1.0 scan (see Ref 03 in the clouds and snow), 16-bit, raw RGB values are R>28,000 and G&B > 30,000, well above what the D-H curves indicate I should be getting as the brightest scan from Kodachrome. And when the profile is applied, those values increase. I'm not sure anything sensible can be gained by a person of my limited colour knowledge trying to correct a scan by using the Kodachrome D-H curves, because I don't know how those curves relate to an actual scan.


Quote
Guy, are you still with me?
Yes, still here playing around, though I can see an end in sight. At some stage I have to move away from testing and start actual scanning. I've been generating profiles for Agfa, Velvia and Ektachrome and testing them based on the Kodachrome results. A well-exposed Velvia RVP50 slide is so easy to scan and edit compared to Kodachrome. In some cases (see Gil 06: http://www.mediafire.com/?o1y5c7cpstedj3n, one of several Velvia slides I've borrowed from a photographer mate), the unedited profiled scan is superior to my attempt at editing it. I've never come across that when editing Kodachrome, which seems to always need editing.


Preliminary Overall Results
Targets tested: Kodachrome, Ektachrome, Fuji (Vevlia, Sensia, Provia, Astia), Agfa. I scanned all my IT8 targets at G1.8 with the Coolscan V ED scanner, then made a second "corrected" copy of each target by averaging 40% of certain GS patches and applying that to the whole patch. For Kodachrome I altered GS15-GS23. Alterations for the other films varied, depending on how much flare from the surrounds was present. All films except Kodachrome showed an increase in density from GS22 to GS23; only Kodachrome showed a slight reversal. Because of this reversal, for the corrected version of Kodachrome I replaced GS23 with the colour value of unexposed Kodachrome (Lab 56, 21, 12); for the uncorrected version, I replaced GS23 with a copy of GS22 (so that the "uncorrected" scan wasn't corrected very much).

Knockout Rounds
All S+M profiles when applied to the target showed colour changes in certain colour patches compared to Lab and XYZ (which appeared identical). So S+M was knocked at round 1. For both XYZ and Lab, the difference between "Uncorrected" and "Corrected" was minimal, in most cases undetectable, the only difference being a lightening of the darkest GS patches. Because the "corrected" versions should theoretically give better profiles, and because the differences between "uncorrected" and "corrected" were minimal, the "uncorrected" versions were knocked out in round 2.

That left the "corrected" versions of XYZ and Lab to play off in the final. The difference came down in XYZ's favour because of the way it retained the contrast in certain grainy patches (typically the patches GS17-19) i.e. the XYZ profile kept the grain intact whereas Lab smoothed out the grain. Originally I choose Lab because of this, but after further thought I realised the Lab had the lower contrast in the darkest regions (thus smoothing the grain), so I opted for XYZ as the best.


Summary
The best profile from the Nikon Coolscan was obtained by:

1. Using a gamma of 1.8 while scanning at 4000 dpi;

2. Resampling in PS to 2000 to reduce file size for archiving;

3. Correcting certain GS patches to reduce the problem of flare, and in the case of Kodachrome alone, to remove the density reversal between GS22 and 23.

4. Resampling to 1000 for input into Coca for profile generation. I did not alter the Box Shrink parameter from the default 3.5 to my theoretical best of 7.5, because the patch correction of step 3 would have minimized the effect of flare.

5. Editing will be in the IT8 profile space with 2-4 Curve layers applied. There are several reasons for not converting to a wider gamut space. I have arrived at this tentative decision after a few hundred test edits, but the reasons are not yet final:

(a) Testing seems to indicate that editing in a wider-gamut space makes editing more difficult. I'm not convinced that this is a real phenomenon (i.e. a change in editing procedure might fix the problem), and until I work out why this might be the case, this finding is open to change.

(b) Editing in the IT8 profiled space by applying Curve layers is non destructive. Converting to another profile alters the colour numbers and the process can't be exactly reversed. By staying in the IT8 profile space and editing only by Adjustment Layers, the colour numbers are always only one step removed (the gamma 1.8 step) from what the scanner sees on the slide. This is a significant space saving when archiving, because I won't have to archive the original scan as it is non-destructively incorporated in the edited scan.

(c) All my scans are destined for Rec 709 output (effectively sRGB) on a digital projector. I don't require a wide gamut.


Additional Tests
1. One of my long-standing photographer mates wants to learn how to scan his Kodachrome slides, so he sent me some slides to play around with and comment on. I asked for "difficult" sides, and he complied. Check out my thoughts at: http://www.mediafire.com/?ymh90cvds5c3w2j

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 27, 2011, 09:40:54 am
Cliff, thanks for the four curves. I'll play around with Ektachrome and the Hutch method to see what comes of it.

The Ektachrome profile might be the solution to your KC shadow issues. Your Velvia profile might work, too. In the Ektachrome log plot, above, look at how the three channels track nearly perfectly right down to DMAX. Even the wayward red channel has been brought into line. I haven't tried an Ektachrome profile with gamma, yet, since I have only, ever made linear scans with my scanner.

Quote
And maybe I should choose Lab clut as my preferred profile, instead of XYZ (see Knockout Rounds, below).

If not your Ektachrome profile. I was using Lab clut only because XYZ does not work as well for linear scans. For gamma >1 scans, I think XYZ is usually better.

Quote
For a Gamma 1.0 scan (see Ref 03 in the clouds and snow), 16-bit, raw RGB values are R>28,000 and G&B > 30,000, well above what the D-H curves indicate I should be getting as the brightest scan from Kodachrome. And when the profile is applied, those values increase. I'm not sure anything sensible can be gained by a person of my limited colour knowledge trying to correct a scan by using the Kodachrome D-H curves, because I don't know how those curves relate to an actual scan.

Hmm, my scanner also gives RGB values for DMIN in the same range as yours. I think it's just the scanners giving a little extra exposure. The effect would be to shift the characteristic curves down, so that DMIN is closer to zero density, an equal subtraction of density from the three channels.

All transparency films seem to have a DMIN in the same ballpark. (For that matter, so do negative films.) In that case it would make sense for scanner manufacturers to calibrate so that the maximum RGB numbers are assigned to a density closer to film DMIN, instead of to density = 0 (no film in the scanner).

All that matters is that the highest RGB values in a scan do not exceed the values for DMIN in the Q60 profiling target. If they do exceed DMIN, then the profile can be adjusted by one of the Hutcheson techniques.

If you Convert to Profile using Absolute Intent, the lightest white on the slide will be L* = 88, which is how DMIN is defined for the Q60 target. It gets darker, not lighter. Then just adjust lightness to a preferable level while editing.

Quote
Preliminary Overall Results
Targets tested: Kodachrome, Ektachrome, Fuji (Vevlia, Sensia, Provia, Astia), Agfa. I scanned all my IT8 targets at G1.8 with the Coolscan V ED scanner, then made a second "corrected" copy of each target by averaging 40% of certain GS patches and applying that to the whole patch. For Kodachrome I altered GS15-GS23. Alterations for the other films varied, depending on how much flare from the surrounds was present. All films except Kodachrome showed an increase in density from GS22 to GS23; only Kodachrome showed a slight reversal. Because of this reversal, for the corrected version of Kodachrome I replaced GS23 with the colour value of unexposed Kodachrome (Lab 56, 21, 12); for the uncorrected version, I replaced GS23 with a copy of GS22 (so that the "uncorrected" scan wasn't corrected very much).

Did you see a visible improvement with the corrected Kodachrome?

Quote
Knockout Rounds
All S+M profiles when applied to the target showed colour changes in certain colour patches compared to Lab and XYZ (which appeared identical). So S+M was knocked at round 1. For both XYZ and Lab, the difference between "Uncorrected" and "Corrected" was minimal, in most cases undetectable, the only difference being a lightening of the darkest GS patches. Because the "corrected" versions should theoretically give better profiles, and because the differences between "uncorrected" and "corrected" were minimal, the "uncorrected" versions were knocked out in round 2.

That left the "corrected" versions of XYZ and Lab to play off in the final. The difference came down in XYZ's favour because of the way it retained the contrast in certain grainy patches (typically the patches GS17-19) i.e. the XYZ profile kept the grain intact whereas Lab smoothed out the grain. Originally I choose Lab because of this, but after further thought I realised the Lab had the lower contrast in the darkest regions (thus smoothing the grain), so I opted for XYZ as the best.

Matches my results, exactly.

Quote
5. Editing will be in the IT8 profile space with 2-4 Curve layers applied. There are several reasons for not converting to a wider gamut space. I have arrived at this tentative decision after a few hundred test edits, but the reasons are not yet final:

(a) Testing seems to indicate that editing in a wider-gamut space makes editing more difficult. I'm not convinced that this is a real phenomenon (i.e. a change in editing procedure might fix the problem), and until I work out why this might be the case, this finding is open to change.

(b) Editing in the IT8 profiled space by applying Curve layers is non destructive. Converting to another profile alters the colour numbers and the process can't be exactly reversed. By staying in the IT8 profile space and editing only by Adjustment Layers, the colour numbers are always only one step removed (the gamma 1.8 step) from what the scanner sees on the slide. This is a significant space saving when archiving, because I won't have to archive the original scan as it is non-destructively incorporated in the edited scan.

(c) All my scans are destined for Rec 709 output (effectively sRGB) on a digital projector. I don't require a wide gamut.

Earlier in the thread I showed that sRGB and Adobe RGB can't contain all of the colors of the Kodachrome Q60 target. But chances are your images don't contain colors as saturated as those on the Q60, so you might not have to worry about clipping colors in those smaller color spaces.

Although you don't necessarily have to convert to a wide-gamut space, I think it's important to convert to some space. Otherwise, by not converting at the beginning and editing in the profile space, all your edits to the image are, in effect, modifying the scan before applying the profile, and the scan will no longer match the conditions under which the profile was made. The edits invalidate the profile. You are also giving up the advantages of a standard working space such as knowing that R=G=B is a neutral color.

That's the theory - but if it's working for you, why not?

Quote
Additional Tests
1. One of my long-standing photographer mates wants to learn how to scan his Kodachrome slides, so he sent me some slides to play around with and comment on. I asked for "difficult" sides, and he complied. Check out my thoughts at: http://www.mediafire.com/?ymh90cvds5c3w2j

I will check them out! :)

Try using an Ektachrome profile with your Kodachrome scans and let me know what you think.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: crames on June 27, 2011, 11:49:09 pm
Additional Tests
1. One of my long-standing photographer mates wants to learn how to scan his Kodachrome slides, so he sent me some slides to play around with and comment on. I asked for "difficult" sides, and he complied. Check out my thoughts at: http://www.mediafire.com/?ymh90cvds5c3w2j

Well done, Guy. That's a great article!
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: SergeyT on August 12, 2011, 10:28:39 pm
I think you would have had a better success with making a scanner profile vs. a "specific film" profile.

If shadow details was your concern why not to try making a *scanners* profile based on the HCT target instead? It has the grays located far away from the saturated colors and most importantly the darkest patch far way from any white. So that would overcome your scanners flare problem and solve the dark tones issue as well. I find using the colprof with "-ax" producing the best profiles.

Profile evaluation should be done by comparing appearance of a scanned slide( with the color profile applied) to the slide's appearance on a light-table with a color corrected light temp.

If they look alike then the job with profiling is done well.
It should not matter if the scan looks bluish on the screen for as long as it looks the same (bluish) on the light table.
All required color correction should be done to the scanned image after its conversion into a working RGB (ProPhotoRGB).

Hope this helps,
SergeyT.
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on August 13, 2011, 10:02:17 am
I thought this thread was long buried. Well, at least I was hoping it was.

Successful profiling assumes a good quality scanner and an accurate target. The HCT target you mention may well be superior to the other targets, but if the scanner is sus, what's the point? And as reluctant as I am to say this, the Coolscans have problems (and I'll include the 5000 and 9000 here, in addition to my V ED, although that may change once I see the test scans from the upmarket models).

You ever had to change your mind about something that you had always assumed? It can be a long, slow process to come to a different viewpoint. I'm that way with my two Coolscan V EDs. Give me another month or so, and I'll probably come around to accepting that a flatbed scanner – to me the idea used to be anathema – that a flatbed without profiling (the Epson V700), can give as pleasing an image as a dedicated slide scanner with profiling. Often the image is more pleasing. I still shake my head about that.

The Science & (Black) Art of Scanning Kodachrome is going to be a good read when I finish it.

Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: WombatHorror on August 18, 2011, 02:34:54 am
I thought this thread was long buried. Well, at least I was hoping it was.

Successful profiling assumes a good quality scanner and an accurate target. The HCT target you mention may well be superior to the other targets, but if the scanner is sus, what's the point? And as reluctant as I am to say this, the Coolscans have problems (and I'll include the 5000 and 9000 here, in addition to my V ED, although that may change once I see the test scans from the upmarket models).

You ever had to change your mind about something that you had always assumed? It can be a long, slow process to come to a different viewpoint. I'm that way with my two Coolscan V EDs. Give me another month or so, and I'll probably come around to accepting that a flatbed scanner – to me the idea used to be anathema – that a flatbed without profiling (the Epson V700), can give as pleasing an image as a dedicated slide scanner with profiling. Often the image is more pleasing. I still shake my head about that.

The Science & (Black) Art of Scanning Kodachrome is going to be a good read when I finish it.



Are you sure? I'm getting results that seem decent enough to me with a Coolscan 9000 and scanning K64 (using Silverfast calibrated with a K64 IT8 slide).
I should say that the 9000 is noticeably better IMO than the V at K^4. The type of semi-diffuse lighting it uses doesn't overdue the grain, bubbles, scratches quite as badly as the more direct lighting in the V and it doesn't have the halation flaring when you get say really bright sky right next to a dark tree line and it seems to be able to dig a bit deeper into the shadow detail. Although I'm probably not being as picky as to getting everything 100% exact from exact tone curve to every last shade without needing to adjust a single thing.

actually these are from some early attempts so they may not be the best samples but I don't have much scanned stuff online yet and I hadn't yet figured out my current sharpening/detail settings yet:
(http://skibum4.smugmug.com/Travel/Africa/Egypt/R3P2EgyptICEps/984847467_KC9SY-X2.jpg)
(http://skibum4.smugmug.com/Travel/Africa/Egypt/R3P17EgyptICEs/984858856_QJdSC-X2.jpg)
(http://skibum4.smugmug.com/Travel/Africa/Egypt/R3P12EgyptICEs/984855156_CCwRR-X2.jpg)
(http://www.smugmug.com/photos/992335563_ar2Gu-X3.jpg)
(http://www.smugmug.com/photos/992335679_XnDAz-X3.jpg)
(http://www.smugmug.com/photos/992335858_ZEABE-X3.jpg)
Title: Re: Generating a Kodachrome profile from an IT8 target
Post by: guyburns on August 18, 2011, 08:50:27 am
Hi Larry. Thanks for the comments and the scanned images.

Quote
Are you sure? I'm getting results that seem decent enough to me with a Coolscan 9000 and scanning K64 (using Silverfast calibrated with a K64 IT8 slide).
I'm as certain as I can be that, taken overall (sharpness, contrast, colour, lack of flare, and general "feel"), that the Epson flatbed unprofiled gives results that are as good as a profiled Coolscan V ED in most cases. Sometimes better, sometimes inferior. I have come to that conclusion after detailed side-by-side comparison, results of which I will make available when my testing is finished.

This photographer (http://www.gerardhorsman.com/) seems to think the Epson is pretty good. That's what he uses to scan his 6x17 Velvia images. He's just about to hold an exhibition in Hobart showcasing his 2 metre-wide images. He lives close by, and I checked out his process a few weeks ago. The fellow I went with was so impressed, that within a few days he went out and bought a top-of-the-line 27" iMac and the Epson scanner (same as Gerard's setup). Then he gave them to me to set them up for him. It's great to have mates who let you have $3000 of new equipment for a fortnight. That's when I began testing the Epson. I'm not spruiking any particular scanner though. I simply want to find out which one does the best job across a variety of slides.

The slides I have chosen as reference slides are challenging to scan (that's why I chose them). The scans you have posted appear to me to not present too many challenges, and because of that I'd say that most scanners would do a pretty good job on them. My aim is different. If an idea I have comes to fruition, I'll be travelling all over the place scanning slides of a landscape now submerged, and those slides may be Kodachrome, Agfachrome, Anscochrome, Ektachrome in all sorts of conditions. And I may only get one shot at them. Before I embark on this project, I have to be confident that no matter what type of slide a person hands me (overexposed, underexposed, covered in dust, seriously colour-faded), that I know I will come away with the best possible scan given the limitations of my equipment (whatever that turns out to be). Kodachrome, as such, is just one part of my testing.

My theory is: if I can scan challenging slides, the ordinary slides will be easy.

Five of my test slides are in Canada now, to be scanned on a 5000. Then when I get them back, they're off to Germany for the once over on a 9000. If the latter doesn't come off, are you willing to scan them for me? You can gmail me at gdburns.