Pages: [1]   Go Down

Author Topic: Scanning Colour Negative Film - Device profiling?  (Read 1827 times)

msantana

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
    • Portfolio
Scanning Colour Negative Film - Device profiling?
« on: April 27, 2021, 01:30:04 am »

Hi all! My first post here. This ended up being way longer than intended but I struggled to boil it down as there’s so much to set up before my questions at the end. Hope that’s okay!

First off, it’s a little daunting for me to post here as I come from a background of motion picture post-production (by day I’m a VFX compositor) so my understanding and implementation of colour management and colourspace conversion workflows (at least on a professional level) takes place in node-based software like Nuke and Resolve, which does things slightly differently to PS. Also, my specific job doesn’t involve input device profiling - so please excuse any gaffes and/or concepts in this regard that might not quite translate or be worded in the same way, or where my lack of expertise in this regard will show! I’m not trying to pass off anything to do with this as fact - just sharing my findings in order to ask for professional advice where things get a little murky.

I’ve seen some recent threads online about this topic that prove just how slippery things can get when it comes to specific language 😬

With all that said, something has been playing on my mind lately regarding input device profiling with a translucent it8 target (be it for traditional scanners or camera scans) specifically when it comes to scanning colour negatives. And note - I’m not talking about profiling the neg stock being scanned; but the capture device + light source. In other words, the ability to capture accurate scene data of the negative as a positive transparency before inversion work can begin, purely to correct device errors in colour rendition.

I must’ve read and seen just about every article, technical paper, blog or forum post on the topic I could find dating back to the early 00’s, and can see very logical arguments made for both profiling and not profiling devices when it comes to CN. Most of the things I read don’t go into a ton of detail and others are so old that I wonder if the industry has maybe adopted new/more accurate workflows since.

In the NO PROFILING camp; several people seem to suggest that due to the orange mask and how narrow a space the positive neg colours occupy in that target space, any profiling effort is pretty meaningless and can cause more harm than good in certain cases. Especially with anything other than a matrix-only profile - since a nonlinear curve bending ever so slightly wrong through the neg’s range could be catastrophic once inverted and scaled into the positive image. I tend to agree with this logic and based on my experiments find it to mostly be the case. But this is very anecdotal and I am very open to this being the wrong conclusion!

The downsides to No Profiling are that there’s no real guaranteed consistency across scanners. Differences in device sensitivity will yield wildly different results, and the same neg will look a million different ways on a million different scanners. This tracks with what I’ve seen at work when it comes to even very very expensive projects - scans of the same negative reels by different A-list motion picture film labs can come back looking so different that they may as well be different film stocks altogether. But again, anecdotal.

In the YES PROFILING camp, people say that it would be silly not to use corrections for the native defects/offsets in the individual’s scanner’s reproduction, which when left unchecked will only make correction of the negative harder by compounding one problem on top of an already very complex operation. Downsides as I understand would be what I mentioned above; one microscopic wrong assumption in the correction profile and it could be game over anyway, since the profiling data must be so much sparser for the colours a neg can represent in its positive/orange state. It’s not as simple as profiling for normal scenes or even slides, since those images are much better spread across the it8 data, and likely to inherit the profile more successfully.


So maybe allow me to share where I’m at with my workflow, with which I’ve had most success to date (I must have tried them all by now). I still have concerns that it gets a little hand-wavy, but overall very happy with the results. And if anyone spots problems with what I’m doing, I’d appreciate a hand!

1. Scan using either the scanner’s software with colour management disabled (eg. Epson scan, which generates a Gamma 2.2 Tiff in what I assume is the device’s own RGB space) or Vuescan’s Raw linear TIFF with ‘Device RGB’ set as the colourspace.

2. In my image processing software, apply a Gamma conversion from 2.2 to 1 in the Epson case, or nothing in the already linear Vuescan file.

3. Execute the neg inversion maths, whatever that might be. I have a method I developed in Nuke that I’m testing on PS too, based on maths from the well documented kodak/cineon system and Arri papers. It inverts using a log curve + specific gamma values & parameters. Long story short I feed the algorithm a linear scan and receive a roughly scene-linear positive, on which I can apply RGB gains to white balance. I validated this linear response by comparing with linearised digital exposure brackets of the same test scene I shot side by side with some film rolls - and it’s pretty much a fit for for most of the dynamic range. So let’s assume that’s working and valid for now!

4. Take my scene-linear film image to anywhere between gamma 1.8-2.2 for display purposes and apply a shoulder compression curve to round off the highs. Now I have a correctly white balanced image that was produced using scene-referred tools. And depending on the device/scanner, it could be lacking saturation. I’m assuming (and please stop me here if this is wrong) this is because even though it’s now a positive image it still behaves as though it’s in the the scanner’s wider RGB space, because I believe I did not break scene-linearity. And this is where it gets pretty hand-wavy:

5. Simply add chroma to taste via LCH or HSL... or -

6. Assign a profile that makes the image look good. Depending on the device it could be srgb, it could be Adobe RGB, some J Holmes profiles I like, or it could even be the feared it8 calibration profile. Which has sometimes worked okay for me in fairness... I am again assuming (!) this is because of the technically well-inverted process that doesn’t break linearity up to now, so the matrix could still be valid to reach the intended primaries for the scanner. Does this even make ‘colour management sense’? Whether or not it does, it produces good results 😅. Note I’m only mentioning assigning a profile this late on - but it could be done at any stage. But can only be judged once the underlying base inversion/correction has been done.

7. After the neutral base inversion and ‘finding’ the right profile to assign, can then convert into a bigger working space if needed (or not) for further creative work before final conversion to srgb/web.

What I don’t like about this, is the slight guesswork when it comes to which profile to tag the scanner RGB as...  But I find it hard to argue with the results I get this way. What I do like, is that I’m working in the scanner’s linear RGB data/space for the inversion instead of some arbitrary converted-into working space, and no matter what profile I apply or chroma treatment I add once we’re out of linear land, the base inversion & white balance is always done and always looks correct underneath (albeit desaturated due to its wide-ish colourspace).

The alternate workflow is:

1. Either assign the device RGB or the it8 calibration profile.
2. Convert to a working space (Adobe, Prophoto, Ektaspace, ACEScg etc)
3. Invert using x method, e.g linear workflow as mine above.

What I really dislike about this method is that depending on which working space you convert into before inverting, you get wildly different results. Overall I find a space just big enough to hold all the film’s colours (but not much bigger) works well. Like Dcam3 from J Holmes or his free Ekta Space, and sometimes AdobeRGB. The huge spaces just produce way too much colour contrast and hue shifts. Overall feels incredibly arbitrary to pick one of these before the inversion as the working space. Which also now has to be done from scratch per colourspace ‘trial’ since the original colours get changed due to the conversion.

My questions (sorry it took so long!)

- Is there anything monumentally stupid about my first pipeline? Any suggestions to decrease the guesswork? It works for me, but sadly each device seems to need its own profile tag and/or chroma recipe... but there usually is at least one. Doesn’t seem very colour-science-y to fish for profiles, but then again you’d be surprised by the not very colour-science-y things we sometimes do to get things into cinemas and no one complains. 🙃

- Does it make sense for the it8 calibration matrix profile to somehow be valid when used in this way, or is it luck/coincidence when it does? I’m still very suspicious, but since it’s not touching the data underneath... if it looks good then it is good.

- How do commercial labs deal with this for their devices & consistency between scanners for negs? I.e. do they assign an input profile and work in device native like my workflow, or do they convert from device to working space up front. And if so, which working space is usually the norm if such a norm is even possible?

Apologies for the long post, and for any stupidity I have subjected you folks to.

Cheers!

Miguel
« Last Edit: April 27, 2021, 01:50:32 am by msantana »
Logged
Miguel Santana da Silva / Digital Compositor @ ILM London

Stephen Ray

  • Full Member
  • ***
  • Offline Offline
  • Posts: 217
Re: Scanning Colour Negative Film - Device profiling?
« Reply #1 on: April 27, 2021, 05:22:19 am »

 How do commercial labs deal with this for their devices & consistency between scanners for negs?

They simply use the default machine intelligence and / or operator decisions of scene interpretation. Maybe research the likes of Fuji Frontier and Noritsu photography mini lab technology which might correlate with Cineon.
Logged

msantana

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
    • Portfolio
Re: Scanning Colour Negative Film - Device profiling?
« Reply #2 on: April 27, 2021, 07:18:58 am »

They simply use the default machine intelligence and / or operator decisions of scene interpretation. Maybe research the likes of Fuji Frontier and Noritsu photography mini lab technology which might correlate with Cineon.

Thanks Stephen! This seems to track with my feeling that when it comes to colour negs it is largely a subjective/manual thing. I had a little look for relevant documentation on how Frontier and Noritsu scanners deal with internal colour management/device profiles specifically for colour neg inversion, but not having a ton of luck. I'll keep looking.

Another thing worth mentioning is that I find Nikon Scan + Coolscan devices to produce really natural looking colour rendition for colour negatives. There are a lot of photographers that feel the same but also don't love the fact that it's a bit of a closed system without much customisation when the auto-invert algorithms don't quite work or crush highlights/shadows... which is often!

Interestingly my algorithm seems to produce really similar results to that of the Nikon scan software, with the benefit of being completely 'live' in a 32 bit floating point working environment (in Nuke, anyway). I suspect Nikon software uses similar math, which is also similar to the published CF Systems' (makers of Colorperfect plugin) theory on negative to positive conversion.



None of these match 100% but without A/B comparing I'd say they're relatively close for something out of the box. With a bigger difference on the Epson v550 scan which doesn't seem to pick up as much colour gradation nuance and tend a little green on skin hues... but it is a much cheaper device. Notice also how on the Coolscan 9000 inversion I tagged sRGB to the device RGB, but on the Epson the most correct-looking result was by tagging it as Adobe RGB. So both inversions were done on each device's 'native' linear RGB data underneath the 'guesstimated' profile tag. Maybe the most elegant solution would be to implement an ACR style raw calibration tab that allows for dialing in hue and saturation of primaries by eye through a 3x3 matrix in linear space after inversion, instead of trying to find a fitting off-the-shelf colourspace for each roll/device?

P.S. This is if I apply the transformation to the whole film strip of the Epson scan. The only adjustment I made was a linear exposure on the indoor shots through a mask for those frames, some of which were really underexposed. The hand one especially was pretty ruined. No per-shot white balancing was done.



Cheers!
« Last Edit: April 27, 2021, 07:44:49 am by msantana »
Logged
Miguel Santana da Silva / Digital Compositor @ ILM London

MichaelKoerner

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • www.medien-atelier.at
Re: Scanning Colour Negative Film - Device profiling?
« Reply #3 on: April 27, 2021, 07:27:57 am »

If you were actually filming on CN material, you could film a hardcopy it8 target and apply the profiling workflow AFTER whatever scanning/negative conversion process you choose.

But I assume you work with already processed, aka "old", films, so...  :(

My experience with scanning transparencies using Vuescan RAWs + different profiling tools and transparent it8-targets is similar to yours: Still too much guess work included. Sorry!

But thanks for the lengthy description of your workflow, interesting read.

msantana

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
    • Portfolio
Re: Scanning Colour Negative Film - Device profiling?
« Reply #4 on: April 27, 2021, 07:59:36 am »

If you were actually filming on CN material, you could film a hardcopy it8 target and apply the profiling workflow AFTER whatever scanning/negative conversion process you choose.

But I assume you work with already processed, aka "old", films, so...  :(


Thanks Michael! I think touches on where I suspect a lot of debate and miscommunication comes from.

The intention I think most people want isn't to profile the film itself - if it was, we may as well shoot digital since if you could or bothered to create perfect profiles of each photographed scene on film, it should in theory look no different to a digitally shot image profiled from the same target. Textural qualities aside, obviously. Nowadays I guess we mostly shoot film for all the beautiful idiosyncrasies and nonlinear responses in tone and hue and saturation, which isn't something we want to 'correct out'. I've done this by shooting film + digital exposure brackets of a Colorchecker SG target for multiple rolls, and when you properly, meticulously 'correct' film in line with digital it loses a lot of the colour magic. In fact, colour scientists at big post facilities have the task of creating a complex series of reversible film-to working scene-linear matrices and tone curves so that we can integrate our VFX work into film plates and work as normal in something like ACEScg, before reverting the finished composites back to the film scan state losslessly.

The intention of the kind of device profiling I'm talking about is making sure the scanner delivers an image that accurately represents the colours of the neg as a positive. So that the inversion work has a starting place closer to the actual colours of the physical transparency, i.e. the orange mask is the actual orange mask colour and isn't subjected to the scanner's uncorrected hardware deficiency. But all the reservations from the 'alternative workflow' with an it8 profile in my post still apply.

My hope is that those deficiencies are linear enough, and therefore not worth worrying too much about for any half-decent device. But this is something I don't have experience in and would welcome a professional opinion on!

This is how I see it anyway - could be wrong  :)
« Last Edit: April 27, 2021, 08:40:15 am by msantana »
Logged
Miguel Santana da Silva / Digital Compositor @ ILM London

MichaelKoerner

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • www.medien-atelier.at
Re: Scanning Colour Negative Film - Device profiling?
« Reply #5 on: April 27, 2021, 09:46:01 am »

Well, I'm far for from a professional in that area - just interested in color management topics generally.

In my understanding of ICC profiling you always need a substrate to measure - the "device" itself could only be measured via its spectral behaviour. This could be a topic for Franz Herbert (chameleo eu).

msantana

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
    • Portfolio
Re: Scanning Colour Negative Film - Device profiling?
« Reply #6 on: April 27, 2021, 09:54:48 am »

Likewise - I'm by trade more of a pixel-pusher than a colour scientist. Take any conclusions I suggest with a pinch of salt! Would be interested to hear Franz Herbert's thoughts.
Logged
Miguel Santana da Silva / Digital Compositor @ ILM London

MichaelKoerner

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • www.medien-atelier.at
Re: Scanning Colour Negative Film - Device profiling?
« Reply #7 on: April 27, 2021, 10:36:02 am »

AFAIK, Franz is one of the founding fathers of modern color management and has already developed some solutions in matters of spectral imaging.

Don't know whether he reads LuLa, so I'd point this conversation out to him - if he has something to say, he might chime in. I'm sure you could also contact him directly.

One additional comment (perhaps off topic): In point 4 of your workflow description you mention additional gamma correction and shoulder compression. At this point you leave the "holy path" of scene-referring, I guess? I was of the mind that "compressed shoulder behaviour" in the highlights were one of the main reasons for using CN film in general. Am I wrong or do you ajust add some more compression in post for aesthetic reasons?

msantana

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
    • Portfolio
Re: Scanning Colour Negative Film - Device profiling?
« Reply #8 on: April 27, 2021, 10:53:34 am »

One additional comment (perhaps off topic): In point 4 of your workflow description you mention additional gamma correction and shoulder compression. At this point you leave the "holy path" of scene-referring, I guess? I was of the mind that "compressed shoulder behaviour" in the highlights were one of the main reasons for using CN film in general. Am I wrong or do you ajust add some more compression in post for aesthetic reasons?


Correct - purely for aesthetic reasons. After white balancing and exposure in scene-linear it's just about creating a pleasing display-referred image. So for the highlight rolloff it's so I don't have to apply a darker exposure just to avoid hard clipping in the linear gamma - just rounds specular highlights off nicely. So it can be as simple as a gamma & shoulder thing for a relatively neutral scan similar to the Nikon Scan software, or transform from linear to cineon gamma and apply a motion picture print emulation LUT for example. Totally a personal taste thing at that point.
Logged
Miguel Santana da Silva / Digital Compositor @ ILM London

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: Scanning Colour Negative Film - Device profiling?
« Reply #9 on: April 27, 2021, 10:56:02 am »

In my understanding of ICC profiling you always need a substrate to measure - the "device" itself could only be measured via its spectral behaviour.
Absolutely correct, which is one (of many) reasons it's not possible to produce an ICC profile for a color neg. Nor is one needed. There's no target sure, but worse; you MUST measure the actual color values of each color patch as a reference just for creation of an ICC profile. The end result is a profile and a profile 'profiles/fingerprints' the device behavior. But with a color neg, you must invert that color and more importantly, remove the orange mask. A mask that varies based on many values. Absolutely not the case with a transparency: with proper tools (expensive), one can measure each color of the IT8 or Q60 patch to produce a reference of that color. One can produce a fingerprint of device behavior that doesn't vary (you don't profile a moving target). We can dive deeper into why it's both not possible and pointless to run down a rabbit hole of color neg scanner ICC profiles.

How was this scanning of negs done prior to robust color management? On a calibrated display. Later on a calibrated and profiled display in some workflows. One needs good software at the scanning stage to deal with the orange mask and inversion just to provide a positive preview to start working (should anyone assume, you load the neg, hit a button and it's all just as you wish the neg to appear; no!). Anyone here beside me spend countless hours in the darkroom printing color negs? What did you do; place a color filter pack you assume as a starting point in enlarger, make test strips, process and view. Alter color packs to produce a rendering you prefer (see Rendering the Print article which covers this and digital). Same scanning color negs. OR scan by DSLR and inversion.

The answer is simple* get a good piece of software designed for this task. I've owned, reviewed and consulted with dozens of scanner manufacturers as well as owners. I can list (from memory) a few, very few products that dealt with color neg scanning well; providing a starting point for RENDERING the image as the scan operator/photographer desired. Leaf (my first scanner, a Leafscan 45) did a pretty good job. LinoColor from Heidelberg and FlexColor/ColorFlex from Imacon come to mind. My ScanMate 5000 PMT drum scanner software did a good job. But once a preview of the positive is seen, the job isn't done! This is a starting point for rendering the image as the creator desires. A lot like dealing with raw data! Which is why  raw is so often called a 'digital negative'.

Here's a raw image. An actual raw image and we should all know, it needs, like a color neg, to be rendered. NOT all raw converters do the same job, as well initially or ever. Ditto with the users driving the scanning software.



There are products still available for color neg scanning that do a good job as an initial preview for subsequent rendering if desired; SilverFast is very good at this. No need to attempt to reinvent a process that's been developed for decades and has worked for so many. As for DSLR, raw conversions, there are reported solutions that work well too.

Can we or do we need ICC profiles for color negs? No. Do we need post scan gyrations to produce good rendering of color negs instead of just using a good tool designed for this? Not really but if someone has a 'unique' workflow they wish to 'sell' so be it. It's a tough sell and many have no need for it. In the end, this entire concept presented is well summed up by the late, great Bruce Fraser:

"You can do all sorts of things that are fiendishly clever, then fall 
in love with them because they're fiendishly clever, while 
overlooking the fact that they take a great deal more work to obtain 
results that stupid people get in half the time. As someone who has 
created a lot of fiendishly clever but ultimately useless techniques 
in his day, I'd say this sounds like an example."
Bruce Fraser

*“There seems to be some perverse human characteristic that likes to make easy things difficult.” -Warren Buffett
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: Scanning Colour Negative Film - Device profiling?
« Reply #10 on: April 27, 2021, 10:59:39 am »

One additional comment (perhaps off topic): In point 4 of your workflow description you mention additional gamma correction and shoulder compression. At this point you leave the "holy path" of scene-referring, I guess?
Was scene referred in the first place? Trust but verify.....  ;)

According to the ICC:
3.36
scene-referred image state
image state associated with image data that represents estimates of the colour-space coordinates of the elements of a scene
Note 1 to entry: When the phrase “scene-referred” is used as a qualifier to an object, it implies that the object is in a scene-referred image state. For example, scene-referred image data are image data in a scene-referred image state.
Note 2 to entry: Scene-referred image data can be determined from raw DSC image data before colour-rendering is performed. Generally, DSCs do not write scene-referred image data in image files, but some may do so in a special mode intended for this purpose. Typically, DSCs write standard output-referred image data where colour-rendering has already been performed.
Note 3 to entry: Scene-referred image data typically represent relative scene colourimetry estimates. Absolute scene colourimetry estimates may be calculated using a scaling factor. The scaling factor can be derived from additional information such as the image OECF, FNumber or ApertureValue, and ExposureTime or ShutterSpeedValue tags.
Note 4 to entry: Scene-referred image data may contain inaccuracies due to the dynamic range limitations of the capture device, noise from various sources, quantization, optical blurring and flare that are not corrected for, and colour analysis errors due to capture device metamerism. In some cases, these sources of inaccuracy can be significant.
Note 5 to entry: The transformation from raw DSC image data to scene-referred image data depends on the relative adopted whites selected for the scene and the colour space used to encode the image data. If the chosen scene adopted white is inappropriate, additional errors will be introduced into the scene-referred image data. These errors may be correctable if the transformation used to produce the scene-referred image data are known and the colour encoding used for the incorrect scene-referred image data has adequate precision and dynamic range.
Note 6 to entry: The scene may correspond to an actual view of the natural world or may be a computer-generated virtual scene simulating such a view. It may also correspond to a modified scene determined by applying modifications to an original scene to produce some different desired scene. Any such scene modifications should leave the image in a scene-referred image state and should be done in the context of an expected colour-rendering transform.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Doug Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2197
Re: Scanning Colour Negative Film - Device profiling?
« Reply #11 on: April 27, 2021, 01:01:51 pm »

Scene referred workflows are rabbit holes. They have their uses but mostly in things like scientific documentation or replication and this is non trivial. And even harder with film which isn't intrinsically colorimetric nor linear. At least digital sensors are quite linear even if their filters aren't colorimetric.

You might find some good info by researching academic papers published by Kodak scientists and people at Rochester Institute of Technology. Go back to 1970-80, just before digital sensors took over. There was a small but significant need back then. Might be worth it to use Google Scholar to round up some interesting papers then visit your local university library. Also, the Academy of Arts and Sciences has done a lot of work preserving film as well as scene referred processing to archive original positives.
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: Scanning Colour Negative Film - Device profiling?
« Reply #12 on: April 27, 2021, 04:32:31 pm »

AFAIK, Franz is one of the founding fathers of modern color management and has already developed some solutions in matters of spectral imaging.
Great guy, and way back, with the introduction of ColorBlind, one of the first products of many to provide a dongle (although I did find an easy way back then to get around it). Over the years, the dongle collection for color management products has grown as seen below. I still have a very old MacBook that can boot OS9 and earlier just in case I'd want revisit and use, ColorBlind Edit, which I believe was one of the first profile editors. Even before Kodak had such a solution (albeit an editor inside of Photoshop). Franz was ahead of his time!
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

smthopr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 612
    • Bruce Alan Greene Cinematography
Re: Scanning Colour Negative Film - Device profiling?
« Reply #13 on: May 01, 2021, 09:55:20 am »

Included with Nikon scan, there was a device profile for the scanner. You might want to try that one to tag your scan before converting to a working space.
Logged
Bruce Alan Greene
www.brucealangreene.com
Pages: [1]   Go Up