Does a raw file have a color space?

[papa v2.0]

Therefor the conversion with the matrix (actually, with its inverse) is not satisfactory. I think this is the reason, that no matter how hard one calibrates ACR, one can not get all colors right, not even all colors of the small color checker.
[a href=\"index.php?act=findpost&pid=168058\"][{POST_SNAPBACK}][/a]

In ACR the conversion matrices were made from a target based characterisation. A spectral characterisation method could produce more accurate results. (Far more complicated and time consuming, a day at least on the monochromator!)

But I think the problem with 3rd party raw processors is that they use a 'one characterisaion fit all'
approach for each camera. (Thomas Knoll uses two, illuminant  A  balanced and a D65 i think, correct me if im wrong).

What you have to remember is that characterisation is system specific. Ie only valid for that body lens, filter etc combinations.

If you have a different set up from the one used for the characterisation, this will put you at a disadvantage straight away.

I know that this will grate with some people but I would use the manufacturers RAW converter and compare with ACR – just a thought.

[Panopeeper]

I find it essential for a raw processor to have analyzed the spectral respones of the sensors and base the comlor converson on that. I don't think the "system" issue is of relevance. The characteristics of a given sensor (with the filters) are important. I see the effect of non-neutral lenses in the area of white balancing, i.e. the set white balances, like daylight, incandescent, etc. should depend on the lens, but this iis independent of the method of color conversion.

It is well-known - except among the creators of raw processors - that the camera manufacturer's software yields the best color. Unfortunately, those raw processors are far from "professional" raw converters on the area of features and comfort.

[paulbk]

The latest version of Canon's DPP (version 3.2.0.4) is very usable. If you have tried earlier versions and lost patience because of the impossible user interface, I understand. If you are a Canon shooter, well worth giving the newer version a few hours try.

And I agree, DPP gives the best RAW convert in color fidelity, sharpening, and shadow detail. Albeit not as flexible as Lightroom or ACR.

[Josh-H]

And I agree, DPP gives the best RAW convert in color fidelity, sharpening, and shadow detail. Albeit not as flexible as Lightroom or ACR.

I have found this to be true - with DPP anyway.

However, [and its a big however] - the workflow advantages of Lightroom make it a no brainer to us LR.

And.. you can actually get the same results [or better] than DPP - it just takes more work to get there.

As good as DPP conversions are it just cant compete with the power of LR for processing. And I am a big DPP fan.

[Schewe]

The "Essentially Grayscale" comments date back to another respected color geek and in my case, mentor, Bruce Fraser. I believe I quoted him from his book in the previous debated thread. IF Bruce were alive and had he asked the same questions to this group of experts and received the same answers, I suspect he, (as I will from now on),  would use better language than "essentially Grayscale" to describe this aspect of Raw data.
[a href=\"index.php?act=findpost&pid=168063\"][{POST_SNAPBACK}][/a]

Well, that's where my understanding and expectations come from as well. I have no problem accepting the words of Thomas and Chris...they obviously know more than I about the subject. But I still think it's wrong to attribute the term "color space" as defined by general acceptance because (even Bruce has pointed out) that it's more of a color mixing function and a camera sensor will have different spectral responses from different spectral sources. Hence the difficulties for creating a profile for a digital camera.

So, I'll also refrain from calling what the camera captures as "grayscale data" but I'n not convinced a camera has a fixed "color space" either...which is why ya gotta adjust for the white balance of the raw file.

[tived]

Hello,

this is very interesting, but how does this help us processing our images better? can we profile better with this knowledge? where do we go from here?

Sorry if this is a trivial question

Henrik

[Panopeeper]

quote=Schewe,Jan 18 2008, 08:44 PM]
I have no problem accepting the words of Thomas and Chris...they obviously know more than I about the subject[/quote]

Jeff, this entire discussion is totally inconsequent. Does it really matter if the raw image has a color space or not? Not the very least, *except* for a better understanding of the subject in general.

Therefor one should not accept anyone's word as a deus ex machine, but one should try *reasoning* to come to a result. This starts with the very basics, like

a camera sensor will have different spectral responses from different spectral sources

the sensor has only *one* spectral response per channel. The spectral response includes the response of the sensels of a given channel *for all visible light waves*. In other words, the spectral response shows, how the sensor behaves with *any* light source.

Hence the difficulties for creating a profile for a digital camera

The source of difficulties is, that the transformation between the camera's color space and another one (basically CIE, as that is the starter point for the others) is not as straightforward, as it is with the "artificial" color spaces, which have been defined just so, that they can be transformed to and from CIE via a matrix multiplication.

Look at the spectral responses of the three channels of different cameras and see, how different these responses are. Do you believe, that these can be described with only nine parameters? The transformation based on the conversion matrices is an approximation, sometimes better, sometimes worse.

I'n not convinced a camera has a fixed "color space" either...which is why ya gotta adjust for the white balance of the raw file

For the sake of your own understanding, do you mind explaining what you mean with the adjustment for white balance?

[Panopeeper]

how does this help us processing our images better? can we profile better with this knowledge?

Not the least; it is academic pure.

[eronald]

Bjanes:

I would say, THE ANSWER IS NO. Andrew seems correct. At present most Raw files are numbers only, with no clear unambiguous color information. Sadly.

I need only refer to the below fragments of your own post: That EOS 1Ds2 matrix comes from Adobe. If the Raw was intended as a vehicle for colorimetric information  surely some colorimetric data would be present in the file itself? Absent such "reference" data, why should  the  American  "Adobe  Interpretation" be accorded more credence than the Danish "Phase One Color Interpretation" or the privately held French  "Edmund Ronald Color Interpretation" ? Just because Mr. Coffin has found Adobe the easiest place to liberate a Cam2xyz matrix from ?

Of course, were the camera manufacturer to cooperate and measure the colorimetric characteristics of the camera, eg spectral responses,  and write this data into the Raw file, then the situation would change: A map with a clearly marked scale is a very different object from a map with no scale. Although of course there is just so much information about "color" to be derived from data viewed through the 3 filters in a Bayer array.

Oh, and by the way, I use camera profiling software I have written myself to determine the color of the images I take with my Phase back- I eat my own dogfood. I've had occasion to discuss camera color with quite a few other people and while all these smart guys have some working solution to the Raw color issues, everybody agrees that existing technology gets you only so far.

Edmund


PS. It has been described in detail how Thomas Knoll initially derived the matrices in ACR for various cameras by imaging a testchart exposed in a Gretag lightbooth switched to various light settings. While I have the greatest respect for Thomas and find the ACR interpolated matrix color model an ingenious solution to a difficult problem, there is no way he or anyone else would claim that this model yields any colorimetric ground truth.

Does a raw file have a color space?

The question is, “Does a raw file have a color space?” Some answer no for various reasons.


----snip----

   Thanks to Adobe for providing these excellent CAM -> XYZ matrices!
 */
  { "Canon EOS-1Ds Mark II", 0,
   { 6517,-602,-867,-8180,15926,2378,-1618,1771,7633 } },
[a href=\"index.php?act=findpost&pid=167946\"][{POST_SNAPBACK}][/a]

[papa v2.0]

Bjanes:

THE ANSWER IS NO. Andrew is correct. At present most Raw files are numbers only, with no clear unambiguous color information. Sadly.

well thats not completely right

a camera does have a 'colour space'- a very big 'colour space'!
The camera 'colour'  space (or to be precise, sensor space) is determined by its spectral response.

its the raw file that doesn't have a 'colour space' as the data is yet to be correctly interpreted

A raw file is the un calibrated image data for a particular scene - sensor referred data - its not colormetric data, it has to be converted to that in the next step.! Via the camera matrix to CIE XYZ.

Now we have color data.

[eronald]

A raw file is the un calibrated image data for a particular scene - sensor referred data - its not colormetric data, it has to be converted to that in the next step.! Via the camera matrix to CIE XYZ.

Now we have color data.
[a href=\"index.php?act=findpost&pid=168138\"][{POST_SNAPBACK}][/a]

Absolutely right. I love the "It has to be converted". Reminds me of Leonardo, designing the helicopter saying well, all I need now is the engine


Edmund

[digitaldog]

well thats not completely right

a camera does have a 'colour space'- a very big 'colour space'!
The camera 'colour'  space (or to be precise, sensor space) is determined by its spectral response.

its the raw file that doesn't have a 'colour space' as the data is yet to be correctly interpreted

A raw file is the un calibrated image data for a particular scene - sensor referred data - its not colormetric data, it has to be converted to that in the next step.! Via the camera matrix to CIE XYZ.

[a href=\"index.php?act=findpost&pid=168138\"][{POST_SNAPBACK}][/a]

Now we run full circle. The initial question was, does a Raw have a color space (not does a camera have a color space).

Just when we were getting somewhere....

[bjanes]

Bjanes:

I would say, THE ANSWER IS NO. Andrew seems correct. At present most Raw files are numbers only, with no clear unambiguous color information. Sadly.
[a href=\"index.php?act=findpost&pid=168126\"][{POST_SNAPBACK}][/a]

There is no way to get any color from numbers only unless you know the meaning of those numbers. Furthermore, are you impervious to the statements made by Andrew's experts? As Mr. Knoll has stated, it is not possible to get perfect unambiguous  colors from the CFA sensor, since the spectral characteristics of the filters do not match those of the human eye. Furthermore, recent research has shown that there are polymorphisms  in human visual pigments and all people do not see color in the same way. Some are tetrachromats.

I need only refer to the below fragments of your own post: That EOS 1Ds2 matrix comes from Adobe. If the Raw was intended as a vehicle for colorimetric information  surely some colorimetric data would be present in the file itself? Absent such "reference" data, why should  the  American  "Adobe  Interpretation" be accorded more credence than the Danish "Phase One Color Interpretation" or the privately held French  "Edmund Ronald Color Interpretation" ? Just because Mr. Coffin has found Adobe the easiest place to liberate a Cam2xyz matrix from ?

Of course, were the camera manufacturer to cooperate and measure the colorimetric characteristics of the camera, eg spectral responses,  and write this data into the Raw file, then the situation would change: A map with a clearly marked scale is a very different object from a map with no scale. Although of course there is just so much information about "color" to be derived from data viewed through the 3 filters in a Bayer array.
[a href=\"index.php?act=findpost&pid=168126\"][{POST_SNAPBACK}][/a]

The Adobe coefficients only approximate the behavior of the sensor. It may be best to allow each software vendor to use their own methods according to the circumstances and shooting conditions. A spectral power distribution taken over the visible spectrum at 1 nm intervals would take up a lot of space and you would still have to derive the coefficients. The current approach is to tag the file with a few key characteristics and the remaining data can be filled in by reference as is done with the DNG conerter.  The absolute perfection you seek is not attainable, and compromises must be made and these involve judgment.

Oh, and by the way, I use camera profiling software I have written myself to determine the color of the images I take with my Phase back- I eat my own dogfood. I've had occasion to discuss camera color with quite a few other people and while all these smart guys have some working solution to the Raw color issues, everybody agrees that existing technology gets you only so far.

[a href=\"index.php?act=findpost&pid=168126\"][{POST_SNAPBACK}][/a]

Thus far we have been talking about Matrix Profiles, which apply only to relatively behaved devices such as CRTs and some sensors. I don't know what type of profiles you are generating, but most camera profiles generated by third party software are look-up table based and are much more involved than a few coefficients. These latter profiles may work under defined conditions, such as in the studio taking pictures of models under flash, but many authorities such as Bruce Fraser and Andrew Rodney are not enthused with their use under general conditions.

[bjanes]

Now we run full circle. The initial question was, does a Raw have a color space (not does a camera have a color space).

Just when we were getting somewhere....
[a href=\"index.php?act=findpost&pid=168160\"][{POST_SNAPBACK}][/a]

Chris Murphy: "So yes a camera (and thus a Raw file) has a color space. "

[digitaldog]

Chris Murphy: "So yes a camera (and thus a Raw file) has a color space. "
[a href=\"index.php?act=findpost&pid=168167\"][{POST_SNAPBACK}][/a]

Yes he does. And a good lawyer could continue to use the above words of Thomas to try to sway the jury, for example:

An early processing step fills in the missing values, and you have a 3 or 4 channel image as a result.

When is this early process step happening (before or after we have a raw)? If after, then the lawyer could say "well until that step, this Raw hasn't filled in the missing values, there's no color (yet)".

Chris further lumps the camera and the Raw color space issue into one:

So yes a camera (and thus a Raw file) has a color space.

We agree a camera has a color space/color mixing function but what about the Raw? He's somewhat unclear here.

The lawyer could even argue that what Eric says here doesn't apply to the Raw data:

The image was recorded through a set of camera spectral sensitivities which defines the intrinsic colorimetric characteristics of the image. An simplistic way to think of this (while not purely accurate) is that the image was recorded through a set of "primaries" and these primaries define the color space of the image.

All heading us back to the "Essentially Grayscale", somewhat ambiguous original statement about the Raw data itself.

I'm not a lawyer and don't want to mince the words here.

The clincher here, for me is what Eric wrote:

In the context you described, a simple transform is applied to convert from the camera primaries to a new set of primaries (eg CIE or working space) that have more desirable characteristics than the RAW primaries.

Characteristic of Raw primaries is the key in my mind. He is associating the Raw to primaries which seems to strongly suggest color (space).

Of course the lawyer could nitpick his use of RAW instead of Raw (its not an acronym, a pet peeve of some) and try to get the jury to look the other way.

The lawyer could further use this from Thomas to say he doesn't seem to strongly believe the statement:

it certainly seems like a color space.

In the end, too many semantics!

[Panopeeper]

If the Raw was intended as a vehicle for colorimetric information  surely some colorimetric data would be present in the file itself?

What exactly do you miss?

1. the public knowledge of the colorimetric information?

This can not be made a condition for being a color space. If the characteristics of a color space are known only among a certain circle of people, then it is still a color space IMO.

2. the presence of it in the file?

If that is a condition, then sRGB, aRGB, etc. are no color spaces, because their characteristics are not present in the files.

Btw, DNG files do contain the conversion matrices. Does a DNG file have a color space, while the native raw file does not?

Absent such "reference" data, why should  the  American  "Adobe  Interpretation" be accorded more credence than the Danish "Phase One Color Interpretation" or the privately held French  "Edmund Ronald Color Interpretation" ? Just because Mr. Coffin has found Adobe the easiest place to liberate a Cam2xyz matrix from ?

As any such matrix is an approximation only, this question is reduced to "who can approximate better". As such, it plays no role in the topic.

Of course, were the camera manufacturer to cooperate and measure the colorimetric characteristics of the camera, eg spectral responses,  and write this data into the Raw file, then the situation would change: A map with a clearly marked scale is a very different object from a map with no scale

What if they would only publish that information, but not put it into the file? Just like sRGB?

[Panopeeper]

Characteristic of Raw primaries is the key in my mind

Primaries exist only in an orthogonal system. Raw files do not have primaries.

Therefor raw files/images/cameras (pick whatever you like) do not have color spaces in the sense of those definitions, which expect the description of a color in orthogonal form, like this one of those I posted yesterday:

A color model is an abstract mathematical model describing the way colors can be represented as tuples of numbers, typically as three or four values or color components. When this model is associated with a precise description of how the components are to be interpreted (viewing conditions, etc.), the resulting set of colors is called a color space

This was the reason I suggested to lay done a definition first.

[digitaldog]

My fourth expert has responded. More food for thought:

On 1/19/08 10:11 AM, "Jack Holm"  wrote:

I think a couple of things in Eric’s response could be confusing.

- Unless the camera spectral sensitivities are colorimetric, they do not define the intrinsic colorimetric characteristics of an image.

- Also, primaries are for synthesis and can cause some wrong thinking if discussed in relation to analysis.

The second paragraph of Thomas’ response is important.

The short answer to the question is:

Raw image data is in some native camera color space, but it is not a colorimetric color space, and has no single “correct” relationship to colorimetry.

The same thing could be said about film negative densities.

Someone has to make a choice of how to convert values in non-colorimetric color spaces to colorimetric ones. There are better and worse choices, but no single correct conversion (unless the “scene” you are photographing has only three independent colorants, like with film scanning).

A purist might argue that a color space not based on colorimetry is not really a color space because it is not an assignment of numerical values to colors, defining colors as a human sensation. In the standards committees we decided it is useful to be able to talk about non-colorimetric color spaces so we allow them and use “colorimetric color spaces” when appropriate.

Jack

Jack often differs with Eric from clearly a semantic standpoint, something most of us here appear to be doing as well. We're talking two world class color scientist with extreme expertise in digital imaging from the point of digital capture.

[papa v2.0]

this is  good to see the views on such things

But it is obvious there is a lot of confusion over terminology and definitions
"In the end, too many semantics!"

The way I like to think is

Camera -sensor space-

RAW file - just a data file

CIEXYZ colour encoding space.

so any way im off to the pub for a pint and to see what the general consensus is! (any excuse)

have fun

[digitaldog]

Some here may recall another recent debate about colors outside the spectrum locus not being colors and the quote I got from another expert, Karl Lang:

Color, is a perceptual property. So if you can't see it it's not a 
color. Color is not a particular wavelength of light. It is a 
cognitive perception that is the end result of the excitation of 
photoreceptors followed by retinal processing and ending in the 
visual cortex. We define colors based on perceptual experiments.

A coordinate in a "colorspace" outside the spectrum locus is not a 
color. We often refer to these as "imaginary colors" but this is by 
and large also erroneous (you can't map an imaginary color from one 
colorspace to another as the math (and experimental data) for each 
colorspace breaks down outside the spectrum locus.

Jack's post seems to use the same ideas here to discuss non-colorimetric color space no?

A purist might argue that a color space not based on colorimetry is not really a color space because it is not an assignment of numerical values to colors, defining colors as a human sensation

We seemed to agree about this post of Karl's and if so, now we have to rework our ideas about color spaces based on non colorimetric data I suspect. Chris also discusses this above (imagery colors).