Pages: 1 2 [3] 4 5   Go Down

Author Topic: DCamProf for dummies?  (Read 50078 times)

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf for dummies?
« Reply #40 on: September 06, 2015, 12:16:01 pm »

Updated the http://www.ludd.ltu.se/~torger/photography/camera-profiling.html#the_easy_way with an alternative to render a DCP which has a custom look with a an all-around look provided designed by me (added the same to the Capture One case).

I've designed the look with the intention for it to be subtle and generic. The warmup of greens and yellows is pretty typical (many commercial profiles I've looked at contains some sort of warmup), there's a subtle saturation increase there too (skin tone range excluded) which also is very typical.

If you think it adds any value or just messes things up, or you would like it to be a bit different is personal of course, but I think it's nice to have the subjective look feature represented in the basic use case too (which will be the only use case most users will do).
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf for dummies?
« Reply #41 on: September 06, 2015, 12:31:22 pm »

If a precise light temperature reading is important I could still make a spotread with our spectro when we shoot the chart, unless it's overkill ?

As for flare, we'll probably manage to deal with it using the right modifiers and checking the raw histograms.
Regarding those white and black patches I read an article suggesting synthetic ones be added to ti3 files. It features some other advices too, and made me wonder if they'd apply to Dcamprof as well ?

http://ninedegreesbelow.com/photography/well-behaved-camera-profile.html

A precise light temperature reading is not really important. If you shoot something way off from D50 there can be a value to have the spectrum, but if you shoot something close to D50 and want to make D50 colors you don't need to measure it.

I haven't read that article in detail, but you don't need synthetic white and black patches for DCamProf, it's been designed to handle those situations. Argyll is not really designed for camera profiles (it's more about scanner profiles when it comes to input profiles) which makes some quirks, that's one reason I made DCamProf. The manual edits Elle does in the article seems to be to trick Argyll into making decisions good for a camera profile rather than a scanner profile. DCamProf already makes those things "under the hood" as it was designed towards the camera case rather than the scanner case.
Logged

hk1020

  • Newbie
  • *
  • Offline Offline
  • Posts: 19
Re: DCamProf for dummies?
« Reply #42 on: September 06, 2015, 12:53:25 pm »

Tested the workflow, I don't get the same result. I suspect that your 0.9.6 build actually is an older. Can you run dcamprof without parameters and see what the first line it outputs is? It should say "DCamProf v0.9.6 - a digital camera profiling tool."

I've attached the profile I get when running your script. Let us know if it works better.

Thanks. I suspected something silly like this myself but I am definitely using 0.9.6. Numerical problems maybe? I am using Cygwin 32bit on win10.  I noticed it takes quite some time with interpolation and smoothing so there might be something critical numerically. Anyway, I just ran everything again and put a log file in that dropbox directory (build.log).

Your profile is indeed different and with it I can sort of come up with a curve that provides dark output in the shadows. But it seems still strange there.

And you may use my picture.  I've uploaded both raw and ooc jepg to the dropbox directory. Unfortunately, it is not really suited to judge the colors.

I am not sure how much time I can spend on this today so expect some sluggish response of mine.
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf for dummies?
« Reply #43 on: September 06, 2015, 01:12:20 pm »

Thanks. I suspected something silly like this myself but I am definitely using 0.9.6. Numerical problems maybe? I am using Cygwin 32bit on win10.  I noticed it takes quite some time with interpolation and smoothing so there might be something critical numerically. Anyway, I just ran everything again and put a log file in that dropbox directory (build.log).

Your profile is indeed different and with it I can sort of come up with a curve that provides dark output in the shadows. But it seems still strange there.

And you may use my picture.  I've uploaded both raw and ooc jepg to the dropbox directory. Unfortunately, it is not really suited to judge the colors.

I am not sure how much time I can spend on this today so expect some sluggish response of mine.

Thanks for the raw file. I definitely see problems in it also with the file I generated. What's special about that file is that it clips to black a lot, many cameras do that, but the one's I've used most in my testing (P45+, Hasselblad H4D-50, Canon 5D Mark II) don't to the same extent. There's still some problem with the elements in the LUT close to 0, which becomes clear on these type of pictures for these type of cameras. I'll be looking at that. Hopefully it's easy to fix this time around too... we'll see.
Logged

professorbalrog

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
Re: DCamProf for dummies?
« Reply #44 on: September 06, 2015, 01:23:52 pm »

Yeah, 0.9.6 is working a lot better in the shadows, but still some weird clipping (that's what it looks like at least) in the areas close to 0. If you want another file to test with, I have a mark 3 file that's got a lot of black and dark areas that I've been using to try and get the curve handles right for the LUT. Let me know if you want to give it a spin, I'll PM it to you
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf for dummies?
« Reply #45 on: September 06, 2015, 01:42:00 pm »

Yeah, 0.9.6 is working a lot better in the shadows, but still some weird clipping (that's what it looks like at least) in the areas close to 0. If you want another file to test with, I have a mark 3 file that's got a lot of black and dark areas that I've been using to try and get the curve handles right for the LUT. Let me know if you want to give it a spin, I'll PM it to you

I don't think I need any more test file yet, the on I got from hk1020 was great. I know exactly what the problem is now. There's some noise in the shadows, and then the raw data is clipped at some black level, this creates "impossible" combinations, like no signal on green but some blue etc. This is just noise, but it gets mapped out of gamut and it gets crazy values. My LUT code did not take into account that there could be noise. The DNG profile LUT works in a different way so it's unaffected by this problem.

It's not so easily fixed though. The problem is about identifying which LUT positions that are "crazy" and then decide where to map those. There is such code already which is about mapping to nearest valid neighbor, but unfortunately the nearest valid can be a muuuch brighter value in this case, this is what causes noise clipped shadows to go haywire.
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf for dummies?
« Reply #46 on: September 06, 2015, 04:29:50 pm »

I think I have it working now... here's a profile

Did a release of the code, v0.9.7. I'm going to bed now, not sure how much time I can code next week. There's a quite large change to the ICC LUT generation code and I did not get much time to test (just wanted to get something out before sleep), so there's a risk it's still broken, but if so at least in a different way than before :-)
« Last Edit: September 06, 2015, 04:43:40 pm by torger »
Logged

hk1020

  • Newbie
  • *
  • Offline Offline
  • Posts: 19
Re: DCamProf for dummies?
« Reply #47 on: September 07, 2015, 07:19:44 pm »

I think I have it working now... here's a profile

Did a release of the code, v0.9.7. I'm going to bed now, not sure how much time I can code next week. There's a quite large change to the ICC LUT generation code and I did not get much time to test (just wanted to get something out before sleep), so there's a risk it's still broken, but if so at least in a different way than before :-)

Thanks, I just tried it but didn't have time to do much. It sure looks better than anything before. Finding a proper curve is still something I need to do.

I also tried to build from source again but this time the program crashes. Tried to investigate but then my whole pc crashed. So more later.
Logged

hk1020

  • Newbie
  • *
  • Offline Offline
  • Posts: 19
Re: DCamProf for dummies?
« Reply #48 on: September 09, 2015, 07:11:37 pm »

Thanks, I just tried it but didn't have time to do much. It sure looks better than anything before. Finding a proper curve is still something I need to do.

I also tried to build from source again but this time the program crashes. Tried to investigate but then my whole pc crashed. So more later.

I've got news on the crash. The assert in icclut.c line 97 fails. It happens with this command:
dcamprof make-icc -n nex5 -f auto_DSC04159_C11.tif -t tone-curve.json profile.json prelim-profile.icm
...
Interpolating out of gamut entries...assertion "max1 != 0" failed: file "icclut.c", line 97, function: cam_similarity
./doit.new: line 8:  5488 Aborted                 (core dumped) ../dcamprof make-icc -n 'nex5' -f auto_DSC04159_C11.tif -t tone-curve.json profile.json prelim-profile.icm

However, this only happens on 32bit Cygwin. If I compile it on a 64bit Cygwin installation the program works. The difference between 32bit and 64bit gcc is the different default for the -mfpmath option, which controls usage of the 80bit registers (google it for details). On 64bit the default is -mfpmath=sse and on 32bit it is -mfpmath=387 If I add -mfpmath=387 to the makefile and compile it on 64bit the program crashes there as well.  The strange thing is I can't use -mfpmath=sse on 32bit.  I suspect this to be a current Cygwin bug as I have used -mfpmath=sse in the past although with gfortran only. Simply compiling a helloWorld.c with -fpmath=sse immediately produces an error:

 vega> gcc -mfpmath=sse hello.c
hello.c:1:0: warning: SSE instruction set disabled, using 387 arithmetics
 #include <stdio.h>
 ^


So I can't produce a working dcamprof for 32bit Cygiwn at the moment. Disabling the assert is no option, you end up with NaNs if you do.

Anyway, no need to send me a binary or anything, I found the mingw compiled one in the other dcamprof thread which works just fine. And so does the 64bit Cygwin one. The results printed to the console are almost identical.

I just thought you might be interested in this result as I believe it might point to some numerical sensitivities in the code.

More on the actual profile in another reply later.
Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf for dummies?
« Reply #49 on: September 09, 2015, 07:38:09 pm »

I found the mingw compiled one in the other dcamprof thread which works just fine. And so does the 64bit Cygwin one. The results printed to the console are almost identical.
shouldn't they be all identical ? might be something worth looking into code-wise
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf for dummies?
« Reply #50 on: September 10, 2015, 03:48:13 am »

Interpolating out of gamut entries...assertion "max1 != 0" failed: file "icclut.c", line 97, function: cam_similarity

Thanks for the report. Floating point stuff can be dodgy at times. In theory the results should always be the same, but sometimes some compiler math optimizations can cause slight differences to occur. Could be such a thing, or it's just some sort of bug. I'll look into it.

EDIT: tried checksumming the LUT data, there is some bug that causes varied results when running with OpenMP, but with OpenMP disabled I get the same result. This is a quite typical bug to have (concurrency bug), so if you have stability issues disabling OpenMP is a good first step. Another step is to disable optimization (remove -02 from makefile), if the Cygwin compiler has math optimization issues that can help solving the problem.

I'll fix the parallelism bug to the next release.
« Last Edit: September 10, 2015, 03:59:21 am by torger »
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf for dummies?
« Reply #51 on: September 10, 2015, 04:31:57 am »

I'm doing some other stuff to the next release so it's a bit messy to make a quick release just now, but if you change

#pragma omp critical
                    {
                        if (r_ != 0 || g_ != 0 || b_ != 0) {
                            ig_offsets[3*ig_offsets_len+0] = r_;
                            ig_offsets[3*ig_offsets_len+1] = g_;
                            ig_offsets[3*ig_offsets_len+2] = b_;
                            ig_offsets_len++;
                        }
                    }


to

                    if (cam.v[0] > 0 || cam.v[1] > 0 || cam.v[2] > 0) {
#pragma omp critical
                        {
                            ig_offsets[3*ig_offsets_len+0] = r_;
                            ig_offsets[3*ig_offsets_len+1] = g_;
                            ig_offsets[3*ig_offsets_len+2] = b_;
                            ig_offsets_len++;
                        }
                    }

in icclut.c you should hopefully get rid of the immediate crash problem.

I've fixed the parallelism issue in my local code too, but it's non-critical. It just an array that ends up in different order from time to time which make the smoothing of out of gamut entries be mildly different. Fix for that will come in next release.
« Last Edit: September 10, 2015, 04:34:35 am by torger »
Logged

hk1020

  • Newbie
  • *
  • Offline Offline
  • Posts: 19
Re: DCamProf for dummies?
« Reply #52 on: September 10, 2015, 04:44:00 pm »

Thanks for the report. Floating point stuff can be dodgy at times. In theory the results should always be the same, but sometimes some compiler math optimizations can cause slight differences to occur. Could be such a thing, or it's just some sort of bug. I'll look into it.

EDIT: tried checksumming the LUT data, there is some bug that causes varied results when running with OpenMP, but with OpenMP disabled I get the same result. This is a quite typical bug to have (concurrency bug), so if you have stability issues disabling OpenMP is a good first step. Another step is to disable optimization (remove -02 from makefile), if the Cygwin compiler has math optimization issues that can help solving the problem.

I'll fix the parallelism bug to the next release.

Don't know yet but there is certainly a numerical problem. I've managed to compile dcamprof on 32bit Cygwin now. The options to add are -mfpmath=sse -march=native and that certainly indicates something is wrong.  With -mfpmath=387 you get higher precision for most calculations so the non-zeroes you see in icclut with sse math might actually be spurious. Unfortunately, the effects of 387 math are rather more complicated and you might get less precision. Check the articles on the net.  The sse math is gcc's default on 64bit installations, no matter if Cygwin, mingw or Linux.

And none of your suggested changes work: modified source, remove -O2, remove -fopenmp.  Everything crashes.
Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf for dummies?
« Reply #53 on: September 10, 2015, 05:09:28 pm »

I think that shall be moved to the main technical topic here, rather than stay in "for dummies" topic, no ?
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf for dummies?
« Reply #54 on: September 11, 2015, 10:02:17 am »

I think that shall be moved to the main technical topic here, rather than stay in "for dummies" topic, no ?

Heh, yes when talking about source code it's no longer for dummies ;D

I don't manage to recreate the crash issue. Works well here, sse or 387. The modified source should have at least changed how the software crashes as it makes a specific test that the value is not zero first, and then the assert() on not being zero of the same value should not crash, unless there's some overwrite bug that sets it to zero again in-between the first check and the assert.

One should not need to worry about precision in this case, it's 64 bit floating point which is already big overkill. However if some calculation end up at zero so there's division by zero somewhere it can crash on some assert() somewhere. That's probably what happens in hk1020's crash, I can't reproduce it with my data though.
Logged

Frederic_H

  • Full Member
  • ***
  • Offline Offline
  • Posts: 156
    • www.fredericharster.com
Re: DCamProf for dummies?
« Reply #55 on: September 11, 2015, 02:30:03 pm »

This one belongs to the dummies section  ;)

"Export the target image again (or any other image) in the same way as before but this time with the standard film curve, "Film Standard" or "Auto". Let's call this curve.tif. DCamProf can now compare this file with the file with linear curve and that way figure out the shape of the embedded tone curve:
 dcamprof tiff-tf -f cc24.tif curve.tif tone-curve.json"


If I apply the C1 standard curve to my ETTR linear pic (CC24.tiff unclipped), C1 reports clipped highlights (curve.tiff clipped). Should this be ignored, or shall I use a shot with a lower exposure ?
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf for dummies?
« Reply #56 on: September 11, 2015, 02:36:55 pm »

This one belongs to the dummies section  ;)

"Export the target image again (or any other image) in the same way as before but this time with the standard film curve, "Film Standard" or "Auto". Let's call this curve.tif. DCamProf can now compare this file with the file with linear curve and that way figure out the shape of the embedded tone curve:
 dcamprof tiff-tf -f cc24.tif curve.tif tone-curve.json"


If I apply the C1 standard curve to my ETTR linear pic (CC24.tiff unclipped), C1 reports clipped highlights (curve.tiff clipped). Should this be ignored, or shall I use a shot with a lower exposure ?

This can be ignored. For curve extraction tiff-tf doesn't look at the actual image contents, but just at the embedded "transfer function" which is a special tiff meta data tag.
Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf for dummies?
« Reply #57 on: September 11, 2015, 02:45:45 pm »

This can be ignored. For curve extraction tiff-tf doesn't look at the actual image contents, but just at the embedded "transfer function" which is a special tiff meta data tag.
did you check how embedded "transfer function" (the data in that special tiff meta data tag) changes when you make .tiffs in C1 with different exposure corrections (in C1's UI) x different curves
Logged

Macao

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
Re: DCamProf for dummies?
« Reply #58 on: September 11, 2015, 03:35:56 pm »

Hello i want testing DCamProf ...but i dont have colorchecker passport

Where i can download colorchecker passport raw image for Single-illuminant profile ....before buy the colorchecker.

..thanks

Sorry for my poor English
Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf for dummies?
« Reply #59 on: September 11, 2015, 03:52:58 pm »

Hello i want testing DCamProf ...but i dont have colorchecker passport

Where i can download colorchecker passport raw image for Single-illuminant profile ....before buy the colorchecker.

..thanks

Sorry for my poor English

1) DCamProf allows you to use any target, but then they all have their own challenges - some are too glossy, for some you can't rely on published data (like spectral or Lab, etc) and you need to measure it yourself... X-Rite Passport or ColorChecker Classic certainly on the best part of the spectrum... for some cameras you might find SSF/CMF so you can do w/o target, albeit target is always good to have test the profile (if you don't trust your eyes)

2) http://www.imaging-resource.com is a source of raw files with targets for almost all cameras (MF backs might be a notable exception)... but their illumination is not very ideal.. but still you can use to polish your process
« Last Edit: September 11, 2015, 03:56:43 pm by AlterEgo »
Logged
Pages: 1 2 [3] 4 5   Go Up