Pages: 1 ... 4 5 [6] 7 8 9   Go Down

Author Topic: How to convince Sony to do lossless raw on A7RII and others?  (Read 69657 times)

BrianVS

  • Full Member
  • ***
  • Offline Offline
  • Posts: 164
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #100 on: July 13, 2015, 08:10:15 pm »

Dithering reduces resolution.
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #101 on: July 13, 2015, 11:25:48 pm »

Dithering reduces resolution.

Dithering is inevitable with current sensor technology, by RN. Dithering is inevitable when photon detection is involved, with Poisson statistics thanks to physics.

I have tested very few sensors where the combination wasn't adequate to prevent posterization. The D810 at ISO 64 in 12 bit mode is an exception in deep shadows, but not in 14 bit mode.

http://blog.kasson.com/?p=8770

So are a few Sony sensors at base ISO in 12 bit mode in deep shadows, but not in 13 bit mode.

http://blog.kasson.com/?p=8586

Jim

BrianVS

  • Full Member
  • ***
  • Offline Offline
  • Posts: 164
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #102 on: July 14, 2015, 04:57:47 am »

Dithering is an intentional introduction of noise. Better to have enough bit-depth to resolve the real noise of a sensor. Maybe unless the sensor has a lot of fixed-pattern noise in it, and some of them certainly do.

I don't want noise in an image. I prefer a clean signal with lots of bit-depth. Gives you more to work with.

If people are happy with a lossy compression scheme used to destroy the image of a 42MPixel sensor in a $3000+ camera- great. Sony has better marketing than they do engineers.

« Last Edit: July 14, 2015, 06:05:47 am by BrianVS »
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11310
    • Echophoto
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #103 on: July 14, 2015, 07:17:21 am »

Hi,

The noise that Mr. Kasson refers to is a combination of shot noise and redout noise. If you collect 100 pothons the shot noise will be around 10 photons. So some noise is also present. Some of the noise is coming from reading the sensor.

Best regards
Erik


Dithering is an intentional introduction of noise. Better to have enough bit-depth to resolve the real noise of a sensor. Maybe unless the sensor has a lot of fixed-pattern noise in it, and some of them certainly do.

I don't want noise in an image. I prefer a clean signal with lots of bit-depth. Gives you more to work with.

If people are happy with a lossy compression scheme used to destroy the image of a 42MPixel sensor in a $3000+ camera- great. Sony has better marketing than they do engineers.


Logged
Erik Kaffehr
 

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #104 on: July 14, 2015, 12:17:32 pm »

Dithering is an intentional introduction of noise.

I find that definition overly constrained in two aspects.

The first in the use of the word noise. Wholly deterministic waveforms can provide useful dither.  I myself have a patent on one such technique:

http://www.google.com/patents/US4187466

The second is the word intentional. The result is the same whether the non-signal component is natural or made for the use, and the electrons can't understand intention.


Better to have enough bit-depth to resolve the real noise of a sensor.

If shot noise is real noise, and I think it is, that means that you need to be able to resolve an individual electron. That would take a precision greater than (how much greater than is subject to some debate) log2(FWC).

Maybe unless the sensor has a lot of fixed-pattern noise in it, and some of them certainly do.

I don't want noise in an image. I prefer a clean signal with lots of bit-depth. Gives you more to work with.

You don't get a choice. Even with a ideal sensor, you'll have shot noise. With real sensors, you'll have PRML, and various flavors of RN.

If people are happy with a lossy compression scheme used to destroy the image of a 42MPixel sensor in a $3000+ camera- great. Sony has better marketing than they do engineers.

Your use of the word destroy is curious. I have done a great deal of testing and simulation of cRAM compansion. I think I know what it can do well, and where it has problems. I've never seen, in real images, or in simulations where photon noise is simulated, any artifacts that merit the use of the word destroy. I invite you to post examples.

Would I be happier if Sony didn't use cRAW compression? Sure. Will their use of that form of compression stop me from buying more cameras from them? Heck, no.

Jim

BrianVS

  • Full Member
  • ***
  • Offline Offline
  • Posts: 164
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #105 on: July 14, 2015, 05:56:14 pm »

You need more than 11-bits in the Sony sensor to get down to the noise. Using a shallow bit depth and introducing artificial noise to obscure banding- a poor substitute for proper bit-depth to resolve the charge collected. I would rather have as close to the actual count collected then some 11-bit a/d using a non-linear scale and then destroying the bit-depth further compressing it to 7 bits. But, that's why I use cameras with 14-bits. Why Leica switched to 12-bits with the M246 is a mystery, the sensor is better than that. The M8 output was destroyed by the DNG-8; now having unlocked 14-bit pixels proved that.
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #106 on: July 15, 2015, 11:16:51 am »

You need more than 11-bits in the Sony sensor to get down to the noise.

You continue to make bold (and wrong,  IMHO) statements with no supporting material.

Here's Sony's latest shipping a7x camera modeled by moi:

http://blog.kasson.com/?p=8194

The relevant numbers are: FWC = 60,000 electrons, frame-to-frame variable read noise at base ISO of 6+ electrons, rms.

At base ISO, in the m=1 part of the cRAW tone curve, 1 LSB = 7 electrons. Therefore RN is about 1 LSB rms.

That’s enough dither even with no photon noise at all. Here’s why:

http://blog.kasson.com/?p=6513

Other researchers report perfect dither at higher values, up to 1.3 LSB. Jack Hogan is the local expert on that. In any event, 1 LSB is in the ballpark of even the most stringent criteria.

No signal is the worst case, since the photon noise helps as m changes higher on the tone curve.

Jim
« Last Edit: July 15, 2015, 11:20:55 am by Jim Kasson »
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #107 on: July 15, 2015, 11:20:07 am »

Using a shallow bit depth and introducing artificial noise to obscure banding- a poor substitute for proper bit-depth to resolve the charge collected.


Are you asserting that Sony introduces "artificial" noise in the a7x cameras? If so, please tell us why you believe that to be the case.

Jim

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #108 on: July 15, 2015, 11:44:56 am »

...I would rather have as close to the actual count collected then some 11-bit a/d using a non-linear scale and then destroying the bit-depth further compressing it to 7 bits...


This is the second time you have asserted that the a7x ADCs are nonlinear, rather than linear with a tone curve applied digitally after conversion. That has been speculated, but, AFAIK, never proven. So, I ask you again, how do you know for sure?

Jim

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #109 on: July 15, 2015, 12:32:56 pm »

This is the second time you have asserted that the a7x ADCs are nonlinear, rather than linear with a tone curve applied digitally after conversion. That has been speculated, but, AFAIK, never proven. So, I ask you again, how do you know for sure?

Jim

what is cheaper to do - linear ADCs on die or non linear ADCs on die ... does it make sense for Sony Semi to manufacture for example 24mp sensor specifically for Sony Imaging with non linear ADCs and for Nikon with linear ADCs - hurts the economy of scale ... or they have ADCs which can operate in both modes (then more complex - why the need ?)
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11310
    • Echophoto
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #110 on: July 15, 2015, 02:24:48 pm »

Hi,

I would assume that the kind of ACD-s Sony uses can be adjusted by firmware. I would assume they are digital ramp type devices.

I would assume that Jim is quite right about quantisation errors being masked by natural noise. AFAIK there are some references to Emil Martinec discussing this earlier on this thread.

I would say that the Sony solution should be pretty close to loss less except some extreme scenarios, like the star trail images published by Diglloyd and on the DPReview site. In both cases the problem is caused by delta compression rather than tone curve.

I have much less worry about Sony raw compression that may affect perhaps one in a million images than lack of OLP filtering that will affect 100% of image that have been shot in optimal conditions.

Best regards
Erik



what is cheaper to do - linear ADCs on die or non linear ADCs on die ... does it make sense for Sony Semi to manufacture for example 24mp sensor specifically for Sony Imaging with non linear ADCs and for Nikon with linear ADCs - hurts the economy of scale ... or they have ADCs which can operate in both modes (then more complex - why the need ?)
Logged
Erik Kaffehr
 

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #111 on: July 15, 2015, 02:30:44 pm »

I would assume that the kind of ACD-s Sony uses can be adjusted by firmware. I would assume they are digital ramp type devices.

so Sony is putting more complex ADC devices on die that can do non linear conversion or linear conversion on demand as the camera manufacturer wants, right ?
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11310
    • Echophoto
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #112 on: July 15, 2015, 02:40:25 pm »

Hi,

The ramp type converter is actually very simple and quite accurate. It should offer great flexibility in conversions. It may be slow, though but Sony uses several thousands of them in parallel.

Best regards
Erik


so Sony is putting more complex ADC devices on die that can do non linear conversion or linear conversion on demand as the camera manufacturer wants, right ?
Logged
Erik Kaffehr
 

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #113 on: July 15, 2015, 02:56:55 pm »

so Sony is putting more complex ADC devices on die that can do non linear conversion or linear conversion on demand as the camera manufacturer wants, right ?

There is no question that a single-slope ADC can be made nonlinear by controlling the ramp, which could be generated by a DAC, and thus flexible. The question is: "Do a7x ADCs work that way in fact?"

Jim

LKaven

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1060
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #114 on: July 16, 2015, 04:56:59 pm »

I find that definition overly constrained in two aspects.

The first in the use of the word noise. Wholly deterministic waveforms can provide useful dither.  I myself have a patent on one such technique:

http://www.google.com/patents/US4187466

The second is the word intentional. The result is the same whether the non-signal component is natural or made for the use, and the electrons can't understand intention.

If shot noise is real noise, and I think it is, that means that you need to be able to resolve an individual electron. That would take a precision greater than (how much greater than is subject to some debate) log2(FWC).

You don't get a choice. Even with a ideal sensor, you'll have shot noise. With real sensors, you'll have PRML, and various flavors of RN.

Your use of the word destroy is curious. I have done a great deal of testing and simulation of cRAM compansion. I think I know what it can do well, and where it has problems. I've never seen, in real images, or in simulations where photon noise is simulated, any artifacts that merit the use of the word destroy. I invite you to post examples.

Would I be happier if Sony didn't use cRAW compression? Sure. Will their use of that form of compression stop me from buying more cameras from them? Heck, no.

:-)

The word "intentional" is appropriate here actually.  Recall my claims on "noise" as being a kind of signal that is judged deleterious to the "proper function" of an information consumer.  A sensor coupled with a processing chain /may, by design, permit/ signal components, otherwise often considered as noise in other functions, to be recruited as a part of its proper function. 

Meanwhile.  As much as I can understand the claims that the missing codes do not make a practical difference, I remain incensed at the very idea that Sony throws away good bits before giving me the package.  I feel somehow, *ahem* sure, that one day I will find cases where this makes a difference.  But meanwhile, I just want the bits.  Give me the bits, all the bits, and nothing but the bits.  I'll decide what I need and what I don't.

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #115 on: July 16, 2015, 07:32:01 pm »

:-)

The word "intentional" is appropriate here actually.  Recall my claims on "noise" as being a kind of signal that is judged deleterious to the "proper function" of an information consumer.  A sensor coupled with a processing chain /may, by design, permit/ signal components, otherwise often considered as noise in other functions, to be recruited as a part of its proper function.  

I'm familiar with that approach. Pushed far enough, it can lead to telephone poles, acne, cigarette butts, and discarded beer cans being called noise. There was a long philosophical discussion on DPR about that. Were you part of it? I encouraged a much more narrow definition of noise. But I don't feel like rehashing the discussion here.

Meanwhile.  As much as I can understand the claims that the missing codes do not make a practical difference, I remain incensed at the very idea that Sony throws away good bits before giving me the package.  I feel somehow, *ahem* sure, that one day I will find cases where this makes a difference.  But meanwhile, I just want the bits.  Give me the bits, all the bits, and nothing but the bits.  I'll decide what I need and what I don't.

I'm with you. I want all those bits, too, but their absence won't keep me from buying and using the Sony a7x cameras. I've already gotten many excellent photographs from them that I'd have been unlikely to have made using other cameras.











Jim
« Last Edit: July 16, 2015, 07:33:50 pm by Jim Kasson »
Logged

LKaven

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1060
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #116 on: July 16, 2015, 08:00:09 pm »

I'm familiar with that approach. Pushed far enough, it can lead to telephone poles, acne, cigarette butts, and discarded beer cans being called noise. There was a long philosophical discussion on DPR about that. Were you part of it? I encouraged a much more narrow definition of noise. But I don't feel like rehashing the discussion here.

Jim, here the teleological theory of noise came to your defense. 

Since you mentioned it, what would be wrong with the idea of phone poles, acne, cigarette buts, and discarded beer cans being called "noise"?  They can be considered noise at the appropriate level of functional architecture (human image-making), a level somewhat above the level of architecture you're thinking of (e.g., digital image sensors).

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #117 on: July 16, 2015, 08:03:41 pm »

Jim, here the teleological theory of noise came to your defense. 

Since you mentioned it, what would be wrong with the idea of phone poles, acne, cigarette buts, and discarded beer cans being called "noise"?  They can be considered noise at the appropriate level of functional architecture (human image-making), a level somewhat above the level of architecture you're thinking of (e.g., digital image sensors).

They certainly can. A weed is a plant out of place.

Jim

howgus

  • Newbie
  • *
  • Offline Offline
  • Posts: 4
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #118 on: July 22, 2015, 04:35:35 pm »

Try raising the levels in deep shadows.   You will see artifacts in Sony RAW images.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11310
    • Echophoto
Re: How to convince Sony to do lossless raw on A7RII and others?
« Reply #119 on: July 22, 2015, 04:40:46 pm »

Nope, that is BS. My Sony Alpha 99 runs circles around my P45+ back that has 16 bits data and no compression.

If you make a statement, please share a raw image showing it.

Best regards
Erik

Ps. Sony cameras may switch to 12-bit output on many obscure settings. That could cause some artefacts, may be…


Try raising the levels in deep shadows.   You will see artifacts in Sony RAW images.
Logged
Erik Kaffehr
 
Pages: 1 ... 4 5 [6] 7 8 9   Go Up