Luminous Landscape Forum
Site & Board Matters => About This Site => Topic started by: NikosR on February 03, 2009, 06:41:09 am
-
This is a sensible write up. I cannot help but agree with most of the basic ideas underlined in this article. The DxO MFDB issue is a challenge since one needs to consider what is being measured. If we agree that MFDB raw is half baked, aren't we doing the dSLR camera injustice by considering theirs 'baked enough'? After all, neither raw file is the final 'viewable' result. Where does one draw the line? DxO draws it at the raw level, they do produce a levelled comparison and one is free to interpret their data at will.
If you're comparing raw files (in an attempt to disengage from the intricacies of raw development) why would the DxO comparison be invalid or unfair to the MFDBs? On the other hand, indeed it sounds like it is...since they are intended to be used with their dedicated raw converters... but who says Nikons (for example) are not intended to be used with NX? Maybe it would have been best for DxO to refrain from getting in that mousetrap in the first place.
On the other hand, I think I partially disagree with Michael's assertion about DxO not taking resolution into account. In some sense they do, in some sense they don't.
They do, because they publish results normalised for a standard 'print' size of 8x10. In this respect increased resolution results in an increase of their measured metrics (as they explain in their site).
On the other hand, they don't consider the increased USEABILITY of the larger resolution files (e.g. being able to print larger). In this respect they fail, in a similar way as not taking the price into consideration. But for both of these items it is hard (impossible?) to devise objective measurements on the raw data or even just a meaningful coefficient or weighting factor to apply to the final results. We are in the realm of subjective criteria and intended use which cannot be objectively measured.
I believe, and I think Michael agrees with me, that both objective measurements and subjective evaluations are appropriate and complimentary. After all, even in the High End audio world, measurements are still being used which often correlate with the subjective impression,s as any reader of the standard publications like Stereophile will be able to attest to, although measurements have progressed a lot from the time of simple S/N, Total Harmonic Distortion and turntable Wow & Flutter measurements.
-
I have to agree with Michael's assessment of the significance of rating camera sensors with DxO. I've always believed that the proof was in the print. (no pun intended) The DxO thing is an interesting point of reference, but I've never seen any correlation between a DxO mark rating of a camera and how a photograph appeals to the eye. The very things that make a photograph interesting are the things that can't be measured. Normalizing resolution to a 8x10 print is pointless, at least for me. I only make 8x10 prints for family members who want snapshots I took. Just about any camera will look like just about any other camera at 8x10, so what's the point of the exercise? I give Michael credit for at least taking a stance on the issue. He's actually got a fairly good "eye" for judging the significance of things. I think so anyway.
This is a sensible write up. I cannot but agree with the notions underlined in this article. The DxO MFDB issue is a challenge since one needs to consider what is being measured. If we agree that MFDB raw is half baked, aren't we doing the dSLR camera injustice by considering theirs 'baked enough'? After all, neither raw file is the final 'viewable' result.
If you're comparing raw files (in an attempt to disengage from the intricacies of raw development) why would the DxO comparison be invalid? On the other hand, indeed it sounds like it is... Maybe it would have been best for DxO to refrain from getting in that mousetrap in the first place.
On the other hand, I think I partially disagree with Michael's assertion about DxO not taking resolution into account. In some sense they do, in some sense they don't.
They do, because they publish results normalised for a standard 'print' size of 8x10. In this respect increased resolution results in an increase of their measured metrics (as they explain in their site).
On the other hand, they don't consider the increased USEABILITY of the larger resolution files (e.g. being able to print larger). In this respect they fail, in a similar way as not taking the price into consideration. But for both of these items it is hard (impossible?) to devise objective measurements on the raw data or even just a meaningful coefficient to apply to the final results. We are in the realm of subjective criteria and intended use which cannot be objectively measured.
I believe, and I think Michael agrees with me, that both objective measurements and subjective evaluations are appropiate and complimentary. After all, even in the High End audio world, measurements are still being used which often correlate wuth the subjective impression,s as any reader of the standard publications like Stereophile will be able to attest to, although measurements have progressed a lot from the time of simple S/N and Wow & Flutter measuremants.
-
It might be useful to quote here a portion of the DxO press release regarding the availability of their MFDB measurements. I only quote this because I think it provides some food for discussion and some counterarguments to some of Michael's thoughts:
'How is it possible to make valid comparisons between cameras with widely disparate sensor sizes? It's possible by reviewing the cameras' rankings for each of the three separate metrics that make up the DxOMark Sensor scale (Color Depth, Dynamic Range, and Low-Light ISO).
For example, if medium-format cameras do not receive top marks on the overall DxOMark Sensor scale because of their inherent Low-Light ISO limitations, DxO Labs has found these models' Color Depth and Dynamic Range performance to be very striking when compared to high-end DSLRs. Also, despite the clear challenge from DSLRs across all DxOMark sensor metrics, medium-format cameras still lead the way for large-print photography because of their very high resolution performance capability.'
(ΒΤW I don't think that even Michael would argue with the notion that most MFDB's do suffer comparatively at high ISO, even after their output is 'baked' in the proprietary or 3rd party raw converters.)
PS. I do believe that DxO measurements CAN be largely trusted and DO relay much of what can be exposed by subjectively evaluating the cameras, though one really HAS to understand what is being measured and how it is presented.
I believe the biggest problem in the DxO Mark testing is the inclusion of their DxO Mark SCORE (a marketing invention, no doubt). This is what confuses people, since it is produced by quite arbitrarily weighting the different measurement results and this is what often keeps people from really delving deeper into the DxO Mark measurements.
-
Nikos,
DxO's argument about MF backs and high ISO does not address the issue of reporting on raw data that has not been noise reduced on one type of camera and yet doing so on another. The playing field isn't level.
Michael
-
Nikos,
DxO's argument about MF backs and high ISO does not address the issue of reporting on raw data that has not been noise reduced on one type of camera and yet doing so on another. The playing field isn't level.
Michael
Michael,
I'm not sure I'm getting your point. Failure of MFDB manufacturers to optimise their S/N ratio by using various methods in both the analog and the digital domain, both in HW and in Firmware (assuming that dSLR manufacturers do indeed perform digital noise reduction prior to committing to raw) and relying only at digital level noise processing at post, cannot surely be discounted as 'it's just the way they work'. Post processing noise reduction can be also applied to the dSLR raw output skewing the viewable results as we all know.
Where do you draw the line and why?
-
Michael,
I'm not sure I'm getting your point. Failure of MFDB manufacturers to optimise their S/N ratio by using various methods in both the analog and the digital domain, both in HW and in Firmware (assuming that dSLR manufacturers do indeed perform digital noise reduction prior to committing to raw) and relying only at digital level noise processing at post, cannot surely be discounted as 'it's just the way they work'. Post processing noise reduction can be also applied to the dSLR raw output skewing the viewable results as we all know.
Where do you draw the line and why?
The issue is which sensor can get the ultimate best quality - however achieved (We are not talking about in camera JPEG here). MF backs take a different approach to Nikon which also takes a different approach to Canon. DxO's measurements could favour a particular approach (and certainly favour low resolution/density chips like the D3 since they are not concerned by resolution but are concerned with noise). It would be interestiog to consider what a sensor designer would do to artiifcially maximise his DxO score. I am sure the scientists at Canon and Sony coiuld come up with a super high score on a chip that delivered horrible pictures (THD = 0.001%)
Andrew
-
You call it a "failure" on the part of MFB makers.
I'm not an engineer, but I have spent quite a bit of time talking with the guys that design the circuit boards, write the firmware and also the software, and if you were sitting with them at the pub over a beer and described their work as a "failure" I think you'd get a bit of an argument. They have solid engineering reasons for doing what they do, just as Canon, Nikon et al have for their approaches.
The point is really simple. DxOMark's tests are done on raw data from the camera, therefore any maker that choses to do their noise normalization in software rather than in firmware is penalized, just as are readers who are unaware of this fact.
Michael
-
You call it a "failure" on the part of MFB makers.
I'm not an engineer, but I have spent quite a bit of time talking with the guys that design the circuit boards, write the firmware and also the software, and if you were sitting with them at the pub over a beer and described their work as a "failure" I think you'd get a bit of an argument. They have solid engineering reasons for doing what they do, just as Canon, Nikon et al have for their approaches.
The point is really simple. DxOMark's tests are done on raw data from the camera, therefore any maker that choses to do their noise normalization in software rather than in firmware is penalized, just as are readers who are unaware of this fact.
Michael
Sorry, but if your data is inherently comparatively noisy it is a failure in my book. Your argument above assumes all S/N optimisation takes place in the digital domain, which is NOT true and also assumes that dSLR manufacturers do perform noise reduction processing in the digital domain before commiting to raw which AFAIK is not proven and even if its true, good for them.
If I take your argument to its natural limit it would mean that testing or comparing raw data IS NOT a valid point of comparison for ANY digital camera. Fair enough, we are at the mercy of the raw converters then be them proprietary or 3rd party, which is opening a whole new can of worms as you very well know.
-
Yes, they indeed do do noise reduction in the digital domain, and yes, indeed good for them.
But the cake ingredients and the baked cake are not the same thing. As for me, I prefer my cakes baked, which is an analogous way of saying – it's what one sees that matters, the rest is just baking – to mangle a metaphor.
Michael
-
Yes, they indeed do do noise reduction in the digital domain, and yes, indeed good for them.
But the cake ingredients and the baked cake are not the same thing. As for me, I prefer my cakes baked, which is an analogous way of saying – it's what one sees that matters, the rest is just baking – to mangle a metaphor.
Michael
I don't subscribe to this point of view. I like to understand WHY I'm seeing what (I think) I'm seeing and WHY I'm not seeing what I should be seeing. Subjective evaluation and technical testing go hand in hand otherwise we're talking about magic.
Sure, technical measurements might not cater for what one is seeing but in this case questioning and investigation should proceed concurrently in two ways:
1. Make sure I'm seeing well (i.e. am I looking into the right things, am I not masking differences by introducing other variables when the differences would be obvious if I was looking at something else etc. etc.)
2. Check that I'm measuring the right things in the right way and if not try to improve on what and how I'm measuring.
Both of the above are equally important. After all, it's obvious that the sun is revolving around the earth isn't?
-
I don't subscribe to this point of view. I like to understand WHY I'm seeing what (I think) I'm seeing and WHY I'm not seeing what I should be seeing. Subjective evaluation and technical testing go hand in hand otherwise we're talking about magic.
Sure, technical measurements might not cater for what one is seeing but in this case questioning and investigation should proceed concurrently in two ways:
1. Make sure I'm seeing well (i.e. am I looking into the right things, am I not masking differences by introducing other variables when the differences would be obvious if I was looking at something else etc. etc.)
2. Check that I'm measuring the right things in the right way and if not try to improve on what and how I'm measuring.
Both of the above are equally important. After all, it's obvious that the sun is revolving around the earth isn't?
I like to know why as well, but the point is that from capture to output, there are many variables in between that make drastic differences. It seems out of sorts when someone points to reviews/numbers to show why their A900/D3x/5Dii is the best thing out there, and then they go off and use Lightroom to process. RAW conversion itself is so nuanced that it deserves it's own large section in any technical review of a camera.
-
Beyond Color Depth, Dybanic Range and Low-Light ISO, not only Resolution should be taken into account, but also Individual Taste and Cultural Habit when the quality of various digital camera's is being compared.
And what is more, how does the printer perform? Michael referred back to the audio world. Well, not only the amplifier's specs made the quality of the sound but also the speakers.
The evaluation has to be based on measurable data together with unmeasurable factors. As long as this is reckognised, we should be glad to have the high quality DxO data as a sound basis to start our evaluation from. Furthermore, the measurable data are a reference point for camera manufacturers.
The proof of the pudding is in the eating. It may look delicious, but how does it taste?
-
The MFDB backs do not produce jpegs, therefore since Raw conversion is mandatory and noise reduction can be done here, there is no reason for on chip noise reduction. If the DSLRs didn't produce jpegs I wonder if they would bother with noise reduction on chip.
-
DxO's argument about MF backs and high ISO does not address the issue of reporting on raw data that has not been noise reduced on one type of camera and yet doing so on another
If the DSLRs didn't produce jpegs I wonder if they would bother with noise reduction on chip
The only DSLRs applying noise reduction to the raw data are some Sonys. I don't believe DxO have used Sony NRed raw files for anything. The following captures show the effect of Sony NR on the noisiest channel in the sample, namely the red (I don't know which level of NR was selected, I guess it was High).
No other DSLRs are performing noise reduction on the raw data, except single pixel related cleansing, which has nothing to do with the generally accepted meaning of "noise reduction".
(http://www.panopeeper.com/Demo/SonyA900_ISO1600_NROff_Red.GIF)
(http://www.panopeeper.com/Demo/SonyA900_ISO1600_NROn_Red.GIF)
-
The MFDB backs do not produce jpegs, therefore since Raw conversion is mandatory and noise reduction can be done here, there is no reason for on chip noise reduction. If the DSLRs didn't produce jpegs I wonder if they would bother with noise reduction on chip.
I think that both Gabor and I have responded to this argument, put forward by Michael in his essay, quite satisfactorily in various threads. In a nutshell, it sounds very much like a a red herring argument.
-
In some other threads the analogy between the DxO testing of sensors and HiFi testing from years ago was pointed out. In there people were "hearing" stuff differently from what the measurements showed, which in turn led to better measurements (don't know if they ever fully aligned)
So don't forget that DxO with these measurements and scores is only a few month old and the fact people are "seeing" stuff differentlty from what the measurements show is for me not surprising at all. Let's reward and encourage DxO with positive input for going out there in the first place, rather than bash them for not getting it fully right (according to some). I'm sure as time goes on these measurements and alignment with what we see will improve.
Before DxO the only comparisons that were mostly available were subjective noise comparisons on in camera jpg's in photo magazines. From my perspective DxO is a huge leap forward.
Also for me in the end the sensor is only one aspect of a camera, it's also range of available lenses (both in your own closet and as well as the ones on the market), ergonomics, weight/size, available accessories, etc etc.
So to answer the question "eyes or numbers" it's still eyes for me but the numbers are pretty darn usefull as well, imperfect as they may be.
-
This disparity, disconnect, or whatever you want to call it, between DXO measurements and and visual experience, should be demonstrated so we can all understand, and see for ourselves, what the issue is about, precisely.
Michael has already demonstrated that on a particular type of subject, at an A3+ size, there's virtually no discernible difference between a G10 and a P45+, despite the fact that the DXO results imply that there should be a difference.
Michael now seems to claim something along the lines of the opposite. That despite the DXO test results indicating that the D3X is very slightly better than, or the same as, the P45+ at the normalised size of 8x12", the eyes tell a different story, that the P45+ is clearly better, and presumably not just in terms of resolution which we already understand has to be better.
I can't dispute this unless I see some comparisons along the lines of the G10/P45 comparison. Seeing is believing.
C'mon Michael. Back up your assertions with some sample comparisons .
-
This disparity, disconnect, or whatever you want to call it, between DXO measurements and and visual experience, should be demonstrated so we can all understand, and see for ourselves, what the issue is about, precisely.
Michael has already demonstrated that on a particular type of subject, at an A3+ size, there's virtually no discernible difference between a G10 and a P45+, despite the fact that the DXO results imply that there should be a difference.
Michael now seems to claim something along the lines of the opposite. That despite the DXO test results indicating that the D3X is very slightly better than, or the same as, the P45+ at the normalised size of 8x12", the eyes tell a different story, that the P45+ is clearly better, and presumably not just in terms of resolution which we already understand has to be better.
I can't dispute this unless I see some comparisons along the lines of the G10/P45 comparison. Seeing is believing.
C'mon Michael. Back up your assertions with some sample comparisons .
I've read a lot of these threads about subjective or objective, apple and oranges, DxO this and dpi that yada yada yada...
No-one, and I mean no-one, has ever actually come up with a comparison that is a true level playing field both technically and aesthetically. This is because such a beast does not exist.
The only way you can come close is to very tightly define the task for which the camera is to be used - it is merely a tool at the end of the day. Only then can you attempt to make a meaningful judgement. For instance a D3X is 'better' than a P45 on a Cambo Wide at sport and wildlife (but again only if we are talking about hand held, long lens, high shutter speed work). It's not a better camera, it's just better at that specific task.
This is why DxO testing and rating can never be broadly meaningful - all it does is rate camera sensors by it's own narrow set of rules which may or may not be relevant to the real world. By it's own rules the D3X is indeed better than a P45 back. I could make the argument that my old 400D is clearly superior to a 1DsMk3 - if I use price and weight as my criteria.
Even print making is suspect as a criteria - size does matter. I'd venture to suggest that a 6x4 print from a high street lab would look much the same from a D3X or a P45 - although forn low light work the D3 would clearly 'win'. So if your test criteria is 6x4 prints then the D3 wins. OTOH if we set the print criteria at 30x40 then the balance shifts. The P45 will look superior generally, except again for low light or action work.
There are too many variables but at the very least I'd like to see the task, or intended end use, included in any discussions about camera superiority - without it any arguments are pointless.
-
I have been strongly against raw noise reduction, there was a major issue with this on the A700 (which was addressed via firmware update)
Though if you apply high ISO NR on the sony's, it does effect the raw data (why they do this, is unknown, should be just jpeg really) Off does not appear to have any influence on the raw.
Back on topic, hard to disagree with anything in the article, logical and sensible..I never went on numbers myself, so yes..the real world is more important. I take the numbers with a pinch of salt..so should most
-
In some other threads the analogy between the DxO testing of sensors and HiFi testing from years ago was pointed out. In there people were "hearing" stuff differently from what the measurements showed, which in turn led to better measurements (don't know if they ever fully aligned)
Yet, it remains to be shown how exactly their results differ from what one sees...
Other than the fact that a P65+ A2 print looks nicer than the same print from an A900/D3x, I have not read any convincing example from Michael to explain what exactly he means by this supposed gap between perception and numbers.
If anything, he commented in a recent thread that most backs and DSLRs have the same 12 stop DR in real world applications, which is basically exactly what DxO is saying also.
Is the disconnect real?
Cheers,
Bernard
-
Is the disconnect real?
Cheers,
Bernard
Maybe
Actually I think that a master printer / digital guru could get a better print out of a D3X file than a lesser being with a P45. Up to say 20x16, maybe 20x30. if you want to go bigger then I think the P45 will 'overtake' the D3.
My opinion is that almost all modern sensors in mid and top line cameras are pretty similar in their tonal range and a well made print within the native res of the original file will look excellent. I think you'd be really hard pushed to pick a P45, P65, D3X, a D3, a 5D, a 5DMk2 etc if you were shown prints from each at A3 size (420x297mm).
-
Maybe
Actually I think that a master printer / digital guru could get a better print out of a D3X file than a lesser being with a P45. Up to say 20x16, maybe 20x30. if you want to go bigger then I think the P45 will 'overtake' the D3.
My opinion is that almost all modern sensors in mid and top line cameras are pretty similar in their tonal range and a well made print within the native res of the original file will look excellent. I think you'd be really hard pushed to pick a P45, P65, D3X, a D3, a 5D, a 5DMk2 etc if you were shown prints from each at A3 size (420x297mm).
Absolutely, but DxO never claimed to be taking resolution into account.
What you write supports the contention that DxO numbers are actually a good representation of what our eyes see and there is in fact no disconnect.
Cheers,
Bernard
-
According to the manual it won't be effective below ISO1600. According to Sony it was added because DRO induces extra noise. I'm not certain about any of this.
Best regards
Erik
I have been strongly against raw noise reduction, there was a major issue with this on the A700 (which was addressed via firmware update)
Though if you apply high ISO NR on the sony's, it does effect the raw data (why they do this, is unknown, should be just jpeg really) Off does not appear to have any influence on the raw.
Back on topic, hard to disagree with anything in the article, logical and sensible..I never went on numbers myself, so yes..the real world is more important. I take the numbers with a pinch of salt..so should most
-
Absolutely, but DxO never claimed to be taking resolution into account.
What you write supports the contention that DxO numbers are actually a good representation of what our eyes see and there is in fact no disconnect.
Cheers,
Bernard
I don't doubt that they are accurate - it's the significance that I dispute. They are only a good representation of what we see in very general terms, as soon as you apply a 'task' then they can fall down badly.
Hence my assertion that there can be no broadly meaningful tests, only highly specific ones.
-
I don't doubt that they are accurate - it's the significance that I dispute. They are only a good representation of what we see in very general terms, as soon as you apply a 'task' then they can fall down badly.
Hence my assertion that there can be no broadly meaningful tests, only highly specific ones.
That's true indeed.
Cheers,
Bernard
-
A long time lurker I decided to finally register and add my voice.....
I agree with MR's take on the DxO Mark application for a bunch of reasons. In addition to MR's points I suppose my real concern is the assumption that you can combine probably hundreds of engineering specs for a sensor into just three or four measures, take a single data point of those measures and extrapolate that out to create a single number that says (or at least implies) that one sensor will always have better IQ than another in all shooting situations.
Many test sites (and photographers) make the mistake of assuming that performance at the extremes is indicative of performance in the 'core' operating areas when in fact the opposite is very often the case. How often have you read something like the XX lens has better IQ than YY because YY is slightly softer in the corners when wide open while totally ignoring that neither lens is at their best then and that in their respective sweet spots YY actually moves ahead. Similarly with sensors, there are sensors that produce a lot of 'shot' noise but control thermal noise very well, hence they are extremely clean at low isos but fall away fast as iso rises. Others have well controlled shot noise so work well at high isos but have more thermal noise so are actually noisier low down. In this situation how can any single data point seek to define IQ?
So all in all I think we need to be extremely wary of any of these quasi engineering level tests.
But this doesnt mean these tests (whether its DxO Mark or DPReviews infamous 100% crops) are entirely pointless. They are useful as diagnostic tools to try an understand why we see what we see. If you see a photo from xx and yy and you find you prefer yy but cant quite understand why then these 100% crops or DxO data analyses can help. If you are trying to optimise your particular camera's processing then examining these sorts of tests can assist. But really thats about it. These tests can explain what you have seen but cannot tell you what you will see.
-
This is why DxO testing and rating can never be broadly meaningful - all it does is rate camera sensors by it's own narrow set of rules which may or may not be relevant to the real world. By it's own rules the D3X is indeed better than a P45 back. I could make the argument that my old 400D is clearly superior to a 1DsMk3 - if I use price and weight as my criteria.
Nick,
You would be spot on if you were to make the argument that your old 400D is superior to the 1Ds3 in terms of price and weight. Who could argue with that, assuming that a lower price and a lower weight actually is superior, or better, or more desirable?
DXO not only rates the cameras it tests by its own narrow set of rules, but also by the rules and jargon that most of us use when assessing, discussing and arguing about a camera's performance. Issues such as resolution are largely a matter of pixel count and are also greatly influenced by choice of lens, so I can't see much point in their testing the sensor for resolution. Cameras without an AA filter seem to have a marginal resolution advantage at the expense sometimes of additional moire.
DXO tests for signal-to-noise, dynamic range, tonal range and color sensitivity at various ISOs. They produce two sets of results; one at a pixel for pixel view on the screen, and the other at a downsampled size of 8x12" at 300 ppi, which represents an 8mp image.
What else should they be testing? My personal view is that they should have two sets of normalized images, 8x12" and 16x24". But maybe that would introduce additional problems regarding the manner of interpolation.
If DXO is failing to test for certain important factors that influence image quality, at the sensor level, then what are they? We all know that the final print can be influenced by a whole range of post processing factors, including the choice of raw converter and the operator's skill in Photoshop.
One can hardly expect DXO to objectively review the performance of converters since they are in the business of producing their own raw converter.
-
How often have you read something like the XX lens has better IQ than YY because YY is slightly softer in the corners when wide open while totally ignoring that neither lens is at their best then and that in their respective sweet spots YY actually moves ahead.
Hardly ever. All good lens reviews test lens sharpness at the centre and at the borders, at most apertures, or at least at maximum aperture and at F8. Good resolution in the centre is generally more highly valued. When rating lens performance with a single figure, a weighted rating will be applied that gives higher points for better centre performance and a smaller penalty for poorer edge performance.
-
I am going to make a point here, made more than a few times before.
I see no "quality" variable for some tests, esp thing like DR. It's all very well bashing out a 12.5 stops number, but if the image is unprintable, then it's pointless statistics, v of real world use. That is just one example, of why I would not take them too seriously. Of course the quality aspect is open to debate too, hence the never ending problem.
I can see some point to tests for cameras, and the technology improves etc, but I wouldn't want to nail hard numbers down on them.
The good news is that nobody has yet, made a software solution for "measuring a great photograph" It could provide statistics on composition, exposure. colour balance, noise, sharpness..and provide an in depth analysis, even offer some suggestions as to how to improve the shot. That way we could simply batch run our photos through it, and it could auto discard the bad ones, and rank the good ones. Wow..now I gave somebody an idea ;-) lol.
Sometimes, you just need to trust your eyes..and use that as the main influence
-
There are nothing wrong with DxoMark measurements, but rather interpretation. There is so many misinterpretations around forums. To understand what exactly mean each number you should study a DxoMark measurement data technology.
May be I’m wrong, but DxoMark technologies is made (and described) by physics/engineers, not photographers. They manly studied photosite physical characteristics, but not whole sensor as a tool for capturing. For example for photographers much more important is to know sensor resolution changes at different ISOs than error of photosite response to light (SNR18%). Furthermore for me it is much more interesting to know whole sensor dynamic range, than only in shadow.
-
Being involved in high end audio on a professional basis, all i can say that measurements as commonly shown in brochures and reviews have little if any bearing on the sound quality delivered. For the electronics, it is simple, the more feedback the lower the distortion. BUT, no one has been able to prove that is then sounds better, on the contrary. The best sounding electronis are still those that are already well designed and wel performing without feedback. In other words, how the measured result is achieved (what is in the black box) is not shown directly in the measurements, but is fundamental in the perceived performance. None of the measurements typically used in audio have any bearing on how human hearing perceives sound.
When the audio world went from analog to digital, initally the quality of the digital processed sound was basically very bad. Perceived as harsh, flat (no 3-dimensionality), lifeless, unbalanced, definitely not engaging.
One of the marketing themes was cheap reproduction electronics, no need for very expensive turntables et to get high quality. Yet within a few years the best sounding CD-players were as expensive, horrendously expensive, as the best turntables and cartridge combo's.
It actually took over 20 years to be able to fully deliver to the red-book standard (also known as the CD). But commercially it was the only viable option.
Actually already in the analog area quality was compromised to get more minutes of sound on an LP, and to make it more easy on the electronics.
I was once confronted by music makers that their standard was that it should sound great on a ghetto blaster, this was about CD's.
Adding some amount of distortion is a common practice to emphasise a particular sound in a recording.
At the end of the day a good music reproduction on a system with a good source (LP, CD, SA-CD, etc) is perseiced as emotionally engaging, 3 dimensional, and even after playing it several times a continuous discovery of new facets in the reproduced audio "image" or picture if that is a better suiting word.
Bringing this to the picture capturing and reproduction, in other words photography. There is a strong parallel.
Putting reproducable measurements and results does not give insight in the perceived picture quality, bears little relation with the visual perception, but becomes a major element in selecting a piece of equipment. Commercially it is very hard to stay in the analog solution.
However it is not just the digital part of the total chain. With some friends and aquintances we once compared pictures, using same slid film make and type, same scene, but with a Leica M and with a Canon A1 with FD lenses. Slides were put in a projector tray in random order and projected. A 100% selection of canon vs leica by all persons present. Not on color, sharpness etc. Simply on 3-dimensionality and engagement. Yet the Canon lenses measured in reviews equal or better.
Wrapping it ll up, the digital photogaphic technology is still not there, numbers are just one of the elements subjective evaluation is still the only significant element in the evaluation, and will probably remain as such, but only to those that dare to rely on it.
Jan R. Smit
-
This is a sensible write up. I cannot help but agree with most of the basic ideas underlined in this article. The DxO MFDB issue is a challenge since one needs to consider what is being measured. If we agree that MFDB raw is half baked, aren't we doing the dSLR camera injustice by considering theirs 'baked enough'? After all, neither raw file is the final 'viewable' result. Where does one draw the line? DxO draws it at the raw level, they do produce a levelled comparison and one is free to interpret their data at will.
If you're comparing raw files (in an attempt to disengage from the intricacies of raw development) why would the DxO comparison be invalid or unfair to the MFDBs? On the other hand, indeed it sounds like it is...since they are intended to be used with their dedicated raw converters... but who says Nikons (for example) are not intended to be used with NX? Maybe it would have been best for DxO to refrain from getting in that mousetrap in the first place.
On the other hand, I think I partially disagree with Michael's assertion about DxO not taking resolution into account. In some sense they do, in some sense they don't.
They do, because they publish results normalised for a standard 'print' size of 8x10. In this respect increased resolution results in an increase of their measured metrics (as they explain in their site).
On the other hand, they don't consider the increased USEABILITY of the larger resolution files (e.g. being able to print larger). In this respect they fail, in a similar way as not taking the price into consideration. But for both of these items it is hard (impossible?) to devise objective measurements on the raw data or even just a meaningful coefficient or weighting factor to apply to the final results. We are in the realm of subjective criteria and intended use which cannot be objectively measured.
I believe, and I think Michael agrees with me, that both objective measurements and subjective evaluations are appropriate and complimentary. After all, even in the High End audio world, measurements are still being used which often correlate with the subjective impression,s as any reader of the standard publications like Stereophile will be able to attest to, although measurements have progressed a lot from the time of simple S/N, Total Harmonic Distortion and turntable Wow & Flutter measurements.
The problem is they sort of translate that to a standard print size as a normalisation of performance.
I have made quite recently pictures of my mother, with a Nikon D3, with a Panasonig TZ3 and a Leica M. The Leica picture is Fuji provia, scanned in a very modified Nikon LS50-ED scanner at max resolution. I use PWP (Picture Window Pro) to print all three on a 4x6 size, with significant difference in perception of the picture shown, the scanned leica beating the nikon, the nikon beating the panasonic.
Where then does this fit into the DxO approach?
-
I have been strongly against raw noise reduction, there was a major issue with this on the A700 (which was addressed via firmware update)
Though if you apply high ISO NR on the sony's, it does effect the raw data (why they do this, is unknown, should be just jpeg really) Off does not appear to have any influence on the raw
As the captures I posted above demonstrate this, NR Off is now effective on the raw data.
-
Furthermore for me it is much more interesting to know whole sensor dynamic range, than only in shadow.
You might want to do a bit of reading on dynamic range... highlights saturation point is always the start point for DR computation so DxO's DR measurement is not just looking at shadows.
Cheers,
Bernard
-
(michael @ Feb 3 2009, 04:02 PM)
The point is really simple. DxOMark's tests are done on raw data from the camera, therefore any maker that choses to do their noise normalization in software rather than in firmware is penalized, just as are readers who are unaware of this fact.
(NikosR @ Feb 3 2009, 07:18 AM)
Sorry, but if your data is inherently comparatively noisy it is a failure in my book. Your argument above assumes all S/N optimisation takes place in the digital domain, which is NOT true and also assumes that dSLR manufacturers do perform noise reduction processing in the digital domain before commiting to raw which AFAIK is not proven and even if its true, good for them.
Let's consider a very specific example of pixel data processing which improves sensor response ("noise" and hence dynamic range) without harming resolution.
[blockquote]1) Take a calibration image with no light and examine the response of every pixel. You will probably see some systematic pixel-to-pixel offset variability. Correct all subsequent images by subtracting the measured offset from every pixel. (Add back a few counts of constant offset if you do not want to clip blacks at the sensor noise level.)
2) Take a calibration image of a uniform illumination source and examine the response of every pixel. You will probably see some systematic pixel-to-pixel gain variability. Correct all subsequent images by normalizing every pixel to its measured gain. [/blockquote]These calibrations could be performed in camera, before the raw file is written, or they could be performed later, during processing of the raw file. An image which has these calibrations applied at any stage should be given credit for the resulting noise reduction.
I have no idea whether any camera system currently performs these particular calibrations. If something like this is being done, at different processing stages in different systems, then Michael's point is completely valid.
-
You might want to do a bit of reading on dynamic range... highlights saturation point is always the start point for DR computation so DxO's DR measurement is not just looking at shadows.
This is a recurring issue (probably a leftover ingrained from the film era). When I am saying that I need far underexposed images for the DR measurement and I am not interested in seeing any high DR scenery, then invariably some readers become incredulous.
-
These calibrations could be performed in camera, before the raw file is written, or they could be performed later, during processing of the raw file. An image which has these calibrations applied at any stage should be given credit for the resulting noise reduction.
I have no idea whether any camera system currently performs these particular calibrations. If something like this is being done, at different processing stages in different systems, then Michael's point is completely valid.
LOL, this is good. The only system I know of doing this is MFDB (part of the raw pre-processing out of camera).
Most DSLRs offer long exposure noise reduction, which is a dark frame substraction, but it works usually only from 1sec upward, and it's effect is questionable compared to "manual" dark frame substraction.
-
This is a recurring issue (probably a leftover ingrained from the film era). When I am saying that I need far underexposed images for the DR measurement and I am not interested in seeing any high DR scenery, then invariably some readers become incredulous.
It would seem that some people around, including very high end gear owners, believe that there is some magic analog quality to raw files that make highlights behave intependantly from the strict digital world where 255,255,255 is pure white.
One aspect where highlights mesurements might make sense in DR computation though is if sensors don't really believe linearly near clipping, which I have been suspecting for quite some time.
Cheers,
Bernard
-
It would seem that some people around, including very high end gear owners, believe that there is some magic analog quality to raw files that make highlights behave intependantly from the strict digital world where 255,255,255 is pure white.
Yes. I've been thinking of marketing a specially modified version of PS, perhaps called Ph*t*Sh*pPlus, in which all of the numeric scales run from 0 to 300 instead of just to 255. I'll bet I could sell quite a few!
-
One aspect where highlights mesurements might make sense in DR computation though is if sensors don't really believe linearly near clipping, which I have been suspecting for quite some time.
1. The "pure" pixel values are not always linear, but that is in the entire range, with the greatest effect in the very shadows. The black frame shots I asked for would be partly for the purpose of finding out how much this is.
2. I don't know of any case, where the values of individual pixels are not linear specifically at the high end. However, there are cases, particularly earlier Nikons (like the D200) with pixels, which do not saturate at the same level (same Canons too are exhibiting this, but within a narrower range). There is a demonstration of this phenomenon in http://www.cryptobola.com/PhotoBola/Exposure.htm (http://www.cryptobola.com/PhotoBola/Exposure.htm)
As current raw processors do not treate the pixels with different saturation levels (at least I don't know of that), this may cause an apparent nonlinearity.
-
Yet, it remains to be shown how exactly their results differ from what one sees...
Other than the fact that a P65+ A2 print looks nicer than the same print from an A900/D3x, I have not read any convincing example from Michael to explain what exactly he means by this supposed gap between perception and numbers.
If anything, he commented in a recent thread that most backs and DSLRs have the same 12 stop DR in real world applications, which is basically exactly what DxO is saying also.
Is the disconnect real?
Cheers,
Bernard
In the past i have been professionally and personally heavely involved in the HiFi Audio world, both as a consumer and as a lead engineer in designing and producing loudspeakers and amplifiers. I see a strong parallel with the HiFi Image world, and it frightens me. As John Atkinson, the well known and highly regarded editor of Stereophile once put it, that with his many many years of testing and listening to hifi-audio-equipment he still was not able to find consistent correlation between measurements and listening results.
Like Michael said several postings up, his interest is the baked result, and lesser the ingredients. I once designed a phono preamplifier, duing design i had two different circuit topologies, but with the same individual components. Both topo's measured identical but sounded quite different. The best one was declared in a Festival du Son in Paris as one of the top-three in the world. So to take the analogy of baking a bit further, the ingredients do matter, but how it is combined to the end-result is fundamental in the outcome, this is the baking recipe including all parameters.
The scary thing that frightens me is the possibility in electronics to make the measured result look good, but achieved by just fiddling around inside the "black box". Sound engineering principles do allow for that unfortunately.
Regards,
Jan R.
-
Perhaps the problem lie in the measurement principles. Yes the extremes are black and white, yes the tone curve from black to white matters. But what about the smallest change in luminance, in color change that can be faithfully reproduced? What about how the system performance degradation in reproducing linepairs should be versus how a given system actually degrades. I have a Nikon Coolscan LS50-ED, that had a ghosting problem. The measurements of Nikon Professional Services did not show this problem, yet the scanned pictures did. To cut a long sotryshort, after removal of the protective window on the sensor and treating the chamber between the optics and the sensor with a very strong lightabsorbent material, not only the ghosting was gone, THe scanresults now show more detail, and more "alive". Yet the black is till black the white is still white, the curves when looking at the calibration profile of the same calib-slide vary little if any. So in between the extremes something has changed for the better, as if a gray veil has lifted (or in HiFi Audio teminology, clarity improved, micro-contrast repoduced more faithfully, etc, etc), but measurements do not show this.
Regards,
Jan
-
Perhaps the problem lie in the measurement principles. Yes the extremes are black and white, yes the tone curve from black to white matters. But what about the smallest change in luminance, in color change that can be faithfully reproduced? What about how the system performance degradation in reproducing linepairs should be versus how a given system actually degrades. I have a Nikon Coolscan LS50-ED, that had a ghosting problem. The measurements of Nikon Professional Services did not show this problem, yet the scanned pictures did. To cut a long sotryshort, after removal of the protective window on the sensor and treating the chamber between the optics and the sensor with a very strong lightabsorbent material, not only the ghosting was gone, THe scanresults now show more detail, and more "alive". Yet the black is till black the white is still white, the curves when looking at the calibration profile of the same calib-slide vary little if any. So in between the extremes something has changed for the better, as if a gray veil has lifted (or in HiFi Audio teminology, clarity improved, micro-contrast repoduced more faithfully, etc, etc), but measurements do not show this.
You might have a point there, somewhere . If there's evidence out there that the DXO analytical method is failing to take account of certain, significant, observable differences in image quality, differences and qualities that are not reflected in the DXO measurements, then the DXO team would surely be the first to be concerned. This is their business.
But so far, in all the heat of the discussions on this issue in various threads, I've never seen a single comparison between any two cameras which contradict the DXO results.
Michael produced a comparison between the P45+ and the Canon Powershot G10 P&S, in which he demonstrated that at an A3+ size (13"x19") there was no discernible difference between the images from the two cameras, apart from the shallower DoF of the P45+.
However, the DXO results in respect of noise, dynamic range, tonal range and color sensitivity, at a normalised print size of 8"x12", indicate there is a substantail difference between these two cameras. Is this a contradiction or a disconnect?
By substantial, I mean 4 stops greater SNR, for example. At base ISO, the P45+ has over 4 stops (12.2dB) lower SNR than the Powershot G10, at an 8x12" print size, and 3 stops greater dynamic range. Is there anyone who would dispute this? Would proud owners of a P45+ back like to stick up their hand and declare that these figures are nonsense?
If you do want to stick up your hand, would you say that the SNR of the P45+ should be more than 4 stops greater, at base ISO, and the DR more than 3 stops greater, than the G10?
If you do think that this is the case, that DXO are underestimating the SNR, DR, Tonal Range and Color Sensitivity of the P45+, how do you explain that a group of experienced photographers were unable to distinguish between Michael's A3+ size prints from the G10 and P45+?
Do you think it might have something to do with the characteristics of the subject that was photographed?
-
The objects photographed can have something to do with the observation Michael made.
Also DxO measures at the "raw point" in the total production-reproduction chain, whereas Michael measured at the "end point" of the chain: interpretation of what is visually observed.
Those are two totally different points in the total chain, making comparison difficult if not impossible.
Looking at individual parameters: Again comparing to the HiFi Audio, for instance SNR as an absolute number has little meaning on its own, unless it is absolutely not noticed, ie well under the reception threshold of the human being (unfortunately , to my knowledge we do not know this threshold, nor can we achieve it apparently)
But when received/observed/noticed, how noise is perceived is highly dependent on two main parameters, its noise profile(how it sounds) and whether or not it is somehow modulated/influenced by the signal whe are interested in.
If of a smooth/clean profile, and totally independent of the signal, the human processing system filters it out as a constant, effectively reducing the noise level. Several observations on this phenomenon pointed in the direction of up to 10dB or more.
If this is also true for what we see (recorded by our eyes, processed by our brains to something we reckognise) i do not know, but only assume it is, it is the same brains, therefore it is plausible that the same basic principles apply.
It however requires only one component in the total chain to jeopardise the perceived endresult, for instance the noise topic just covered.
This does not imply that i am saying the Michael's reproduction chain is somehow flawed, i would be the last person to make such a statement.
What i am saying is that we need to take the total chain into account and not just one component in isolation.
At the end of the day we only see and aim to enjoy what comes out at the end, the intermediate results are just there to serve the endresult.
Regards Jan R.
-
The objects photographed can have something to do with the observation Michael made.
Also DxO measures at the "raw point" in the total production-reproduction chain, whereas Michael measured at the "end point" of the chain: interpretation of what is visually observed.
Those are two totally different points in the total chain, making comparison difficult if not impossible.
Looking at individual parameters: Again comparing to the HiFi Audio, for instance SNR as an absolute number has little meaning on its own, unless it is absolutely not noticed, ie well under the reception threshold of the human being (unfortunately , to my knowledge we do not know this threshold, nor can we achieve it apparently)
But when received/observed/noticed, how noise is perceived is highly dependent on two main parameters, its noise profile(how it sounds) and whether or not it is somehow modulated/influenced by the signal whe are interested in.
If of a smooth/clean profile, and totally independent of the signal, the human processing system filters it out as a constant, effectively reducing the noise level. Several observations on this phenomenon pointed in the direction of up to 10dB or more.
If this is also true for what we see (recorded by our eyes, processed by our brains to something we reckognise) i do not know, but only assume it is, it is the same brains, therefore it is plausible that the same basic principles apply.
It however requires only one component in the total chain to jeopardise the perceived endresult, for instance the noise topic just covered.
This does not imply that i am saying the Michael's reproduction chain is somehow flawed, i would be the last person to make such a statement.
What i am saying is that we need to take the total chain into account and not just one component in isolation.
At the end of the day we only see and aim to enjoy what comes out at the end, the intermediate results are just there to serve the endresult.
Regards Jan R.
The point I'm trying to make is as follows. Differences in image quality can be lost because the subject photographed is not suitable to display such subtle differences. That's not a fault of the DXO testing methodology.
What would throw suspicion on the DXO results is a comparison between, say, a D3X and a P45+ at an A2 size, which showed an observable, smoother tonality and lower noise in any part of the tonal range. So far, no such comparison has been shown, to my knowledge, so all the protestations and bleating are just hot air.
If someone does dare to show such a comparison, they'd better make sure that both imags are of the identical subject with identical lighting, and that both images have an equally full exposure (ETTR), and equally thorough treatment at the conversion stage and in post processing.
-
The point I'm trying to make is as follows. Differences in image quality can be lost because the subject photographed is not suitable to display such subtle differences. That's not a fault of the DXO testing methodology.
What would throw suspicion on the DXO results is a comparison between, say, a D3X and a P45+ at an A2 size, which showed an observable, smoother tonality and lower noise in any part of the tonal range. So far, no such comparison has been shown, to my knowledge, so all the protestations and bleating are just hot air.
If someone does dare to show such a comparison, they'd better make sure that both imags are of the identical subject with identical lighting, and that both images have an equally full exposure (ETTR), and equally thorough treatment at the conversion stage and in post processing.
Ray,
Taking your criteria to separate hot air from substance, where does that put DxO?
Regards,
Jan R.
-
For the past 30 years, I've been professionally involved in designing signal processing systems for audio, sonar, radar, and telecommunications systems. I've also read way too many statements about the mystical ineffability, the 'magic', the unknowable connection between design/implementation and perceived performance. If your two pre-amps measured "identically" (presumably in frequency response) and performed differently at a level reproducibly perceptible to humans in blind trials, then you didn't do enough measurements. Perhaps it was phase or amplitude non-linearity, or transient ringing effects, etc., but if its a gross enough effect for a human to distinguish, it has a measurable (in some parameter) effect on the signal waveform.
I'm not saying that different designs don't sound different, or that people shouldn't prefer one device's 'interpretation' over another. People can be pretty good at 'different', but we're really terrible at 'better'. Outside controlled blind trials, we're not even very good at reliably detecting differences among similarly performing gadgets. The shameful/successful marketing of $1000 per foot pure crystalline-aligned unobtanium-alloy speaker cables shows how effective the marketing guys are at creating distinctions without differences in the minds of vulnerable, enthusiastic (well-heeled) hobbyists.
Much of what is sold as high-end audio today is quantum snake oil. The tube amplifiers that are so popular today among the 'golden ears' MR refers to sound good in their judgment, but they're (by design) not high-fidelity, if hi-fi is taken to mean faithful reproduction of the audio signal. Tubes are well known for their compressive amplitude transfer characteristics, which in a nice Class A configuration (with little feedback to suppress the harmonic distortion) will produce the sweet 'syrupy' harmonics (distortion) that the golden ears have declared to be High Fidelity (its especially sweetening to acoustic instruments and human voices. I even like it.). Their preference for low-power final amplifiers (or overdriven pre-amp stages) is to insure that the tubes are frequently driven into their compressive performance regime. Its fine that they like it, and that they are willing to pay for it. Defending their 'refined' listening preferences by saying there's no correlation between measurements and quality though, is a rationalization of their preference for harmonically sweetened playback. High-fi it ain't. (I'm sure there's a Kodachrome/Velvia analogy in here somewhere - look at all the pretty colors)
So, while I'm puzzled, along with many others, with the DX0 'resolution independent' quality measures, I can't subscribe to the 'numbers don't matter' school. Badly formulated or mis-applied metrics are common. More common is mis-interpretation of technical measurements by non-specialists. Most common by far, sadly, is over-reliance on, and endless yammering over the the importance of this or that technical parameter in what is, outside of medical and forensic imaging, an artistic endeavor more than a technical one. The numbers, properly formulated and interpreted, can characterize the performance of the instrument, but can't predict the artistic quality of the product.
Regards,
--Tom
In the past i have been professionally and personally heavely involved in the HiFi Audio world, both as a consumer and as a lead engineer in designing and producing loudspeakers and amplifiers. I see a strong parallel with the HiFi Image world, and it frightens me. As John Atkinson, the well known and highly regarded editor of Stereophile once put it, that with his many many years of testing and listening to hifi-audio-equipment he still was not able to find consistent correlation between measurements and listening results.
Like Michael said several postings up, his interest is the baked result, and lesser the ingredients. I once designed a phono preamplifier, duing design i had two different circuit topologies, but with the same individual components. Both topo's measured identical but sounded quite different. The best one was declared in a Festival du Son in Paris as one of the top-three in the world. So to take the analogy of baking a bit further, the ingredients do matter, but how it is combined to the end-result is fundamental in the outcome, this is the baking recipe including all parameters.
The scary thing that frightens me is the possibility in electronics to make the measured result look good, but achieved by just fiddling around inside the "black box". Sound engineering principles do allow for that unfortunately.
Regards,
Jan R.
-
It's not the critical "exactly the same" test that some people want, but I now have about 15 prints from my recent shoot in Antarctica hanging at my gallery. Within the next week I should have about 30. In mid-March I'll have an open house and show. In the meantime if anyone wants to drop by my Toronto gallery when I'm there I'd be happy to show them to you.
What do they show? Most are in the 20" X 28" size range, matted to 28X34". Some are shot with the Sony A900 and some with the Phase One P65+. Can one see a difference at this size? Yes, absolutely – if you know what to look for, and it's not just about resolution. But to anyone except a technically knowledgeable observer these differences don't jump out. They all look pretty terrific.
There are no simple answers.
Michael
-
It's not the critical "exactly the same" test that some people want, but I now have about 15 prints from my recent shoot in Antarctica hanging at my gallery. Within the next week I should have about 30. In mid-March I'll have an open house and show. In the meantime if anyone wants to drop by my Toronto gallery when I'm there I'd be happy to show them to you.
What do they show? Most are in the 20" X 28" size range, matted to 28X34". Some are shot with the Sony A900 and some with the Phase One P65+. Can one see a difference at this size? Yes, absolutely – if you know what to look for, and it's not just about resolution. But to anyone except a technically knowledgeable observer these differences don't jump out. They all look pretty terrific.
There are no simple answers.
Michael
I find that quite reassuring. My position is that any good digital camera with good lenses, good shooting technique and good printing methods will produce excellent results, almost indistiguishable from other cameras as long as they stay within their native resolution .
At 28" the Sony is printing at about 200dpi which is roughly the lower limit of acceptable printing res according to current wisdom. The P65 is well within it's native res of course, and I'm not surprise the prints look mostly as good as each other. It would be only when you went up a print size that the P65 would come into it's own. I absolutely guarantee that a 44" print would look better on the P65 than on any other camera, with the possible exception of the P45. It's just simple math really.
-
At 28" the Sony is printing at about 200dpi which is roughly the lower limit of acceptable printing res according to current wisdom. The P65 is well within it's native res of course, and I'm not surprise the prints look mostly as good as each other. It would be only when you went up a print size that the P65 would come into it's own. I absolutely guarantee that a 44" print would look better on the P65 than on any other camera, with the possible exception of the P45. It's just simple math really.
Yes, of course. The image from the sensor with the greater number of pixels will at some point of enlargement begin to show better resolution. We don't need DXO testing to tell us that.
The issue here is really about the other attributes of image quality; tonality, noise, dynamic range etc. It is surprising that the flagship 35mm DSLR (the D3X), in respect of these other attributes, seems to have caught up with the older generation of DBs which have aprroximately double the sensor area. The P45+ has been around for a while now, and DXO have not tested the P65.
It's surprising to me that the smaller sensor should do so well, because I recently compared my new 50D with my 3 1/2 year old 5D, with respect to noise and tonality. The purpose of the testing was to find out if the 50D had a shutter-speed/image-quality advantage flowing from its inherent ability to produce a greater DoF at a givem F/stop (as a result of its smaller sensor). In other words, if in practice I can use 100th at F4 and ISO 100, with the 50D, hand-held, then in order to get the same DoF at the same shutter speed, I need to use approximately F6.3 and ISO 350 with the 5D.
I wanted to see if the 5D image would be noisier in those circumstances.
In fact, it wasn't, even comparing pixel for pixel on the screen. Both images appeared to have about equal noise. However, comparing both cameras at the same ISO, it was clear than the 5D produced lower noise. The reason for this seems pretty clear. The 5D's sensor is over 2.5x the area of the 50D's sensor. It simply gathers more light at the same ISO for any scene of equal FOV that is correctly exposed.
These results of my own testing carried out before DXOMark existed, are quite consistent with the DXO graphs in respect of noise and tonal range. Despite the 5D being 3 1/2 year old technology, the DXO tests show that the 5D actually does have better SNR and Tonal Range than the smaller-sensor 50D, both 'on screen' and at a normalised size of 8"x12".
-
For the past 30 years, I've been professionally involved in designing signal processing systems for audio, sonar, radar, and telecommunications systems. I've also read way too many statements about the mystical ineffability, the 'magic', the unknowable connection between design/implementation and perceived performance. If your two pre-amps measured "identically" (presumably in frequency response) and performed differently at a level reproducibly perceptible to humans in blind trials, then you didn't do enough measurements. Perhaps it was phase or amplitude non-linearity, or transient ringing effects, etc., but if its a gross enough effect for a human to distinguish, it has a measurable (in some parameter) effect on the signal waveform.
Tom,
Although I'm not an engineer or technician, I tend to agree with your entire post. It makes complete sense to me and accords with my own experience in evaluating hi fi systems for my personal use.
-
Hi!
I'm not familiar with high end audio, but I agree with the snake oil stuff. There are issues with measurements, too. In many cases the measurements are to simple, like the frequency response or total harmonic distorsion thing in audio. I guess that quite many MTF charts are calculated for each lens but only a few curves are published in the popular tests. Most of the images we have are not really in focus, because of the essentially non existing depth of field of today's digital technology (due to resolution) but also because of different failures of focusing technology. Even small focusing errors on the magnitude of microns in the image plane affect MTF. So slightly out of focus imaging may be of greater significance in real photography than optimum MTF.
Many apects like flare and ghosting are not easy to measure objectively. In my humble opinion there are some views which are ignorant or stupid:
1) Trying to convert all data into a single figure of merit (SFOM).
2) If we have a single figure of merit assuming that a product having a slightly higher SFOM is better than one having a slightly lower one.
3) The approach that all measurements are scrap
My opinion is more like:
- Something that measures bad is normally bad, at least if measurement is relevant
- There are things that are hard to measure which may have a high subjective significance
One issue I'd like to point out is that we are obsessed with sharpness and resolution. But any decent equipment today gives very good sharpness and resolution at optimum conditions. Small prints like 8x10 never gets us close to resolution limits of todays DSLRs so if we see that an image from one system is better than another in small prints we probably are not discussing resolution but something else.
Another issue is that the digital imaging chain is very flexible regarding rendition. There are very few software solutions, if any, that really work with pristine non manipulated images, so we always compare the output of two highly configurable processing chains.
A summary of this may be: Relevant measurements are good, but need to be taken with a grain of salt.
Best regards
Erik
For the past 30 years, I've been professionally involved in designing signal processing systems for audio, sonar, radar, and telecommunications systems. I've also read way too many statements about the mystical ineffability, the 'magic', the unknowable connection between design/implementation and perceived performance. If your two pre-amps measured "identically" (presumably in frequency response) and performed differently at a level reproducibly perceptible to humans in blind trials, then you didn't do enough measurements. Perhaps it was phase or amplitude non-linearity, or transient ringing effects, etc., but if its a gross enough effect for a human to distinguish, it has a measurable (in some parameter) effect on the signal waveform.
I'm not saying that different designs don't sound different, or that people shouldn't prefer one device's 'interpretation' over another. People can be pretty good at 'different', but we're really terrible at 'better'. Outside controlled blind trials, we're not even very good at reliably detecting differences among similarly performing gadgets. The shameful/successful marketing of $1000 per foot pure crystalline-aligned unobtanium-alloy speaker cables shows how effective the marketing guys are at creating distinctions without differences in the minds of vulnerable, enthusiastic (well-heeled) hobbyists.
Much of what is sold as high-end audio today is quantum snake oil. The tube amplifiers that are so popular today among the 'golden ears' MR refers to sound good in their judgment, but they're (by design) not high-fidelity, if hi-fi is taken to mean faithful reproduction of the audio signal. Tubes are well known for their compressive amplitude transfer characteristics, which in a nice Class A configuration (with little feedback to suppress the harmonic distortion) will produce the sweet 'syrupy' harmonics (distortion) that the golden ears have declared to be High Fidelity (its especially sweetening to acoustic instruments and human voices. I even like it.). Their preference for low-power final amplifiers (or overdriven pre-amp stages) is to insure that the tubes are frequently driven into their compressive performance regime. Its fine that they like it, and that they are willing to pay for it. Defending their 'refined' listening preferences by saying there's no correlation between measurements and quality though, is a rationalization of their preference for harmonically sweetened playback. High-fi it ain't. (I'm sure there's a Kodachrome/Velvia analogy in here somewhere - look at all the pretty colors)
So, while I'm puzzled, along with many others, with the DX0 'resolution independent' quality measures, I can't subscribe to the 'numbers don't matter' school. Badly formulated or mis-applied metrics are common. More common is mis-interpretation of technical measurements by non-specialists. Most common by far, sadly, is over-reliance on, and endless yammering over the the importance of this or that technical parameter in what is, outside of medical and forensic imaging, an artistic endeavor more than a technical one. The numbers, properly formulated and interpreted, can characterize the performance of the instrument, but can't predict the artistic quality of the product.
Regards,
--Tom
-
Tom,
Just for the record, the measurments included all measurements (bandwidth, harmonics, IM, step signal(ringing)) typically used at that time, and typically used in marketing stuff and to limited extend in reviews. FTT for instance was then limited to highly specialized R&D laboratories, and not generally used.
In my case it was performed with calibrated quipment.
Nevertheless i agree with you that it must be measurable. And there are reproducable measurements that show for instance that cables do show different characteristics, and yes there are audible differences, and yes different is not always better.
Nor are so called blind tests above any error or disruption therefore of absolutely not of undisputable quality, check the archives of HiFi News & Record Review for instance.
I also agree that marketing and sales in the Audio realm preferred mystique (or snake oil, or ..) over real reproducable arguments, it simply made more money.
I do not agree that the "differences without distinctions" is limited to hobbyists, after all it are mostly educated professionals that design and produce the stuff. Just see if you can retrieve some biography on Andy Rappaport.
That is what concerns me in the photography realm, this history pattern repeats itself.
Also pure simple logic says that the more sites in a sensor for a given scene to take a picture form, the more data about the scene is captured in the picture, the bigger a site, if all other parameters are equal, the better the SNR is. So the next thing is the ADC, there a lot can go wrong, etc.
If somewhere in the total chain a limitation in resolution is introduced, some part of the signal is lost and cannot be restored again in subsequent stages of the chain.
So i agree with you about your statement being puzzled by the DxO quality measures, and agree that numbers do matter, and yes they, unless complete and proven correlated with human perceived results, only tell a portion of the total picture. Until then, just do not let them become leading in assessing quality.
I like your indication "artistic quality".
Regards,
Jan R.
For the past 30 years, I've been professionally involved in designing signal processing systems for audio, sonar, radar, and telecommunications systems. I've also read way too many statements about the mystical ineffability, the 'magic', the unknowable connection between design/implementation and perceived performance. If your two pre-amps measured "identically" (presumably in frequency response) and performed differently at a level reproducibly perceptible to humans in blind trials, then you didn't do enough measurements. Perhaps it was phase or amplitude non-linearity, or transient ringing effects, etc., but if its a gross enough effect for a human to distinguish, it has a measurable (in some parameter) effect on the signal waveform.
I'm not saying that different designs don't sound different, or that people shouldn't prefer one device's 'interpretation' over another. People can be pretty good at 'different', but we're really terrible at 'better'. Outside controlled blind trials, we're not even very good at reliably detecting differences among similarly performing gadgets. The shameful/successful marketing of $1000 per foot pure crystalline-aligned unobtanium-alloy speaker cables shows how effective the marketing guys are at creating distinctions without differences in the minds of vulnerable, enthusiastic (well-heeled) hobbyists.
Much of what is sold as high-end audio today is quantum snake oil. The tube amplifiers that are so popular today among the 'golden ears' MR refers to sound good in their judgment, but they're (by design) not high-fidelity, if hi-fi is taken to mean faithful reproduction of the audio signal. Tubes are well known for their compressive amplitude transfer characteristics, which in a nice Class A configuration (with little feedback to suppress the harmonic distortion) will produce the sweet 'syrupy' harmonics (distortion) that the golden ears have declared to be High Fidelity (its especially sweetening to acoustic instruments and human voices. I even like it.). Their preference for low-power final amplifiers (or overdriven pre-amp stages) is to insure that the tubes are frequently driven into their compressive performance regime. Its fine that they like it, and that they are willing to pay for it. Defending their 'refined' listening preferences by saying there's no correlation between measurements and quality though, is a rationalization of their preference for harmonically sweetened playback. High-fi it ain't. (I'm sure there's a Kodachrome/Velvia analogy in here somewhere - look at all the pretty colors)
So, while I'm puzzled, along with many others, with the DX0 'resolution independent' quality measures, I can't subscribe to the 'numbers don't matter' school. Badly formulated or mis-applied metrics are common. More common is mis-interpretation of technical measurements by non-specialists. Most common by far, sadly, is over-reliance on, and endless yammering over the the importance of this or that technical parameter in what is, outside of medical and forensic imaging, an artistic endeavor more than a technical one. The numbers, properly formulated and interpreted, can characterize the performance of the instrument, but can't predict the artistic quality of the product.
Regards,
--Tom
-
Toronto is some distance from Amsterdam, hard to just drop by, especially with this "pond" in between ;-)
However plans are to go to the USA in less than 2 weeks, i will try to get the return flight replanned to get to Toronto and pay you a visit.
I do hope it will be possible.
It would be in the period 20 - 22 February, are you there then?
Regards,
Jan R.
It's not the critical "exactly the same" test that some people want, but I now have about 15 prints from my recent shoot in Antarctica hanging at my gallery. Within the next week I should have about 30. In mid-March I'll have an open house and show. In the meantime if anyone wants to drop by my Toronto gallery when I'm there I'd be happy to show them to you.
What do they show? Most are in the 20" X 28" size range, matted to 28X34". Some are shot with the Sony A900 and some with the Phase One P65+. Can one see a difference at this size? Yes, absolutely – if you know what to look for, and it's not just about resolution. But to anyone except a technically knowledgeable observer these differences don't jump out. They all look pretty terrific.
There are no simple answers.
Michael
-
My opinion is more like:
- Something that measures bad is normally bad, at least if measurement is relevant
- There are things that are hard to measure which may have a high subjective significance
Erik,
That sound reasonable, except I'm very curious as to what it might be that has a high subjective significance but which is hard to measure.
Before anyone jumps in with the notion of 'artistic' merit, let me say that measurements of camera performance have nothing to do with the artistic merit of the photos that may be taken with the camera. This discussion has nothing to do with artistic quality. As Ansel Adams used to say, "There's nothing worse than a sharp image of a fuzzy concept".
One issue I'd like to point out is that we are obsessed with sharpness and resolution. But any decent equipment today gives very good sharpness and resolution at optimum conditions. Small prints like 8x10 never gets us close to resolution limits of todays DSLRs so if we see that an image from one system is better than another in small prints we probably are not discussing resolution but something else.
DXO doesn't address sharpness and resolution directly, and probably for good reason. Those qualities are so variable according to the quality of the lens used. However, I suspect sharpness is affected by a poor SNR. At base ISO, SNR is reasonably good for most cameras, so the resolution of the sensor relates very strongly to the pixel count, with slight variation according to the strength of the AA filter. However, at high ISO where SNR can be either quite good or quite poor, I suspect that a low SNR would indicate poor resolution, but always in relation to the pixel count of course. Is this not so?
Another issue is that the digital imaging chain is very flexible regarding rendition. There are very few software solutions, if any, that really work with pristine non manipulated images, so we always compare the output of two highly configurable processing chains.
That's very true. But the RAW file is the digital negative. There may be many ways of improving the development process of that RAW file as one's skills improve, and as the performance of new RAW converters improve over time. It therefore seems important when choosing a camera to get one which produces the best RAW files, with the lowest noise, the greatest tonality, the highest DR etc, but not of course at the expense of all other considerations. That would be silly.
For example, the A900 has a very desirable feature in the form of an anti-shake sensor which applies to all lenses fitted to the camera. Some people might quite reasonably take the view that the anti-shake sensor of the A900 is more of an asset than the better high-ISO performance of another camera which doesn't have an anti-shake sensor. Others might take an opposite view. That's fair enough. But in order to know what the high-ISO performance is in the first place, one needs the sort of objective measurements that DXO is providing.
Without such objective testing, we are left with endless subjective squabbling and total confusion.
-
Ray,
The "RAW file being the digital negative", perhaps more to the point, the "developed digital negative".
The sensor site voltage values as captured then best compares with the exposed silverhalide crystals in the emulsion.
I take it we all know from B/W development experience how developing methods and materials can influence the outcome.
The DxOLab measures the outcome of the "development". How this "development" is done is company proprietary.
Anyhow, the endresult is what matters.
Regards,
Jan R.
-
Measurements are great - but what do you measure, how do you measure it, and how do you compare it? What is the usefulness of a resolution measure without a corresponding measurement of the degree of aliasing, or usable contrast at that resolution. None of this boils down to a single number.
Measure noise, and it might be ugly noise, or it might be a nicer looking noise. A single number doesn't tell you anything about the character of the noise. Given the prevalence of noise reduction software, how do you account for some types of noise being easier to remove than others in a single number?
Measurements are great for engineers if they're repeatable and tell you something useful to help you with your design. Designing a good test for a certain quality is at least as important as to how you engineer improvements in that certain quality. As someone who has to do this, I spend more time eyeballing and doing direct A/B image comparisons than I do looking at measurement numbers. Numbers are the guide, but the resulting image is ALL.
Graeme
-
Ray,
Some answers:
"That sound reasonable, except I'm very curious as to what it might be that has a high subjective significance but which is hard to measure."
Something I had on my mind was flare and ghosting, they are difficult to quantify because they are highly dependent on the shooting conditions.
Another issue is that there is often some esoteric quality attributed to certain lenses like "3D-look", I don't know what that means. I have two Sony lenses with blue labels saying Zeiss and I would say that they perform better than they test, at least in my own tests. A good explanation is something I don't have. Some clues I may have is that Zeiss may have priority on the central part of the image where they strive for high MTF at some frequencies whereas they put little emphasis on edge performance at large apertures. When I'm testing lenses I'm always looking at the corners at full aperture because I know that is the weakest point of all lenses.
Another such aspect is that knowledgeable and experienced people, like Michael Reichmann, say that the advantage of MFDBs is visible on relative small prints. This may be contrary to my expectation but there must be some reason. I made some experiments where I compared 12 MP A700 pictures with 24 MP A900 pictures printed in A2 and could see little difference, although there was a large advantage to the 24 MP A900 files when looking at similarly uprezzed files at actual pixels.
There are a few comments on this forum essentially claiming that DxO measurements don't have results that fits expectations or experience and therefore they must be trash, In my opinion it's better to discuss what DxO measures and what is does not.
Best regards
Erik
Erik,
That sound reasonable, except I'm very curious as to what it might be that has a high subjective significance but which is hard to measure.
Before anyone jumps in with the notion of 'artistic' merit, let me say that measurements of camera performance have nothing to do with the artistic merit of the photos that may be taken with the camera. This discussion has nothing to do with artistic quality. As Ansel Adams used to say, "There's nothing worse than a sharp image of a fuzzy concept".
DXO doesn't address sharpness and resolution directly, and probably for good reason. Those qualities are so variable according to the quality of the lens used. However, I suspect sharpness is affected by a poor SNR. At base ISO, SNR is reasonably good for most cameras, so the resolution of the sensor relates very strongly to the pixel count, with slight variation according to the strength of the AA filter. However, at high ISO where SNR can be either quite good or quite poor, I suspect that a low SNR would indicate poor resolution, but always in relation to the pixel count of course. Is this not so?
That's very true. But the RAW file is the digital negative. There may be many ways of improving the development process of that RAW file as one's skills improve, and as the performance of new RAW converters improve over time. It therefore seems important when choosing a camera to get one which produces the best RAW files, with the lowest noise, the greatest tonality, the highest DR etc, but not of course at the expense of all other considerations. That would be silly.
For example, the A900 has a very desirable feature in the form of an anti-shake sensor which applies to all lenses fitted to the camera. Some people might quite reasonably take the view that the anti-shake sensor of the A900 is more of an asset than the better high-ISO performance of another camera which doesn't have an anti-shake sensor. Others might take an opposite view. That's fair enough. But in order to know what the high-ISO performance is in the first place, one needs the sort of objective measurements that DXO is providing.
Without such objective testing, we are left with endless subjective squabbling and total confusion.
-
Ray,
The "RAW file being the digital negative", perhaps more to the point, the "developed digital negative".
The sensor site voltage values as captured then best compares with the exposed silverhalide crystals in the emulsion.
I take it we all know from B/W development experience how developing methods and materials can influence the outcome.
The DxOLab measures the outcome of the "development". How this "development" is done is company proprietary.
Anyhow, the endresult is what matters.
Regards,
Jan R.
Jan,
That's not how I understand it. DXO are trying to test the performance of the sensor independently of the performance of any particular RAW converter. They measure the sensor's response before demosaicing has taken place, for example.
When I say a RAW file is a digital negative, I mean it's analagous to an undeveloped piece (or roll) of film. How much would film afficionados pay for a magical roll of film that could be developed and redeveloped in as many ways as they liked and as often as they liked? After developing a 36 exposure, ISO 100 roll of film, suppose you find that there are a couple of shots that should have been 'push processed' because they were underexposed, and another couple of shots that needed less development time because they were overexposed, and yet another couple of shots that could have benefited from a different type of developer.
When this happens with film, you're basically stuffed. You only get one chance at development. With RAW files, the opportunities are endless. I know I can certainly get better results now from my 5 1/2 year old D60 raw files using the latest version of ACR, than I did with the early version of BreezeBrowser that I used before ACR was available.
-
Jan,
I am on vacation in Costa Rica from Feb 14 till 21 and so I should be available in the days after.
If you wish to come by drop me a line beforehand and I'll be sure to be available as long as I'm in town. (I am teaching all day on the 25th).
Michael
-
Something I had on my mind was flare and ghosting, they are difficult to quantify because they are highly dependent on the shooting conditions.
Erik,
Difficult to quantify in the absence of an internationally accepted standard and method of testing, but not necessarily difficult to test in relation to a specific set-up of direct lighting that someone might devise for the purpose of comparison. As long as the methodology is consistent, then comparative tests for flare and ghosting should reveal that Lens A is better than lens B. However, any single numerical rating resulting from one particular method of testing could not be compared with another single numerical rating that is the result of a different methodology.
Another issue is that there is often some esoteric quality attributed to certain lenses like "3D-look", I don't know what that means. I have two Sony lenses with blue labels saying Zeiss and I would say that they perform better than they test, at least in my own tests.
But we don't seem to have any DXO type tests of lenses available. DXO obviously must be making extremely thorough and detailed tests of the lenses they include as modules in their raw converter, otherwise their converter wouldn't be able to automatically correct for the lens deficiencies. But they don't make these tests public.
The MTF charts that are currently available for the lens itself seem to be mostly based on the theoretical performance of the lens design. Photozone's 'system resolution' tests seem to be the most informative tests currently available, but one always has to make a guess as to the role of the camera used.
Another such aspect is that knowledgeable and experienced people, like Michael Reichmann, say that the advantage of MFDBs is visible on relative small prints.
Michael has also said the opposite. A group of experienced photographers could not tell the difference between a G10 image and a P45+ image of the same scene, at A3+ size. My own view is that the nature of the subject being photographed and compared will often determine the differences that are noticeable.
Smooth sheets of ice in the Antartic might reveal very clearly the smoother tonality of the P45+ compared with the G10 at A3+ size.
There are a few comments on this forum essentially claiming that DxO measurements don't have results that fits expectations or experience and therefore they must be trash, In my opinion it's better to discuss what DxO measures and what is does not.
More than a few. There's almost hysteria .
-
Measure noise, and it might be ugly noise, or it might be a nicer looking noise. A single number doesn't tell you anything about the character of the noise. Given the prevalence of noise reduction software, how do you account for some types of noise being easier to remove than others in a single number?
Graeme,
I don't think that anyone is claiming that DXO type tests are a measurement of esthetics. Ugliness and beauty are very subjective matters. However, having identified the characteristics of noise which some people find ugly or beautiful, as the case may be, then I see no reason why it could not be tested and measured. DPP for example seems to produce a slightly finer type of noise than ACR which is noticeably blotchy by comparison, at the extreme pixel-peeping level. That difference didn't just happen by accident. It's a purposeful effect, or an unavoidable consequence of some other design feature of the raw converter.
In any case, DXO measurements are made at an earlier stage to the peculiar characteristic of a specific raw converter, aren't they?
Surely the goal is to have as little noise as possible. If you then want to add some for esthetic reasons, then that's possible, isn't it?
-
Ray,
I have a couple of issues with Photozone. The major one is that they compensate for field curvature, don't use the same focus for the centers, edges and corners. The other issue I have is that all their tests ae on APS-C or 4/3. Nothing wrong about that, except that the tests say nothing abut edge performance on full frame. Photozone testing is based on Imatest, the same program I use.
There are other lens tests on the net, SLR-gear tests are based on DxO technology and DPReview tests are based on "Slanted Edge", like Imatest. Both have relatively few tests.
My favorite tests are done by the Swedish monthly "Foto". They test at Hasselblad using their lens testing equipment. With Foto's tests I also have some issues, they are free for subscribers but others need to pay. So I guess it's hard to refer to those tests on the net. The other issue I have that they publish MTF for 20 LP/mm for full frame lenses, 30 for APS-C and 40 for 4/3. I would prefer the traditional 10/20/40 possibly completed by 60.
One great advantage of the "Foto" tests is that they match my experience.
A very nice aspect of the of the Photozone tests is that they also report on Bokeh and Longitudal Chromatic Aberration.
Best regards
Erik
The MTF charts that are currently available for the lens itself seem to be mostly based on the theoretical performance of the lens design. Photozone's 'system resolution' tests seem to be the most informative tests currently available, but one always has to make a guess as to the role of the camera used.
-
The demosaic can impart a particular character to the noise, and indeed, it does seem that DXO try to eliminate that. However, the noise is in the underlying data and will have a particular pattern or character to it. A PSNR measurement does not tell you that character though. Sure you could invent some measurement that perhaps tells you more, but that is tricky. Noise could be a mix of shot noise, read noise and fixed pattern noise. They all look different. A noise number just totals them up. On the whole, fpn is much more objectionable than the other as it makes for visible patterns in the extreme noise floor.
Given you'll always have noise, the precise character of that noise is very important.
Graeme
Graeme,
I don't think that anyone is claiming that DXO type tests are a measurement of esthetics. Ugliness and beauty are very subjective matters. However, having identified the characteristics of noise which some people find ugly or beautiful, as the case may be, then I see no reason why it could not be tested and measured. DPP for example seems to produce a slightly finer type of noise than ACR which is noticeably blotchy by comparison, at the extreme pixel-peeping level. That difference didn't just happen by accident. It's a purposeful effect, or an unavoidable consequence of some other design feature of the raw converter.
In any case, DXO measurements are made at an earlier stage to the peculiar characteristic of a specific raw converter, aren't they?
Surely the goal is to have as little noise as possible. If you then want to add some for esthetic reasons, then that's possible, isn't it?
-
The demosaic can impart a particular character to the noise, and indeed, it does seem that DXO try to eliminate that. However, the noise is in the underlying data and will have a particular pattern or character to it. A PSNR measurement does not tell you that character though. Sure you could invent some measurement that perhaps tells you more, but that is tricky. Noise could be a mix of shot noise, read noise and fixed pattern noise. They all look different. A noise number just totals them up. On the whole, fpn is much more objectionable than the other as it makes for visible patterns in the extreme noise floor.
Given you'll always have noise, the precise character of that noise is very important.
Graeme
It would be an interesting exercise if someone could find a couple of cameras that DXO have tested, of similar pixel count and similar SNR ratings at a particular ISO, and show us some 100% crops demonstrating the ugliness of one camera's noise compared with the beauty of the other.
What you write sounds fine in theory, but in practice it might be of little significance outside of extreme pixel-peeping.
I wasn't aware that one could do anything about shot noise, so surely that should be out of the discussion. I recall that one of the attractive features of RSP, which I frequently used as my raw converter before Adobe absorbed them, was a slider for removing fixed pattern noise. I recall that it was very effective, but not without the usual penalty of noticeable loss of resolution.
-
I have a couple of issues with Photozone. The major one is that they compensate for field curvature, don't use the same focus for the centers, edges and corners. The other issue I have is that all their tests ae on APS-C or 4/3. Nothing wrong about that, except that the tests say nothing abut edge performance on full frame.
Erik,
I wasn't aware that Photozone used different focussing for the centre and edges. I wonder if that has a practical significance that can be misleading, perhaps photographing brick walls from a close distance and at full aperture, for example.
The reason I have a high regard for the old Photodo tests as well as the newer Photozone tests is that whenever I've taken the trouble to test my own lenses that feature in either of those two bodies of tests, the results seem very similar to either Photodo's or Photozone's tests in respect of resolution at the centre and edges and at the apertures they've tested.
A couple of examples would be:
(1) the relatively poor performance of the Canon 100-400 IS at 400mm and F5.6. Both Photodo and Photozone show that resolution is noticeably better at F8 and F11, and so it is with my copy of the lens.
(2) the stellar performance of the Canon EF-S 17-55/2.8. This lens did not exist when Photodo was producing MTF charts, but Photozone tests show this lens as being as sharp as the Canon 50/1.8 prime at F2.8, but as one would expect, not as sharp at the edges because the 50/1.8 prime is designed for full frame and the EF-S is designed for the cropped format.
My own comparisons with the 50/1.8 prime confirmed this result, but not initially. Focussing at infinity, the 50/1.8 prime at F2.8 looked noticeably sharper, causing me some initial disappointment. Perhaps I'd been unlucky and had bought a substandard copy of the 17-55. I tried again but this time using Live View on the 40D and focussing manually. Bingo! The results were almost exactly in line with Photozone's tests, or as close as matters. I sent the lens to Canon Repair for calibration under the warranty.
The fact that Photozone test their lenses on the APS-C format is not such a big disadvantage. I don't recall ever seeing an MTF chart that had a sloping response out to 12mm from the centre then suddenly a flat response from 12mm to the corners. If a lens has an excellent performance at the borders on an APS-C camera, it's likely to have at least a good performance at the borders on full frame. Likewise, a lens that has a noticeably poor performance at the borders on APS-C, is likely to be very poor at the borders on full frame, wouldn't you say?
-
Isn't this really all about sacred cows and the fact we don't like having ours slaughtered? DxO are guilty of the sin of independence from subjectivity. Surely they should have realised that Photography is all about brand loyalty and preconceptions based on price, not real world performance
Quentin
-
Ray,
To begin with I have much respect for Photozone tests. Some lenses, like the Tamron 17-50/2.8, have significant field curvature. That means if you are focusing at the center the corners will not be sharp. The 17-50/2.8 I have is absolutely lousy in corners at full aperture at 17mm.
A few aberration increase with the square of the distance from the center, so sharpness can fall of rapidly. Many lenses have a "sweet spot" which often can cover the same area as APS-C, for these two reasons I don't feel that testing on APS-C gives any good information on border/corner performance when used on full frame.
Regarding the "old photodo tests", I agree fully. The present tests in "Foto" are essentially the continuance of those tests.
Lars Kjellberg was technical editor at "Aktuell Fotografi" and initiated those tests, after "Aktuell Fotografi" merged with "Foto" Lars Kjellberg started the original Photodo which originally was a publication on paper (bimonthly I think).
Best regards
Erik
Erik,
I wasn't aware that Photozone used different focussing for the centre and edges. I wonder if that has a practical significance that can be misleading, perhaps photographing brick walls from a close distance and at full aperture, for example.
The reason I have a high regard for the old Photodo tests as well as the newer Photozone tests is that whenever I've taken the trouble to test my own lenses that feature in either of those two bodies of tests, the results seem very similar to either Photodo's or Photozone's tests in respect of resolution at the centre and edges and at the apertures they've tested.
A couple of examples would be:
(1) the relatively poor performance of the Canon 100-400 IS at 400mm and F5.6. Both Photodo and Photozone show that resolution is noticeably better at F8 and F11, and so it is with my copy of the lens.
(2) the stellar performance of the Canon EF-S 17-55/2.8. This lens did not exist when Photodo was producing MTF charts, but Photozone tests show this lens as being as sharp as the Canon 50/1.8 prime at F2.8, but as one would expect, not as sharp at the edges because the 50/1.8 prime is designed for full frame and the EF-S is designed for the cropped format.
My own comparisons with the 50/1.8 prime confirmed this result, but not initially. Focussing at infinity, the 50/1.8 prime at F2.8 looked noticeably sharper, causing me some initial disappointment. Perhaps I'd been unlucky and had bought a substandard copy of the 17-55. I tried again but this time using Live View on the 40D and focussing manually. Bingo! The results were almost exactly in line with Photozone's tests, or as close as matters. I sent the lens to Canon Repair for calibration under the warranty.
The fact that Photozone test their lenses on the APS-C format is not such a big disadvantage. I don't recall ever seeing an MTF chart that had a sloping response out to 12mm from the centre then suddenly a flat response from 12mm to the corners. If a lens has an excellent performance at the borders on an APS-C camera, it's likely to have at least a good performance at the borders on full frame. Likewise, a lens that has a noticeably poor performance at the borders on APS-C, is likely to be very poor at the borders on full frame, wouldn't you say?
-
To begin with I have much respect for Photozone tests. Some lenses, like the Tamron 17-50/2.8, have significant field curvature. That means if you are focusing at the center the corners will not be sharp. The 17-50/2.8 I have is absolutely lousy in corners at full aperture at 17mm.
Eric,
Having just looked at the PZ review of the Tamron 17-50/2.8, I see what you mean. The graphs do not give any indication of the problem of field curvature, but at least Photozone mention this problem and the fact that at 17mm and even at 24mm it's pretty extreme with this lens.
The Canon 17-55/2.8 on the other hand, has only slight field curvature at 17mm, which PZ also mentions. I never considered getting the Tamron 17-50/2.8, not because I was aware it had strong field curvature, but because third party lenses lack IS and this zoom is a walk-around lens for me.
I know that some folks like to criticise MTF charts on the basis they don't tell you everything about a lens. And it's true, they don't. But that's no reason for ignoring them. They tell you what they tell you and hopefully if there are other issues such as field curvature or flare or excessive CA, the reviewer will mention such factors. If he doesn't, then other sources likely will.
Regarding the "old photodo tests", I agree fully. The present tests in "Foto" are essentially the continuance of those tests.
I'm interested. Is this magazine available in English?
-
Sorry it is not.
I'm interested. Is this magazine available in English?
-
Isn't this really all about sacred cows and the fact we don't like having ours slaughtered? DxO are guilty of the sin of independence from subjectivity. Surely they should have realised that Photography is all about brand loyalty and preconceptions based on price, not real world performance
Is is apparently about things you are not skilled enough to see.
What remains to be confirmed is whether people blessed with these golden eyes are the same having the golden ears needed to feel comfortable using 20.000 US$+ speakers.
In the mean time I'll forge ahead using my inferior pano kit.
(http://farm4.static.flickr.com/3404/3262015614_9afaa8193f_o.jpg)
Cheers,
Bernard
-
Is is apparently about things you are not skilled enough to see.
True, but I know snake oil when I see it
What remains to be confirmed is whether people blessed with these golden eyes are the same having the golden ears needed to feel comfortable using 20.000 US$+ speakers.
In the mean time I'll forge ahead using my inferior pano kit.
Cheers,
Bernard
Well Bernard, I spent years of my time and some serious money on exotic hi fi and there were moments of sublime sound, but not enough of them to outweigh the wasted time spent listing to the equipment and not to the music. Now I have dumped the valve amps and imposing room cramping speakers, have a good but not absurdly expensive system and enjoy music a lot more. Somewhere in there is a lesson for photo equipment junkies.
Quentin
-
Quentin, you are showering the forum with logic and sound thinking! Please stop it ;-)
I think we all share an interest in technology, and I am not going to deny that. I wouldn't slam the door on some super nice top end gear too.
The pursuit of testing/collecting equipment, for the sake of it, is a false prophet.
Music is a great comparison, I used to admire Jimi Hendrix when I was growing up (forget if you like his music, or the rock n roll lifestyle etc) This is the same guy who used a wooden broomstick to mimic playing on. The guy who used a one-stringed ukulele found in his father's garage. Now he moved onto Fenders, and Gibson's..the nice stuff..but the man was "obsessed with playing guitar". He even used to play in the toilet..with no amp.
What he was not..was "obsessed with technology, or guitars" He was not gripped with testing his equipment, he just used it (heck even smashed it up at times!). And it would be hard to ignore the pure talent, even if you did not care for his style.
The problem with many folks on this forum, is they are too busy trying to find the right guitar, for the "sound they like", and reading the tests that prove it's the best guitar out there. Not that it's unimportant, but hey..you have to call it a day sometime. And not enough time spent playing the damn thing!
So I don't want to hear about the "new guitar just out with a great sound" I want to hear about the tunes you knock out. Testing is pretty boring if we are honest..there will always be a new shiny guitar in the shop, that might be a bit better than the last model you bought.
In the spirit of non testing, I present a photo..well I just decided the other night to stop reading the tests, and take a photo. My apologies for using non high end stuff, I am sure I would get a sharper bigger print with Carl Zeiss optics, and a FF digital body. Sorry about that..shot on a super budget DSLR with a £70 lens, hey lets call it a slightly beaten up guitar with a not bad sound.. ;-)
[attachment=11419:Barry_Po...b_2009_8.jpg]
Have fun talking about testing! lol
ps..it passed my own personal test..aka using the thing!
Forgot to mention, jpeg straight from camera (bar USM and resize)...how dare I??? lol
-
The problem with many folks on this forum, is they are too busy trying to find the right guitar, for the "sound they like", and reading the tests that prove it's the best guitar out there.
Clearly so. Another thing, closely related, is that you probably end up playing nicer if you use the same guitar all the time, since you'll get to really know it in depth.
As far as I am concerned, I have decided to stick to my current body for at least 2 to 3 years. Having understood that the world is not ready to acknowledge the fact that I own the best camera, I will focus on trying to be the best user of that camera in the world... that should keep me busy enough...
Cheers,
Bernard
-
As far as I am concerned, I have decided to stick to my current body for at least 2 to 3 years. Having understood that the world is not ready to acknowledge the fact that I own the best camera, I will focus on trying to be the best user of that camera in the world... that should keep me busy enough...
Cheers,
Bernard
Bernard,
I see that you are going through a phase of 'obsession with temple detail'. Is that right?
-
I see that you are going through a phase of 'obsession with temple detail'. Is that right?
Temple are a fascinating subject!
(http://farm4.static.flickr.com/3489/3267265867_4893361721_o.jpg)
120 megapixels HDR pano shot from D3x. The HDR work was done entirely by hand since I could not get satisfactory results with any of the automated tools I tried. Whitebalance was also difficult to handle well, and single conversions didn't do the trick.
Raw converson also involved different converters for differents parts of the image.
Cheers,
Bernard
-
What I was trying to say, in a round about way..was this.
Places like DxO (with all due respect), have little to no real benefit for photographers. It really is numbers and charts. There are no images to see (real or test shots), no comments on subjective area like handling and performance. Really not a great place to find out about cameras anyway.
You could argue review sites do similar things, but at least they have some worth.
Would you really care if they say x model has 0.3 extra DR over your current one? Even if that were true..it's unlikely to be field relevant.
-
Would you really care if they say x model has 0.3 extra DR over your current one? Even if that were true..it's unlikely to be field relevant.
Yes. I would care very much if subjective reports from proud owners of such a camera claimed it had 2 stops greater DR than my current one. In the absence of objective reports from DXO, I might be tempted to buy such a camera. The fact that model x has only 0.3 stops of extra DR is useful information to have. It means that I can either exclude, or give low priority to that factor when considering the significance of the other features of camera x that I might find advantageous.
-
120 megapixels HDR pano shot from D3x. The HDR work was done entirely by hand since I could not get satisfactory results with any of the automated tools I tried. Whitebalance was also difficult to handle well, and single conversions didn't do the trick.
Raw converson also involved different converters for differents parts of the image.
Cheers,
Bernard
Bernard,
I'm not so sure. You might save yourself a lot of work in front of the computer if you were to get yourself a P65+ .
-
Bernard,
I'm not so sure. You might save yourself a lot of work in front of the computer if you were to get yourself a P65+ .
I don't think it would change that much... I would need a 3 images stitch anyway to reach this resolution, DR is said to be similar so that I would need to take at least 6 frames. The host cameras being heavier I would probably run into problems with the stability of my stitching head... I am really not sure that I would get superior results in fact.
Besides, since I don't charge myself any consulting fee, time is a lot cheaper than 32.000 US$...
Cheers,
Bernard
-
Temple are a fascinating subject!
Quentin
-
The following image was shot with a D700 and 85mm F1.4 Nikkor using three rows of images and PTGui. Like yours, Bernard, hand-held due to tripod restrictions.
(http://qdfb.smugmug.com/photos/403770683_MFQe9-X2.jpg)
Quentin
-
Bernard,
Very impressive, as is your dedication to getting it right. What was wrong with results from stitching software, like PTGui?
Thanks.
The first thing is that it was impossible to get pleasing colors in both shadows and lit parts with a single raw conversion per frame (except for the foreground where I decided to keep bluish shadows that are more natural). So I had to overlay frames anyway to handle that part.
The second thing is that C1, that I initially intended to use for all conversions, was unable to handle purple fringing in the strongly backlit trees, raw Developper did a much better job on that part. So more overlay to do there.
Finally, I had to use two sets of images to handle the DR of the scene (I had shot 3 images 1 stop apart from each other, but ended up only using the darkest and lightest ones). I tried both the Fusion and true HDR rendering of PTgui, as well as the Autopano Pro auto HDR, but I couldn't get something pleasing with either.
In the end, hand overlay was a much better option that would have been reasonnably fast had I decided to go that route since the beginning. I guess that this is the good thing about being mostly an amateur, I can afford to waste time like this to get an image the I want it to be.
Cheers,
Bernard
-
The following image was shot with a D700 and 85mm F1.4 Nikkor using three rows of images and PTGui. Like yours, Bernard, hand-held due to tripod restrictions.
Brilliant Quentin, thanks for sharing. Did you have to do any HDR to handle the contrast or were single frames enough?
Mine was in fact shot on tripod.
Cheers,
Bernard
-
Brilliant Quentin, thanks for sharing. Did you have to do any HDR to handle the contrast or were single frames enough?
Mine was in fact shot on tripod.
Cheers,
Bernard
Hi Bernard,
Thanks - no HDR because of the tripod ban, and I found the dynamic range of the D700 dealt well with the contrast range, being biased towards preserving highlights and using the excellent detail recovery in the shadows. Also the fast prime meant a reasonably realistic depth of field (comparable to a LF film camera) and faster shutter speeds.
If I had the opportunity I would have preferred to use a tripod and pano head but this was in Vienna and I was shooting next to toursits with their compacts, with my camera stuck through the grille as a service was in progress (a plus because in meant fewer people milling about and not seated for the service). Given that the circumstances were not exactly conducive for high quality photography, I was quite pleased with the result.
Quentin
-
Clearly so. Another thing, closely related, is that you probably end up playing nicer if you use the same guitar all the time, since you'll get to really know it in depth.
Cheers,
Bernard
Very much agree. Another testimony on a contentious subject: what is better, in body or in lens anti-shake.
A good friend of mine has a Canon 40D and 100-400IS, I have a Sony A700 and 100-400APO.
We set up a test at 400 mm and ~ 1/60th of a second exposure. Out of ~50 shots we got similar results with regard to motion blur.
After resting our arms a bit (no beer !) we took each others camera and repeated the test. We both got significantly worse results. In talking it over afterwards (now with a beer) we determined that my friend was probably put of by a non stabilized viewfinder image and for me the reverse.
Conclusion : the best camera is probably the one you have and know well.
Not a scientific test by a long shot, just an illustration of a practical thing we both enjoyed doing.
Sorry for a bit off topic, but still partly relevant when comparing numbers in a scientific test vs. performance out in the field.
-
The following image was shot with a D700 and 85mm F1.4 Nikkor using three rows of images and PTGui. Like yours, Bernard, hand-held due to tripod restrictions.
Quentin
Bernard and Quentin, thanks for posting those excellent stitched photographs. They demonstrate once again that the end-point of all this technical discussion is the production of fine photographs, and they prove yet again that one can buy all kinds of gear at widely varied price points, all of which will perform very well when used within their design parameters and the images intelligently processed by competent photographers.
Mark
-
The following image was shot with a D700 and 85mm F1.4 Nikkor using three rows of images and PTGui. Like yours, Bernard, hand-held due to tripod restrictions.
.
-
Quentin,
Impressive interior of a church (cathedral), but I'm wondering about that out-of-focus structure, lower left. Are you trying to divert the attention of the viewer towards the right, which at this size seems sharp and in focus, and perhaps more interesting. But I can't help the feeling of a slight annoyance that the detail on the left is blurred .
Hi Ray,
Its just a depth of field issue. The object left is closer, and I was shooting, out of necessity, pretty wide (about F2 - I ought to check the files but don't have them on this computer) with the 85mm F.1.4 to avoid camera shake. I could crop, but I prefer the whole view.
cheers
Quentin