DxOMark and advanced D-SLRs

The high-end sensor arms race: who is winning?

Michael Reichmann (of the popular site Luminous Landscape) recently concluded that the latest and greatest Nikon and Canon cameras are now indistinguishable as far as sensor noise in concerned:

I see no significant noise difference between the Nikon D3x, Canon 5D MKII and Sony A900 up to and including ISO 800. [..] From ISO 1600 to ISO 6400 the Canon 5D MKII and the Nikon D3x are neck and neck. I think that the visible differences on-screen at 100% are a quibble, and on prints are completely irrelevant.

Reichmann’s review covers the latest (Q4 2008) “full-frame” models with a resolution of 21.1 MPixel (Canon 5DM2) and 24.5 MPixel (Nikon D3x). He also included the Sony A900, which includes a 24.5 MPixel sensor. That sensor is similar to the one Sony supplies Nikon for the D3x. Reichmann’s review is based on visual comparison of images taken at varying ISO speeds of a color test chart with some surrounding props.

DxOMark

If you study the supposedly more rigorous/scientific measurements on sensor performance provided by DxOMark, the conclusions appear to be comparable (the Nikon D3x has yet to be tested):

  • Canon and Nikon have similar performance on recent cameras. In fact, in one graph the Nikon D3 and Canon 1Ds3 markers coincide.
  • The site shows that full-frame sensors are indeed better than the more common 1.5x or 1.6x (“APS-C”) sensors. This is interesting, because most discussions of sensor format have trouble defining what a fair comparison criterion would be – given that the same lens will give a wider image with a larger sensor.
  • Marketing-wise Canon is putting more emphasis on high resolution (their lineup contains 3 models which are 15 MPixel or higher: 15 Mpix 50D, 21 Mpix 5D2, 21 Mpix 1Ds3).
  • You will find that the 12 MPixel Nikon D3 is lower noise (better at high ISO) than any Canon product*. This is presumably because dividing a large pixel into two smaller ones will waste some silicon area and thus performance: you will get more noise. This extra noise can only be partially compensated after taking the picture by downscaling the resolution using software.** This phenomenon is, however, more subtle than is often claimed, and DxO has an interesting, but tough white paper on this.

But the basic message is simple enough: apparently that Canon has caught up with Nikon’s recent lead that was created with the release of the Nikon D3 (12 MPixel) and D700 (12 MPixel). And according to Reichmann, the Canon D1s Mark III (21 Mpixel, not shown) is very comparable to the lower cost Canon 5D2 (21 Mpixel) and thus to the Nikon D3x (25 MPixel). So Nikon’s lead was probably a combination of a technical success (Nikon caught up and even overtook Canon) plus a marketing success: the Canon 1Ds Mark III was launched 1.5 year earlier than the Nikon D3x, both seem to be comparable.

DxOMark methodology

Surprisingly, if you look at the raw performance-versus-ISO graphs at the DxOMark site, you get a different perception of the cameras than by looking only at the final DxOMark index.

Let’s first look at the real measurement data by DxO. I somewhat arbitrarily chose this example. Check out all 4 benchmarks provided by DxO: sensor noise, dynamic range, tonal range and color sensitivity. The respective Nikon and Canon top scorers (12 MPixel D3 and 21 MPixel 1Ds Mark III) are shown, along with a 5-year older model (6 MPixel Canon 10D) which I added essentially to put any differences between comparable camera’s in perspective. It is important to view the graphs in DxOMark’s Print mode rather than Screen mode. This simulates performance when images are compared at the same viewing size (like a print) by compensating for any resolution differences between sensors: noise of smaller pixels will be smaller dots on paper, and thus be less visible than bigger dots.

Canon 1Ds3 versus Nikon D3 (noise figures in "Print" mode)

So far, so good. The problems occur when DxO tries to simplify all this somewhat complex multi-dimension information (remember that there are 4 of these graphs) to a single number. The end-users tends to like a single number because it gives a clear winner. This is why DxO introduced the DxOMark concept, with a clear nod to computer hardware benchmarks that compare speed (3DMark, etc). Although benchmarking of computer speed is a nasty enough problem because the outcome depend somewhat on what you need to compute, benchmarking of image sensors seems even trickier:

  1. The parameters measured by DxO depend significantly on the amount of available light. In the above graph, the signal to noise ratio varies by 100:1 essentially because real world light intensity easily varies by at least 100:1. This implies that on a sunny day noise is not an issue (and to its credit DxO explains this on their site): in studio or for a landscape photographer at 100 or 200 ISO, noise is nowadays invisible (and incidentally better than film ever way). But for a sports or concerts, noise remains an issue.
  2. A single figure of merit needs to combine different sensor properties: noise, dynamic range, color accuracy, etc. Again, different users have different priorities. So this is like judging cars with a single number: speed, fuel efficiency, comfort all go into one number. This is a flawed approach. But to DxO’s credit, you get all the raw info (like the graph above and 3 more graphs like it) and DxO warn about how to interpret the data.
  3. A nasty problem with DxOMark’s methodology is what to do about image resolution: in the above example, we are comparing a 6 MPixel (10D, 2003), a 12 MPixel (D3, 2007), and a 21 MPixel (1Ds3, 2007) model. When viewed in Print mode on the DxO site (and as copied above), the values are made comparable by computing what the result would look like when scaled to a given size image viewed from the same distance (like when you print).
    This is fair to judge the noise, dynamic range and color behavior. But it ignores resolution itself. In fact, you can improve noise at the cost of resolution (by averaging and other forms of filtering). As far as I am aware, the DxOMark figures (both the graphs and the single index) do not reward a camera for having a high resolution. In fact, a high-resolution camera is actually punished a bit for being high resolution, because high resolution makes it difficult to achieve the metrics which DxOMark does include in their assessment.
    Hypothetically I could take any camera, reduce the resolution to 1 pixel by averaging all the sensor pixels. This would give extremely low noise (essentially unmeasurable: probably a 30dB improvement over the original sensor) but detail would be terrible. So the question remains: is it fair to compare camera sensors as if resolution were entirely irrelevant? Before some people say “yes, but beyond a point resolution is irrelevant”, remember that a 21 MPixel full frame sensor has the same pixel density as an 8 MPixel 1.6x camera.

Single-value DxOMark score?

So, to summarize, a single number DxOMark figure-of-merit is risky. If you have the technical skills, it is better to study the multi-dimensional raw data. If you know what image quality aspect is most important to you, DxO even has hints which of its graphs is most important.

So much for the theory. Let’s look at the actual DxOMark index scores for the cameras shown in the graph reproduced above:

  • Canon 10D (Feb 2003): 56.6 points
  • Canon 1Ds3 (Aug 2007): 80.3 points
  • Nikon D3 (Aug 2007): 80.6 points

DxO does not explain how the measurements from the 4 graphs are computed into a single score. But the ratios of 70%:100%:100.4% do not seem to reflect the raw data. This needs clarification. Firstly because there may be a misunderstanding about the methodology. One possible explanation is that the Nikon D3 is somehow “punished” for lacking a 100 ISO mode: at 100 ISO, the Canon tends to exceed the Nikon which cannot go below 200 ISO.

Peter

* Maybe the future Canon 1D Mark IV should be full-frame 12 MPixel (simplifies the product range and marketing) instead of 1.3x 10 Mpixel.

** DxOMark compensates for differences in resolution if you put it in Print mode (all MPixels squeezed into fixed print). Screen mode is what you see at 100% crop (which is a silly way to benchmark as it punishes high-res sensors in an unreasonable way).

[ this posting is longer than 1000 words: 1353. Needs to be split?]

This entry was posted in Photo Technology and tagged , , , , . Bookmark the permalink.

One Response to DxOMark and advanced D-SLRs

  1. Pingback: DxOMark Sensor for benchmarking cameras | Peter.vdHamer.com

Leave a Reply

Your email address will not be published. Required fields are marked *