DxOMark @ 269 cameras

DxOMark score versus (a) sensor size, (b) price, (c) launch date, (d) MPixels. Click on image to view larger.

Here is another update about camera sensors in cameras as tested by DxO Labs (www.dxomark.com). DxO’s test essentially covers the noise and dynamic range of cameras – it doesn’t cover resolution, focus speed, ease-of-use, etc.

Modern cameras span a range of over 70 points in this pretty rigorous benchmark. In reality the range is larger because DxO doesn’t test camera phones and other good-enough-for-a-selphie models. A 3 point difference is barely visible to specialists, 10 points is readily visible, 30 points tends to be noticed by everyone.

Since my previous posting, 18 new cameras have been tested by DxO Labs. I also rescaled the graphs to allow for scores above 100 and extended the timeline somewhat. For an in-depth explanation of what you are seeing here, check my January 2013 article at either DxO Labs or Luminous Landscape.

Some of the highlights hidden within the newer benchmark results:

  • Epic Dragon scores 101 points
    The 19 MPixel Epic Dragon video camera can take better stills than any existing still camera. This is especially usual given that the Epic Dragon has a moderate sensor size. Because it is only a prototype, DxO did not include the Epic Dragon in its normal test result database. After all, you can’t buy it.
    Note that the Epic Dragon is a component of a modular video system: without a lens, we probably wouldn’t recognize it as a camera.
    But its score demonstrates that today’s APS-H (1.3x) and larger sensors can break the psychological 100 point DxOMark barrier. So it is only a matter of time before we see commercial cameras scoring over 100 points in this benchmark. This may be asking too much for the expected Canon 7D Mark II with a 1.5x sensor, but could be achievable for one of the new 50 MPixel medium format cameras with a Sony sensor (Hasselblad, Phase One, Pentax).
  • Mirrorless
    Some of the top models (highest performance in its price class) are now mirrorless cameras. Somehow these are all Sony models (A3000, A5000, A6000, A7, A7r). Sony is currently the leading sensor manufacturer. Manufacturers like Nikon use Sony sensors mainly to boost the image quality of their existing SLR product lines.
  • Sony A7s
    The Sony A7s has a very low resolution (12 MPixel) for a full-frame camera launched in 2014. Its score is a bit low for a modern-full frame sensor (except for its High ISO subscore). From a still camera perspective it is not obvious why Sony introduced a sensor with such large pixels: it already supplies higher image quality at with higher resolution sensors (e.g. A7 and A7r). The answer to the puzzle may be probably related to the A7s unique 4K video capabilities. I wouldn’t be surprised if the A7s is also partly intended to test the market.
  • Nokia Lumia Pureview
    The Nokia’s Pureview smart phone models caught quite some attention with their 41 MPixel resolution. They score somewhat lower than cameras with a similar sensor size (e.g. Fujifilm X10). This is likely because the pixel size of just over 1 μm gives fill factor issues: a non-negligible percentage of the sensor area is lost as overhead and not used for light gathering.


DxOMark @ 251 cameras

DxOMark Sensor scores for 251 cameras (click to view larger)

Here is another update about new cameras tested by www.dxomark.com. The test only looks at the noise and dynamic range performance of cameras – it doesn’t cover resolution, speed, ease-of-use, durability, etc.

Since my previous posting, 5 new cameras have been tested by DxO Labs. Modern cameras span a range of over 60 points. A 3 point difference is barely visible to specialists, 10 points is readily visible, 30 points tends to be obvious even when someone is not paying attention to image quality at all:

  • Sony A3000/A5000 (78 and 79 points).
    The pricing of APS-C system cameras with a state-of-the-art sensor has dropped below US$ 500 with the introduction of the Sony A3000 and A5000. Despite the Alpha branding, these are basically NEX models (Sony has dropped the usage of the NEX brand). They thus have E lens mount (as used in the NEX series) rather than the A-mount (as used in the Alpha 77).
  • Leica S medium format (76 points).
    The Leica S medium format camera, despite its $28,000 price,  does not really have a state-of-the-art sensor. It “still” uses a CCD sensor technology, although  recently medium format models with a Sony-built CMOS sensor have been recently announced (by Hasselblad, Phase One and Pentax). CMOS sensors should manage to make medium format cameras more all-round cameras again. Arguably, because medium format cameras are often used in studios or tripods, they historically had more emphasis on resolution, color fidelity and lens quality than on low light or high dynamic range.
  • Leica X Vario (78 points).
    Leica also gives you the option of buying the X Vario which actually performs similarly to the Sony A3000 or A5000, but at a Leica price.
  • Olympus Stylus 1 (51 points).
    The Olympus Stylus 1 scores surprisingly low for a new camera with its SLR-like looks. But looks are misleading here. If you look carefully at the specs, it turns out to have a very small sensor with a 4.66x crop factor. This puts in in the same league as the Canon Powershot S120. The Stylus 1 (51 points) is outperformed by the more compact S120 (56 points).
Canon Powershot S120 next to the Olympus Stylus 1 (www.camerasize.com)
Canon Powershot S120 and the Olympus Stylus 1 have the same sensor size (www.camerasize.com)

State of the DxOMark (camera nerdiness)

DxOMark scores as of Nov 2013 (click to view larger).

I updated my overview chart with available high-end cameras. See http://www.dxomark.com/Cameras/Camera-Sensor-Database for detailed benchmark results for specific cameras. As usual, there is a lot of interesting information in such an overview:

  • Assuming you care about low light and high dynamic range performance, the best cameras have full-frame sensors (the blue dots). You knew that – right? Well, surprisingly full-frame sensors beat even larger (purple, pink, red) sensors. So don’t bother spending big money on a medium format camera unless you really need the super-high resolution. Or need it show that your equipment is clearly on a price class of its own.
  • The so-called APS-C cameras with 1.5x or 1.6x sensors have improved. Examples: the Nikon D5200 and D5300.
  • The Sony NEX-5R mirrorless (which is 1.5x) has a slightly higher price than an APS-C SLR, but the body is smaller and the performance is competitive. Mirrorless models should have the same performance as an SLR with a comparable sensor. A mirror doesn’t add image quality – it just makes a click sound like ke-lick.
  • Canon still has a long way to go to catch up with its APS-C sensors (1.6x). The Canon 70D performs slightly better than the old Canon 7D, but a comparison to Nikon or Sony tends to be embarrassing.
  • Recent Micro Four Thirds cameras (Olympus & Panasonic) have improved and are even ahead of Canon’s APS-C (1.6x) models.
  • The Sony RX100 and RX100-II are still doing fine – at least considering their small sensor size (2.7x or 1/1″ ). The Nikon Series 1 is technically not state-of-the art, but nice if you like white or pink gear: it targets a young Asian lifestyle market.
  • The premium pocket cameras have improved. Especially the 1/1.7″ sensor models such as the Canon Powershot S120 and G16 and their Nikon equivalents.
  • The best deals if you need a high quality model can be found at the top edge of the cloud in diagram “b”: you get the highest quality in that price range. Note that the prices shown are official prices at introduction, and will differ from current street prices. These deals include:
    • The Nikon D600 and D610. These are essentially the same camera, but the D610 resolves a dust issue.
    • The new Sony A7R mirrorless. Note that this model uses Sony E-mount lenses, but actually requires new Sony full-frame E-mount lenses called “FE”. So it will take a while until there are enough lens options.
    • The Sony RX1 and RX1R. These look overpriced (and probably are – although I ordered one myself), but their price does include an excellent 35mm Zeiss f/2.0 lens. On the other hand, they do not come with an optical or electronic viewfinder. These cost about 500 US $ or Euro extra. Lens hood pricing is  joke (so look into the Photodiox accessories).
    • The Nikon D5200 or D5300. Both have a 24 MPixels state-of-the-art sensor, but the newer one gives sharper images (no AA filter) if your lenses are up to the challenge.
    • The Nikon D3200. Also 24 MPixels with state-of-the-art sensor technology.
    • The Pentax K50 and K500. A somewhat overlooked brand.
    • The Nikon Coolpix P330. A “take me everywhere” camera at a lower price point than the excellent Nikon Coolpix A or FujiFilm’s X-100s models.

Note that some major new camera models are not shown because DxO Labs simply hasn’t tested them yet. These include:

  1. The new full-frame Nikon Df (with the professional Nikon D4’s 16 Mpixel sensor). It should score about 89 (D4) for $3000 – nice, but not sensational unless you insist on a retro look and feel.
  2. Most FujiFilm X-Trans models have not been tested. Tests may be delayed because they have a non-standard color filter array (complicating raw conversion). The CFA design allows the sensor to work without a low pass filter. Alternatively, the missing tests may be because FujiFilm is not enthusiastic about their cameras’ DxOMark scores (pure speculation on my part, but the FujiFilm X-100 didn’t score exceptionally well).  FujiFilm high-end cameras are getting a lot of attention from serious photographers who prefer small, unobtrusive cameras with a classic mechanical feel.
  3. The Sony A7. Many people wouldn’t really benefit from 36 MPixels (Sony A7R) without an image stabilizer or a tripod or high-end lenses.

For a detailed explanation of what the benchmark itself means, see http://www.luminous-landscape.com/essays/dxomark_sensor_for_benchmarking_cameras2.shtml. Note that the number of tested cameras has meanwhile increased from 183 to 236 models.


DxOMark and advanced D-SLRs

The high-end sensor arms race: who is winning?

Michael Reichmann (of the popular site Luminous Landscape) recently concluded that the latest and greatest Nikon and Canon cameras are now indistinguishable as far as sensor noise in concerned:

I see no significant noise difference between the Nikon D3x, Canon 5D MKII and Sony A900 up to and including ISO 800. [..] From ISO 1600 to ISO 6400 the Canon 5D MKII and the Nikon D3x are neck and neck. I think that the visible differences on-screen at 100% are a quibble, and on prints are completely irrelevant.

Reichmann’s review covers the latest (Q4 2008) “full-frame” models with a resolution of 21.1 MPixel (Canon 5DM2) and 24.5 MPixel (Nikon D3x). He also included the Sony A900, which includes a 24.5 MPixel sensor. That sensor is similar to the one Sony supplies Nikon for the D3x. Reichmann’s review is based on visual comparison of images taken at varying ISO speeds of a color test chart with some surrounding props.


If you study the supposedly more rigorous/scientific measurements on sensor performance provided by DxOMark, the conclusions appear to be comparable (the Nikon D3x has yet to be tested):

  • Canon and Nikon have similar performance on recent cameras. In fact, in one graph the Nikon D3 and Canon 1Ds3 markers coincide.
  • The site shows that full-frame sensors are indeed better than the more common 1.5x or 1.6x (“APS-C”) sensors. This is interesting, because most discussions of sensor format have trouble defining what a fair comparison criterion would be – given that the same lens will give a wider image with a larger sensor.
  • Marketing-wise Canon is putting more emphasis on high resolution (their lineup contains 3 models which are 15 MPixel or higher: 15 Mpix 50D, 21 Mpix 5D2, 21 Mpix 1Ds3).
  • You will find that the 12 MPixel Nikon D3 is lower noise (better at high ISO) than any Canon product*. This is presumably because dividing a large pixel into two smaller ones will waste some silicon area and thus performance: you will get more noise. This extra noise can only be partially compensated after taking the picture by downscaling the resolution using software.** This phenomenon is, however, more subtle than is often claimed, and DxO has an interesting, but tough white paper on this.

But the basic message is simple enough: apparently that Canon has caught up with Nikon’s recent lead that was created with the release of the Nikon D3 (12 MPixel) and D700 (12 MPixel). And according to Reichmann, the Canon D1s Mark III (21 Mpixel, not shown) is very comparable to the lower cost Canon 5D2 (21 Mpixel) and thus to the Nikon D3x (25 MPixel). So Nikon’s lead was probably a combination of a technical success (Nikon caught up and even overtook Canon) plus a marketing success: the Canon 1Ds Mark III was launched 1.5 year earlier than the Nikon D3x, both seem to be comparable.

DxOMark methodology

Surprisingly, if you look at the raw performance-versus-ISO graphs at the DxOMark site, you get a different perception of the cameras than by looking only at the final DxOMark index.

Let’s first look at the real measurement data by DxO. I somewhat arbitrarily chose this example. Check out all 4 benchmarks provided by DxO: sensor noise, dynamic range, tonal range and color sensitivity. The respective Nikon and Canon top scorers (12 MPixel D3 and 21 MPixel 1Ds Mark III) are shown, along with a 5-year older model (6 MPixel Canon 10D) which I added essentially to put any differences between comparable camera’s in perspective. It is important to view the graphs in DxOMark’s Print mode rather than Screen mode. This simulates performance when images are compared at the same viewing size (like a print) by compensating for any resolution differences between sensors: noise of smaller pixels will be smaller dots on paper, and thus be less visible than bigger dots.

Canon 1Ds3 versus Nikon D3 (noise figures in "Print" mode)

So far, so good. The problems occur when DxO tries to simplify all this somewhat complex multi-dimension information (remember that there are 4 of these graphs) to a single number. The end-users tends to like a single number because it gives a clear winner. This is why DxO introduced the DxOMark concept, with a clear nod to computer hardware benchmarks that compare speed (3DMark, etc). Although benchmarking of computer speed is a nasty enough problem because the outcome depend somewhat on what you need to compute, benchmarking of image sensors seems even trickier:

  1. The parameters measured by DxO depend significantly on the amount of available light. In the above graph, the signal to noise ratio varies by 100:1 essentially because real world light intensity easily varies by at least 100:1. This implies that on a sunny day noise is not an issue (and to its credit DxO explains this on their site): in studio or for a landscape photographer at 100 or 200 ISO, noise is nowadays invisible (and incidentally better than film ever way). But for a sports or concerts, noise remains an issue.
  2. A single figure of merit needs to combine different sensor properties: noise, dynamic range, color accuracy, etc. Again, different users have different priorities. So this is like judging cars with a single number: speed, fuel efficiency, comfort all go into one number. This is a flawed approach. But to DxO’s credit, you get all the raw info (like the graph above and 3 more graphs like it) and DxO warn about how to interpret the data.
  3. A nasty problem with DxOMark’s methodology is what to do about image resolution: in the above example, we are comparing a 6 MPixel (10D, 2003), a 12 MPixel (D3, 2007), and a 21 MPixel (1Ds3, 2007) model. When viewed in Print mode on the DxO site (and as copied above), the values are made comparable by computing what the result would look like when scaled to a given size image viewed from the same distance (like when you print).
    This is fair to judge the noise, dynamic range and color behavior. But it ignores resolution itself. In fact, you can improve noise at the cost of resolution (by averaging and other forms of filtering). As far as I am aware, the DxOMark figures (both the graphs and the single index) do not reward a camera for having a high resolution. In fact, a high-resolution camera is actually punished a bit for being high resolution, because high resolution makes it difficult to achieve the metrics which DxOMark does include in their assessment.
    Hypothetically I could take any camera, reduce the resolution to 1 pixel by averaging all the sensor pixels. This would give extremely low noise (essentially unmeasurable: probably a 30dB improvement over the original sensor) but detail would be terrible. So the question remains: is it fair to compare camera sensors as if resolution were entirely irrelevant? Before some people say “yes, but beyond a point resolution is irrelevant”, remember that a 21 MPixel full frame sensor has the same pixel density as an 8 MPixel 1.6x camera.

Single-value DxOMark score?

So, to summarize, a single number DxOMark figure-of-merit is risky. If you have the technical skills, it is better to study the multi-dimensional raw data. If you know what image quality aspect is most important to you, DxO even has hints which of its graphs is most important.

So much for the theory. Let’s look at the actual DxOMark index scores for the cameras shown in the graph reproduced above:

  • Canon 10D (Feb 2003): 56.6 points
  • Canon 1Ds3 (Aug 2007): 80.3 points
  • Nikon D3 (Aug 2007): 80.6 points

DxO does not explain how the measurements from the 4 graphs are computed into a single score. But the ratios of 70%:100%:100.4% do not seem to reflect the raw data. This needs clarification. Firstly because there may be a misunderstanding about the methodology. One possible explanation is that the Nikon D3 is somehow “punished” for lacking a 100 ISO mode: at 100 ISO, the Canon tends to exceed the Nikon which cannot go below 200 ISO.


* Maybe the future Canon 1D Mark IV should be full-frame 12 MPixel (simplifies the product range and marketing) instead of 1.3x 10 Mpixel.

** DxOMark compensates for differences in resolution if you put it in Print mode (all MPixels squeezed into fixed print). Screen mode is what you see at 100% crop (which is a silly way to benchmark as it punishes high-res sensors in an unreasonable way).

[ this posting is longer than 1000 words: 1353. Needs to be split?]