DxOMark Camera Sensor article (v2.0 and v2.0.1)

I have been working on an update to my original DxOMark article. That update has just been published on Luminous Landscape, a well-known photography site operated by the Canadian landscape photographer and publicist Michael Reichmann.

A slightly newer version of the article is available at the DxOMark website. It features four extra cameras and almost identical text.

Sample DxOMark results

The article covers various aspects of image sensor size and its impact on image quality. The article is built around original benchmark data measured by DxO Labs. I have rehashed their data (with permission) to stress basic trends and highlight a few topics:

  • Benchmark data for over 180 high-end cameras (starting at about $400).
  • Which benchmark numbers by www.dxomark.com are most relevant for your needs?
  • The technical relationships between sensor noise, dynamic range and resolution.
  • A comparison of what noise does at low ISO and at high ISO (this is trickier than “doubling the ISO reduces the signal-to-noise ratio by 2Ă—”).
  • The implications of using “mirrorless” cameras (and associated smaller sensors) on image quality.
  • The image quality of the new wave of cameras that use Sony’s new Exmor sensor with its excellent low ISO dynamic range performance. With a bit of speculation of whether Canon (that normally doesn’t use Sony sensors) can catch up with Nikon (that does regularly use Sony sensors).

You can contact me about the article via comments on this website. I will also try to keep an eye on comments on the LuLa and the DxOMark fora (forums).

16 thoughts on “DxOMark Camera Sensor article (v2.0 and v2.0.1)

  1. John Reche

    Thank you for your great article on the Luminous Landscape. It is unusual to read well backed technical papers on most “photo” web sites.
    It would have been interesting, but may be adding too much additional complexity, to include the Foveon sensors.
    Best Regards,
    58 years practicing photography from huge overhead photolithography cameras down to Minox.
    - John Reche

    Reply
    1. pvdhamer Post author

      Thanks, John. I heard good things about Sigma’s “Merrill” family that uses Foveon sensors. But somehow DxOMark has no data for any Sigma camera. Furthermore, the article mainly discusses technologies required to explain trends or exceptional cases in the data (I also didn’t mention BSI or even CCD versus CMOS). Foveon sensors are reportedly sharp (out of scope) due to the lack of an AA filter and might have unusually low ISO color sensitivity. If there had been Foveon measurements, I would have the answer the question of how many pixels these sensor have ;-)

      Reply
  2. Phil Galbraith

    The sensor article on LuLa is indeed excellent and I shall spend some time checking out the quoted references. I now look forward to looking at your website with which I was previously unaware.
    One small point is that in note 17 you suggest that the four thirds format is named in relation to the aspect ratio. Surely it is named using the same convention as other sensors i.e. in relation to the standard for a 1″ camera tube with an image diagonal of 16mm. The 21.6 mm diagonal of four thirds is amost exactly four thirds of 16mm.

    Reply
    1. pvdhamer Post author

      Thanks Phil,

      I checked, and Four Thirds seems to denote both size and aspect ratio. I will fix the endnote. I guess the original reports about the format preferred not to discuss imaginary glass spheres, but somebody figured out the naming later. Even the logo is closer to a ratio than a rational number for an irrational measure.

      My website only contains occasional photography-related articles. I can’t produce articles like this too often.

      Peter

      Reply
  3. Thomas M

    Peter,

    thanks a lot for your very interesting article. Your are pointing out that sensor performance is intimately related to manufacturing technology, some brands are lacking behind but can catch up in the future. What can we expect? The readout noise at the lowest ISO settings are close to one electron, the quantum efficiency has reached 60 %. I would expect that the best present sensors ( e.g. Nikon D800) are close to physical limits. Do you see any room for major improvements similar to that what we have seen in the last years? If yes, what would you expect and on which time scale?

    best regards

    Thomas

    Reply
    1. pvdhamer Post author

      What’s coming depends on physical limits, but increasingly on whether there is a market big enough to justify even more extreme designs. I don’t use the term “extreme” in the article, but it may help to think of a full-frame sensor, a milled down BSI sensor, or a chip with thousands of simultaneous A-to-D convertors “extreme”. But the likely answer is that we have just experienced a non-sustainable innovation spurt and will see diminishing returns from now on.

      But based on the performance of the Pentax Q, I think it is reasonable to expect that APS-C and full-frame performance gets another stop improvement (due to scaling). After all, just tiling the Q’s sensor should do the trick according to Fig 6. If the resolution is not too high, it might be possible without backside illumination (pricey/hard for big chip).

      In theory you could get DxOMark’s overall score above 100 by simply scaling the sensor and lens to medium format size. But the medium format makers are not really interested in low noise and high dynamic range: their market is small and who wants a giant camera with a giant lens for street or sport photography?

      You could alternatively increase Dynamic Range if technologists manage to increase full-well capacity (per unit area). I am not aware of fundamentals stopping you there, but it will require innovation at the device level (3D integration?). The market for this might be large enough: not those who process multiple raw files on a desktop computer, but mass users who use single shot HDR and camera vendor supplied algorithms within the camera (resulting in one or two JPG images).

      Actually, the people who can best answer your crystal ball question are full-time sensor experts who have worked on this topic dozens of years. Such people exist in industry (they get paid to use their insights for planning their company road maps). The alternative breed is somewhat more accessible: top names in sensor design in academia give occasional key note presentations. Here are two links which Prof. Eric Fossum sent me after he saw my article:
      http://www.youtube.com/watch?v=JkBh71zZKrM
      http://ericfossum.com/Presentations/2012%20March%20QIS%20London.pdf

      Reply
  4. Thomas M

    Peter,

    thanks a lot for the detailed reply and drawing my attention to the lecture by Prof. Eric Fossum. From all the information I got the impression that we can expect one stop better sensitivity/ noise level, but more or less that’s it. How long this will take? Does that mean that the pixel race is over? The new concepts discussed in the Prof. Fossums lecture are at the present state ideas, and I got the impression, that it not clear at all whether consumer cameras will benefit from them.

    best regards

    Thomas

    Reply
  5. Alex

    Thanks for an informative article.

    I could not agree, though, with the statement about decreasing DoF when switching from 105mm/2.8 to 150mm/2.8 lens. The DoF definition relies on definition of the circle of confusion which is measured in units of length (eg 0.05mm on a 35mm film). When you scale up the imaging geometry, everything scales up in size, but angles remain the same. So, sensor size, focal length, aperture diameter, and circle of confusion will all scale up. Circle of confusion with a certain distance to subject will determine the angular size of the “feature of confusion” at that distance. Then, the imaging geometry is scaled up and the goal is to keep the image characteristics the same, the angular size of the “feature of confusion” must stay the same, thus leading to an increase in the circle of confusion size and the same DoF if the f-number stays the same.

    Reply
    1. pvdhamer Post author

      Alex,

      I stick to the statement that 150mm f/2.8 has the same DoF as a 105mm f/2.0 rather than 105mm f/2.8 – obviously assuming the experiment is done in such a way that you are taking a picture with the same field of view and composition, and “printing” both images with the same size.

      You can confirm this in a number of ways. (1) Create you own set of equations and calculate what the actual DoF will be. Your model would need to cover more than just sharpness criteria: you need to calculate how the criterion translates to DoF. (2) Use one of the many DoF calculators on the web. (3) Examine an extreme case (http://500px.com/photo/11594571) where the image was capture (by a friend of mine) on 27×27 centimeter film using a 19th centrury 420mm f/4.5 lens. The DoF is below 1 centimeter: the photographer focused on the eyes, but the nose and even forehead are out of focus. Your claim would predict that I could get the same DoF with a 2.7×2.7 cm sensor (a square version of full-frame) using a 42mm lens stopped down to f/4.5. Lenses like that are common (Canon just release a nice 40 mm f/2.8 pancake lens). In contrast, as explained in my article (summarizing work by Falk Lumo) I predict that you would need an exotic 42mm f/0.45 lens to take the same picture on a 2.7×2.7 cm sensor (or film). That is quite a different lens. Actually there seems to exists (a few copies of) a Carl Zeiss Super-Q-Gigantar 40mm f/0.33 lens. That lens, on a full-frame digital or analog camera should enable you to create the same kind of images as the collodion “wet plate” image. But a 40mm f/2.8 or f/4.5 lens will have shallower depth of field. Alternatively you could compare a camera phone lens with a full frame lens. When the field of view angles are equivalent, the mini sensor will produce a much larger DoF than the full frame camera when both are used at say f/2.8.

      Reply
      1. Alex

        Peter,

        You are right. I did write my own formulas (should have started with that before posting) and DoF does decrease with focal lengths even though the circle of confusion scales up.

  6. Gerard Turnley

    Hi Peter,

    Thank you for a very informative article on DxOMark as a starting point for photographers investing in new equipment.I used DxO when investing in my new Nikon D3X and have been extremely happy with the results I have managed to achieve
    with my first Nikon having used Canon for the last twenty years.
    What I am struggling to understand is why Hasselblad does not have all its various digital backs tested on DxOMark. When I approached a Hasselblad enthusiast and queried their lack of representation I was informed that certain tests are bias and Hasselblad refuse to comment.

    Reply
    1. pvdhamer Post author

      I don’t have any inside information why Hasselblad doesn’t push DxOMark to have its backs tested. A company seems to have influence on this: as high scoring cameras get tested on average surprisingly soon after release. And weaker models get tested later (maybe because the company was not eager to supply a loan camera; maybe because the company challenged the validity of the test results prior to publication).

      On the other hand, medium format cameras score very well on low iso color noise, but are poor at low iso dynamic range and at high iso usage. You could argue that for landscapes the latter is not too important. And for studio work the latter two are not too important. I am still not sure when low noise color noise is really important. But high resolution might be a benefit. Obviously there are multiple reviews comparing the Nikon D800E against medium format cameras.

      Reply
  7. Sebastian

    Hi Peter,
    thanks a lot for this great, detailed article!

    I did some calculations to make sure I understood everything when comparing sensor quality of different sensor sizes :-)

    E.g.
    Nikon D800E (Sensor: 24.0 x 35.9mm = 861mm2)
    Nikon D5200 (Sensor: 15.7 x 23.6mm = 370mm2 = 2.32x smaller than Fullframe)
    Olympus OM-D E-M5 (Sensor: 13.0 x 17.3mm = 225mm2 = 3.83x smaller than Fullframe)

    When comparing the DXO Mark Low Light scores (I think it should be better to compare the SNR 18% value at e.g. 30dB, but very hard to pick those values from the graphs):
    Nikon D800E = 2979
    Nikon D5200 = 1284
    Olympus OM-D E-M5 = 826

    When taking the sensor size into account:
    Nikon D800E = 2979
    Nikon D5200 = 1284 * 2.32 = 2985
    Olympus OM-D E-M5 = 826 * 3.83 = 3164

    So on a first view the scores for the D5200 and OM-D look bad when comparing against the D800E. But after considering the sensor size all of them end up in the same league
    (and also taking “Table 1. Five theoretically indistinguishable camera configurations” from your article into account).

    Are there any mistakes in my thinking above?

    best regards
    Sebastian

    Reply
    1. pvdhamer Post author

      Hi Sebastian. What you are calculating is comparable to the scaling laws shown by the colored lines in Figure 6:
      you are showing that all three are essentially state-of-the-art cameras and that the differences in performance are well-explained by their differences in surface area. Whether that is being “in the same league” is arguable: the D800(E) performs better than the smaller Olympus sensor – if you use lenses with identical apertures on both. The D800(E) has the same noise performance as the Olympus if the latter is paired with a 4x faster lens. Peter

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>