Tag Archives: camera

DxOMark @ 251 cameras


DxOMark Sensor scores for 251 cameras (click to view larger)

Here is another update about new cameras tested by www.dxomark.com. The test only looks at the noise and dynamic range peformance of cameras – it doesn’t cover resolution, speed, ease-of-use, durability, etc.

Since my previous posting, 5 new cameras have been tested by DxO Labs. Modern cameras span a range of over 60 points. A 3 point difference is barely visible to specialists, 10 points is readily visible, 30 points tends to be obvious even when someone is not paying attention to image quality at all:

  • Sony A3000/A5000 (78 and 79 points).
    The pricing of APS-C system cameras with a state-of-the-art sensor has dropped below US$ 500 with the introduction of the Sony A3000 and A5000. Despite the Alpha branding, these are basically NEX models (Sony has dropped the usage of the NEX brand). They thus have E lens mount (as used in the NEX series) rather than the A-mount (as used in the Alpha 77).
  • Leica S medium format (76 points).
    The Leica S medium format camera, despite its $28,000 price,  does not really have a state-of-the-art sensor. It “still” uses a CCD sensor technology, although  recently medium format models with a Sony-built CMOS sensor have been recently announced (by Hasselblad, Phase One and Pentax). CMOS sensors should manage to make medium format cameras more all-round cameras again. Arguably, because medium format cameras are often used in studios or tripods, they historically had more emphasis on resolution, color fidelity and lens quality than on low light or high dynamic range.
  • Leica X Vario (78 points).
    Leica also gives you the option of buying the X Vario which actually performs similarly to the Sony A3000 or A5000, but at a Leica price.
  • Olympus Stylus 1 (51 points).
    The Olympus Stylus 1 scores surprisingly low for a new camera with its SLR-like looks. But looks are misleading here. If you look carefully at the specs, it turns out to have a very small sensor with a 4.66x crop factor. This puts in in the same league as the Canon Powershot S120. The Stylus 1 (51 points) is outperformed by the more compact S120 (56 points).
Canon Powershot S120 next to the Olympus Stylus 1 (www.camerasize.com)

Canon Powershot S120 and the Olympus Stylus 1 have the same sensor size (www.camerasize.com)

DxOMark @ 246 cameras

DxOMark Sensor scores for 246 cameras (click to view larger)

DxOMark Sensor scores for 246 cameras (click to view larger)

See http://peter.vdhamer.com/dxomark_nov13/ for the previous snapshot and some explanatory text.

In this January update, ten new cameras and four extra labels were added. Sorted on descending price, these are:

    • Sony A7 (a 24 MPixel full-frame mirrorless camera)
    • Sony DSC-RX10 (performs like Sony’s two RX100 models)
    • Sony NEX 5T (likely the last NEX-branded model)
    • Sony A3000 (great image quality at low cost)

Note that all four happen to be Sony. I add labels in the (a) graph  to cameras that are notable from a technical perspective, and to the (b) graph when they have an interesting price for their performance level.

Although not tested yet, it will be very interesting to see how the two medium format backs perform with Sony’s new 50 MPixel sensor. They might becoming the new record holder (although the Phase One IQ250 doesn’t allow you to shoot about 6400 ISO).


DxOMark Camera Sensor article (v2.0 and v2.0.1)

I have been working on an update to my original DxOMark article. That update has just been published on Luminous Landscape, a well-known photography site operated by the Canadian landscape photographer and publicist Michael Reichmann.

A slightly newer version of the article is available at the DxOMark website. It features four extra cameras and almost identical text.

Sample DxOMark results

The article covers various aspects of image sensor size and its impact on image quality. The article is built around original benchmark data measured by DxO Labs. I have rehashed their data (with permission) to stress basic trends and highlight a few topics:

  • Benchmark data for over 180 high-end cameras (starting at about $400).
  • Which benchmark numbers by www.dxomark.com are most relevant for your needs?
  • The technical relationships between sensor noise, dynamic range and resolution.
  • A comparison of what noise does at low ISO and at high ISO (this is trickier than “doubling the ISO reduces the signal-to-noise ratio by 2×”).
  • The implications of using “mirrorless” cameras (and associated smaller sensors) on image quality.
  • The image quality of the new wave of cameras that use Sony’s new Exmor sensor with its excellent low ISO dynamic range performance. With a bit of speculation of whether Canon (that normally doesn’t use Sony sensors) can catch up with Nikon (that does regularly use Sony sensors).

You can contact me about the article via comments on this website. I will also try to keep an eye on comments on the LuLa and the DxOMark fora (forums).


Notes on a tutorial about image sensor noise

During late 2009 and all of 2010, Albert Theuwissen published an 26-part series of postings on image sensor noise on his Harvest Imaging website. The series explores various sources of image sensor noise and their relationship to signal strength. The series targets sensor designers, and those who use sensors in challenging applications. Probably many of the Harvest Imaging readers know Prof. Theuwissen from his courses, workshops and conferences.

The series centers around a proprietary simulation model (written in Matlab/C?) where Theuwissen selectively isolates each noise source encountered in a sensor to show its impact on overall image noise. Every installment of the series centers around a graph (James Janesick’s Photon Transfer Curve, PTC) that plots sensor noise against sensor signal. The graph is thus closely related to the signal-to-noise ratio, but the graph stresses how the ratio varies

  • as the sensor is exposed to darkness for varying durations, and
  • as the sensor is exposed to light for varying exposures.

If you are not a sensor expert, you can try to use the series to learn about sensor behavior – provided you can handle a bunch of basic formulas and are willing to learn the associated terminology (which is is not entirely consistent across postings). Hence these notes which try to follow the terminology used in later postings (where up to 15 noise sources needed to be distinguished).

Incidentally, the word “Harvest” in the domain name of Albert’s website is after the title of a Neil Young album: Prof. Theuwissen is somewhat of a Neil Young fan.


  • CDS = correlated double sampling (a kind of self-calibration technique with differential amplifiers, see Wikipedia)
  • DN = digital number (simply the digital value read out after measuring the analog signal)
  • DSNU = Dark Signal Non-Uniformity (differences in dark current signal build-up due to variations between individual pixels)
  • FPN = fixed pattern noise (small pixel-to-pixel deviations that don’t change over time)
  • k = gain in DN/electron
  • PRNU = photo-response non-uniformity (differences is pixel sensitivity to light)
  • PTC = photon transfer curve = signal versus noise graph
  • RTS = random telegraph signals (random jumping between fixed output levels)

Overview of sources of dark noise (see also posting p=358)

The following table lists all the noise sources that occur in absolute darkness. They also occur when there is light (but with light there are extra noise sources). They are discussed in more detail below.

All noise sources are measured by resetting the pixels, and then reading out the pixel after a short or longer delay. I classified the noise sources based on their time behavior (the table columns) and their source (the table rows). The central message of the Harvest Imaging series is that you can distinguish these noise sources in actual measurements by analyzing noise build-up over time (to distinguish the table columns) and by different ways of averaging the individual pixel measurements (to isolate fixed sensor line and column pattern noise).

The conclusion of the series is that you can distinguish many of the noise sources by appropriate measurements on a sensor. And that the estimated parameter values can be pretty accurate.

Fixed-Pattern Noise Non-uniformity
Temporal Noise Shot Noise (Poisson statistics)
Scales with Constant value time or dark current
Constant std dev Sqrt(time) or Sqrt(dark signal)
Pixel-level Pixel defects (p=329) & RTS (p=344) & Pixel FPN (p=358) Dark Signal Non Uniformity (p=84) Temporal
pixel noise
Dark current shot noise (p=48)
Output-level Amplifier offset (p=154) Output amplifier (p=263)
Row-level Row FPN (p=243) Temporal
row noise
Column-level Column FPN (p=229) Temporal
column noise

The Amplifier offset (p=154) has such a bad effect on low signal measurements that it is assumed to be corrected away in most of the PTC graphs.

A noise source not listed above, Saturation Non-Uniformity (p=142), is only relevant for severely overexposed pixels. This can happen during normal exposures, but this part of the dynamic range is normally hidden from the user because of its non-linearity and non-uniformity.

Careful: there may be multiple definitions of Temporal Pixel Noise: including or excluding temporal row/column noises. When you just measure the pixel noise, you get “including”, but when you do a lot of analysis or create a synthetic model, you get “excluding”. A similar problem may exist for pixel-level FPN.

The sample sensor used in all computations

Calculations are done on a hypothetical 160×120 pixel sensor. Given the assumed full-well capacity of 17,500 electrons, the pixels may have a pitch of around 3-5 μm. So the data would correspond to a small section of a larger sensor with pixel dimensions that are likely between compact camera pixels and SLR camera pixels. The physical dimensions are not directly relevant for any of the calculations.

1. Dark Current Shot Noise (p=48)

  • Dark Current is what a sensor sees during long exposures with no light at all.
  • Signal proportional to exposure time (leakage?)
  • Noise = sqrt(dark signal) ; Poisson
  • measurement of PTC can tell you k of the system (here 0.15 DN/e-)

2. Dark Signal Non-Uniformity (p=84)

  • Noise to pixel having varying levels of dark current (non-uniformity)
  • Noise = fraction of Dark Signal (here 15%)
  • careful: the terminology switches back and forth between FPN and DSNU
  • later in the series this is simply known as DSNU

3. PTN Curve and Temperature (p=120)

  • Dark Signal and its noise both scale exponentially with temperate
  • The PTN graph shows their ratio, and is thus temperature independent

4. Anti-blooming (p=142)

  • Anti-blooming is a kind of safety value for pixels that overexposed.
  • It causes non-linear response above what the sensor considers to be white.
  • The threshold for anti-blooming introduces significant FPN
  • In the anti-blooming range, noise drops because temporal noise is clamped.

5. Amplifier Offset (p=154)

  • Offsets “in analog circuitry” can mess up readings, especially of low DN values.
  • They simply need to be (accurately) compensated

6. Pixel Noise (p=211)

  • At zero integration time, there should be 0 dark current signal and 0 DSNU noise.
  • In reality, there is signal due to imperfect offset compensation.
  • And noise because there is temporal noise in the analog chain.

7. Column FPN and Temporal Noise (p=229)

  • Columns (by def.) share the same bias and readout circuitry. Can thus have FPN.
  • Column FPN will exceed dark current noise for short enough (e.g. 1 second) integration times
  • Can isolate Column noise by averaging all pixel FPN across all pixels in a column

8. Row Noise (p=243)

  • Defined as row-to-row variation in the average of all pixels in a row.
  • Row FPN noise may repeat very N (e.g. 16) columns. No explanation. Timing?
  • Can be measured by calculating Fourier transform of noise-versus-row#.

9. Amplifier Noise (p=263)

  • Gives non-zero (e.g. 1.2 DN) noise at zero exposure time.
  • Differential amplifier has “correlated double sampling” trick.
  • CDS should reduce offset and low-frequency noise sources.

10. Defective Pixels (p=329)

  • Defective pixels are “stuck at” 0 or 1 and give extreme spikes in the image
  • Are normally compensated for by digital processing (e.g. replaced by estimate)
  • After compensation, their impact on statistics should be small

11. RTS Noise (p=344)

  • Random Telegraph Signals pixels are pixels that hop slowly between fixed levels.
  • Their cause is not really understood.
  • e.g. 5x more RTS pixels than stuck-at pixels
  • RTS noise hardly visible in the PTC curve. Why?

12. Integral Fixed-Pattern Noise (p=358)

  • isolate FPN by averaging enough frames, thus getting rid of temporal noise
  • uncorrected, defective pixels dominate the (short integration time) FPN
  • can measure Row and Column contributions by averaging over pixels & frames
  • can isolate Pixel FPN noise by subtracting (squared) Row & Column FPN noise
  • noise parameters can be estimated from the simulated images (about 10% off)

13. Integral Temporal Noise (p=386)

  • standard deviation is calculated per pixel (by using 100 images)
  • this removes all FPN effects
  • at long integration times, dark current shot noise dominates
  • at short integration times, noise from pixel/row/column electronics dominates
  • not directly possible to split temporal noise in pixel/row/column components

14. Dark image data files are available (p=417)

  • due to public demand, Theuwissen provided a 162 MByte file with all images generated by the simulations (at different integration times in the dark)
  • the download does not include the simulation model that generated the images

15. Let there be light (p=430)

  • initial simulations are without any noise sources on the silicon
  • signal is proportional to integration time (until close to saturation)
  • photons result on average in e.g. 0.3 free electron (quantum efficiency)
  • noise is shot noise and proportional to sqrt(time) or sqrt(photons)

Additional noise sources due to light

Fixed-Pattern Noise Non-uniformity
Temporal Noise Shot Noise (Poisson statistics)
Scales with Constant value time or photons Constant std dev Sqrt(time) or Sqrt(photons)
Pixel-level Pixel-level FPN (p=501), Pixel Defects (p=527), RTS Defects (p=546) Photo-response non-uniformity (p=454) Photon shot noise (p=430)
Output-level Output-level FPN (p=501)
Row-level Row-level FPN (p=501)
Column-level Column-level FPN (p=501)

Note that the Fixed-Pattern Noise column are not new noise sources: they exist (presumably with the same magnitude) in the absence of light but are measured again below in the presence of light. PRNU and Photon Shot Noise, in contrast, are really new noise sources that increase noise when photons fall on the sensor.

In addition, photons obviously also cause a signal component (which looks like dark current, but is proportional to the photon flux) which is what the sensor is meant to measure in the first place. It is not shown in the above table because it is not a noise source.

16. Photon shot noise (p=430)

  • exposure times here 0.6 s (compared to 60 s for dark current measurements)
  • linear signal response (to time=photos=electrons) until pixel saturation is reached
  • noise is shot noise (Poisson = sqrt(signal)) when all sensor noise is eliminated
  • PTC graph can be used to estimate system-level gain k

17. Photo-response non-uniformity (PRNU p=454)

  • cause is non-uniformities such as varying pixel size or varying quantum efficiency
  • noise is proportional to signal (e.g. 3%)
  • when the signal saturates, PRNU continues to increase (artifact of model?)
  • I checked with Theuwissen: due to saturation FPN

18. Combining both sources of non-uniformity (PRNU & DSNU p=472)

  • Dark signal and photo response non-uniformity look pretty much the same
  • At high light intensity (short integration times) PRNU dominates e.g. by 100x
  • And photon shot noise dominates the dark current shot noise

19. Combining both sources of non-uniformity (PRNU & DSNU p=486)

  • At low light (long integration times) DSNU becomes comparable to PRNU
  • This causes fixed-pattern noise to increase
  • To distinguish the two you need to vary light intensity
  • Turning on both shot noise sources increases the noise level and signal level
  • So the PTC curve doesn’t shift: if you know the signal, you known the shot noise – regardless of whether the signal is dark current, photons or a mix.

20. Pixel/Row/Column FPN noise (p=501)

  • FPN Noise is a sum of Pixel/Row/Column FPN noise
  • Row and Column data can be isolated by suitable averaging
  • Pixel noise can be calculated by subtracting squared row and column values
  • Looks like the Row-level noise value has a typo (not a big deal, I reported this)
  • This posting doesn’t distinguish between dark current FPN and photo-related FPN, but this could be done by varying the light intensity.

21. Temporal Pixel/Output/Row/Column noise (PORC, p=516)

  • adding Pixel/Output/Row/Column temporal noise dominates very short exposures, but is negligible at longer exposures
  • it is probably not feasible to isolate Pixel/Row/Column noise sources
  • Output noise might be measurable by measuring noise far into saturation

22. Stuck-at Pixel Defects (p=527)

  • Story very similar to the defective pixels in the dark
  • uncorrected, defective pixels dominate very brief exposures
  • they can be corrected efficiently using a processor
  • once corrected, they hardly have an impact on fixed pattern noise

23. RTS Defects (p=546)

  • RTS defects had little impact in the dark, so the same applies with light
  • Still unclear why an RTS pixel contributes at least one order of mag less noise than a defective pixel. I posted the question.

24. FPN noise estimation using the complete model (p=559)

  • 15 noise sources are included. The underlined ones are relevant when measuring fixed-pattern noise:
  1. overall amplifier offset *
  2. dark current shot noise
  3. Dark Signal Non-Uniformity
  4. saturation non-uniformity *
  5. Pixel temporal noise
  6. Output amplifier temporal noise
  7. Row temporal noise
  8. Column temporal noise
  9. FPN for pixels *
  10. FPN for rows *
  11. FPN for columns *
  12. PRNU *
  13. photon shot noise
  14. defective pixels*
  15. RTS pixels
  • until corrected, offset dominates low exposure FPN
  • the splitting of Pixel/Row/Column noise is done just like p=501
  • and results in very similar numbers (no typo this time)
  • the sources that could be estimated are marked with a *
    • DSNU would look just like PRNU in the PTC, but should be negligible
    • RTS should be negligible

25. Temporal noise estimation using the complete model (p=578)

  • 15 noise sources are included. The underlined ones are relevant when measuring temporal noise:
  1. overall amplifier offset
  2. dark current shot noise Medium
  3. Dark Signal Non-Uniformity
  4. saturation non-uniformity
  5. Pixel temporal noise Low
  6. Output amplifier temporal noise Low
  7. Row temporal noise Low
  8. Column temporal noise Low
  9. FPN for pixels
  10. FPN for rows
  11. FPN for columns
  12. PRNU
  13. photon shot noise Medium
  14. defective pixels
  15. RTS pixels
  • At low exposure to light, the sources marked Low dominate. They cannot be distinguished using the PTC. Added together they are called temporal pixel noise, but this is ambiguous.
  • At higher exposure to light, photon shot noise dominates. Dark current shot noise should be negligible for these short integration times.
  • At high exposure the pixels go into saturation and there is little noise left (you might be able to estimate the output amplifier noise if you really cared)

26. Lit image data files are available (p=588)

  • Due to public demand, Theuwissen provided a 141 MByte file with all images generated by the simulations (at different exposure times)
  • The download does not include the simulation model that generated the images
  • This is the counterpart to the dark data files provided in p=417
  • The images should look pretty much the same as the dark images (because the ones in the dark are at much longer integration times). I haven’t checked.
Header for DxOMark Sensor posting

DxOMark Sensor for benchmarking cameras

DxOMark Sensor is a raw benchmark for camera bodies. It is “raw” not just because it looks at Raw file image quality. It is also raw in the sense that it provides data for cooking up hands-on reviews that cover all aspects of a camera.

Note: a version of this article was published on Luminous Landscape on January 28th 2011.

DxOMark Sensor Scope

DxOMark Sensor is the new name of DxO’s original metric for camera body image quality. The name “sensor” is a bit misleading as the benchmark covers whatever happens to the light or signal from the point it has left the lens up to the point when the raw file is decoded. Other camera properties such as image quality, like ease-of-use, speed, price, and lens sharpness, are all out of scope.

Note that DxO also provides a second benchmark called DxOMark Score which tests lens/body combinations and which does include lens sharpness.

DxOMark Sensor applies to:

  • high-end digital cameras (mainly SLRs and interchangeable lens models),
  • when generating Raw output files (JPG introduces too many extra issues),
  • including whatever impacts image quality within the camera (except for the lens!), and
  • regardless of sensor resolution (more on this later).

The DxOMark Sensor benchmark essentially “only” covers noise under varying lighting conditions and in its various manifestations.

Purpose of the Benchmark

Benchmark data such as DxOMark Sensor give photographers a way to compare camera image quality. This helps people decide whether to upgrade or what to buy – despite that having a low noise camera is nowhere near the top of the list of things that make photos great.

Benchmarks may actually also influence future industry direction. This is analogous to, for example, automotive mileage or safety tests: even when the test definitions are not perfect, vendors will try to optimize their designs to score well on important tests.

Although DxO Labs is a commercial organization, it provides this benchmark data for free because DxO needs to measure the data anyway (e.g. for their Raw converter) and because it uses its DxOMark website to increase brand awareness. The measurements and graphs are incidentally not in the public domain, but can be redistributed under certain conditions.

Purpose of This Article

The data shown here is derived from DxOMark’s website. My graphs don’t replace DxOMark’s graphs and tables: you should use the DxOMark website to compare specific camera models. I simply created new graphs to stress certain overall trends and phenomena – originally for my own needs.

This article thus addresses various interrelated questions:

  • What do the DxOMark Sensor results mean?
  • How valid are the benchmark scores?
  • Why do large sensors outperform smaller ones?
  • Why don’t MPixels say much about image quality?
  • What can we learn about the cameras and industry from the DxOMark data?

During the journey I will slip in a basic course on Sensor Performance for Dummies. This is good for your nerd rating because it is actually rooted in quantum physics and discrete-event statistics. And I even threw in a few Greek λetteρs to remind you that we are on the no man’s land between science, engineering and marketing.

If this gets to be a bit too much for your purposes, just concentrate on the graphs containing benchmark results. Questions like “Please define photon shot noise” will not be asked on the exam.

Four Top-Level Graphs

Figure 1. DxOMark Sensor score. Click on image to enlarge.

Figure 1a-d shows the DxOMark Sensor score along each vertical axis. The scores are currently between 20 and 90. Scores above 100 are theoretically possible. Don’t get hung up on differences of only a few score points: 5 points is roughly the smallest visible difference in actual photos (DxO: “equivalent to 1/3 stop”). The measurements themselves appear to be repeatable to within one or two points[1].

The DxOMark Sensor score is itself based on three more detailed scores which we will discuss later. The graphs in Figure 1 show:

a. the impact of different physical sensor sizes on the overall score,
b. the overall score versus a price indication for the camera body,
c. how digital cameras have improved over the years, and
d. how image quality relates to sensor MPixels.

To save you some scrolling (and squinting), each of these four graphs in Figure 1 will be repeated (and enlarged) when it is discussed.

Sensor Size impacts Image Quality

Figure 1a. DxOMark Sensor scores for different sensor sizes.

This is one of the graphs shown in Fig 1.

The horizontal axis in Figure 1a represents relative sensor size. The dimensions of a “full-frame” sensor (24×36 mm) are used as reference. A value of 0.5 thus means that the sensor’s diagonal is half the size of a full-frame sensor and that the crop factor is twice that of a full-frame sensor. The axis is “logarithmic”, meaning that every 2× increase in sensor size spans the same horizontal distance: the steps from 0.2 to 0.4 to 0.8 to 1.6 are all equidistant[2].

Figure 1a shows (from left to right):

  • so-called 1/2.33” sensors in super-zoom bridge cameras,
  • so-called 1/1.7″ sensors (5.7×7.6 mm) typically found in high-end compact cameras,
  • so-called Four-Thirds sensors with a crop factor of 2.0×,
  • mainstream APS-C size sensors with a crop factor of either 1.5× or 1.6×,
  • specialized APS-H size sensors with a crop factor of 1.3×,
  • full-frame cameras (24×36 mm, with a crop factor of 1.0×), and
  • medium-format cameras (crop factor of roughly 0.7×).

Some cameras are labeled with an abbreviated model number. Thus 1D4 is short for Canon EOS 1D Mark IV and α55 is the Sony STL Alpha A55. Please use the original DxOMark graphs for looking up specific cameras.

The color scale[3] used in all my graphs indicates the size of the sensor: orange represents the tiny sensors, 4/3 and APS-C are shown in shades of green, cyan is mainly Canon’s 1.3x EOS 1D APS-H series, blue is for full-frame, and magenta and red are the “medium-format” sensors.

Note that mainstream compact cameras with tiny (1/2.5″) sensors and correspondingly lower image quality are hardly covered in DxOMark’s database – partly because they can’t generate the required Raw files. It is also worth noting that the super-zoom models with the smallest sensors (e.g. Olympus’ SP 570 UZ) at first glance resemble SLRs.

Figure 1a shows quite some interesting information:

  • As a general rule, larger sensors outperform smaller ones….
  • …but newer models generally outperform older models. In particular, two new APS-C models (Nikon’s D7000 and Pentax’ K-5) outperform the older 1.3× sensors and even most full-frame (1.0×) sensors due to a significantly lower noise floor.
  • The performance of the mirrorless Sony NEX-5 is in line with its 1.5× APS-C sensor. Its mirrorless design and its use of an electronic viewfinder have no impact on image quality: a classic SLR swings its mirror out of the way during exposure. So the lack of a mirror doesn’t impact image quality.
  • The Sony Alpha 55 , with its notable semi-transparent[4] mirror, performs roughly as you would expect given its APS-C sensor. But because its semi-transparent mirror doesn’t swing out of the way, 30% of the light never reaches the sensor. Note the performance gap between the Alpha 55 and the Nikon D7000 or Pentax K-5: the higher score (lower noise) of the latter two could be explained[5] by the light diverted by the Alpha 55’s stationary semi-transparent mirror.
  • Surprisingly, except for the 1/1.7″ segment, none of the Canon models are currently best-in-class[6] compared to their competition. This is partly because Canon’s two full-frame models (5D Mark II and 1Ds Mark III) are currently 2 and 3 years old. And because both of Canon’s 2010 APS-C models (550D and 60D) are entry-level models which don’t outperform the fancier Canon 7D introduced in 2009 (see Figure 2).

Figure 2. This is a subset of Figure 1c with extra lines to connect models that form a commercial series. The colors represent sensor size. Click to enlage image.

As we are digressing anyway now, Figure 2 shows that Nikon (gray text labels) originally lagged behind Canon (colored text labels) in terms of the image quality of its D-SLR sensors[7]. But with the introduction of the Nikon D3 in mid 2007, Nikon[8] appears to have overtaken Canon in DSLR image quality – at least for now.

Figure 2 also clearly shows that sensor size has a significant impact on image quality. Even Canon’s two APS-C series (300D-550D versus 10D-60D) have very similar image quality despite their price difference.

Price and Image Quality

Figure 1b. DxOMark Sensor scores for cameras across the price range.

Some highlights that can be seen in Figure 1b:

  • Note the logarithmic horizontal scale: the DxOMark camera data covers a 1:100 price ratio ($400 – $40k).
  • Some models at the bottom of the cloud are older models and are no longer manufactured. Their indicative price is apparently what they cost on the used market. The lowest blue (1.0×) model is thus the original Canon 1Ds from 2002.
  • The new 9.5 k$ Pentax 645D costs half as much as the other medium-format cameras. It costs about the same as the most expensive full-frame model (Nikon D3x). Although it benefits from its large sensor size, its image quality is similar[9] to the new Pentax K-5 which costs merely 15% of the 645D’s price.
  • Doubling your budget should get you more image quality within the price range up to $2000. Above $2000, you have to be very selective to get any significant increase in image quality – regardless of how much you are willing to spend: you are partly paying for the small series in which these products are manufactured.

Older versus Newer Models

Figure 1c. Within a sensor size category, the DxOMark Sensor score tends to increase over time.

The historical data in Fig. 1c shows the 126 models in DxOMark’s database at the start of 2011. Various early digital SLR models that mainly have historical significance were not tested by DxO. Other observations:

  • Most compact cameras are also absent. These numerous models (e.g. by Canon, Casio, FujiFilm, Nikon, Olympus, Panasonic, Pentax, Samsung, Sony) typically have 1/2.3″ or 1/2.5″ sensors (crop factor of 6×). This market segment largely caters to those looking for ease-of-use rather than cutting-edge image quality. Consequently most compact models don’t support Raw mode and were not tested.
  • With the exception of the Panasonic GH-1, the Four-Thirds category (darker green) has not made much progress so far. The GH-2 actually has a marginally lower score than its predecessor. This reflects a slight increase in (resolution-normalized) noise under both high- and low-lighting conditions.
  • The tested Hasselblad models (H3D, 2007) have been gradually overtaken in image quality by full-frame models and even two APS-C models. The newer Hasselblad models (H4D, 2009) have not been tested so far, but should benefit from their increased sensor size.

Having Too Many MPixels Often Doesn’t Help

Apart from the fact that DxOMark Sensor only covers image quality, it is important to realize that the DxOMark Sensor score does not directly reward sensors that have above-average resolutions.

Instead, the score is a measure for achievable print quality for typical use cases where print quality is seldom limited by sensor resolution. So why didn’t DxO somehow factor sensor resolution into the DxOMark Sensor score?

Firstly, this is because current sensor resolution is generally high enough for producing gallery-quality prints. In fact, software typically silently scales down resolution during printing. And secondly, lens sharpness (rather than sensor resolution) is often the weakest link when it comes to achievable resolution. 60 line pairs per mm is considered an exceptional lens resolution. D-SLR sensors have a typical pixel pitch of 4-8 µm, corresponding to 125-60 line pairs per mm.

Let’s check this by estimating the required print resolution. For 250 DPI print resolution, A4 (8.3″×11.7″) or A3 prints require 5 and 10 MPixels respectively when printed with some borders.Because 250 DPI equals 100 pixels per mm², our eyes will have a tough time assessing this sharpness without a loupe. In my own experience with my old 6 MPixel Canon 10D, even slightly cropped images give you great A3 prints without any fancy digital acrobatics[10] – providing that you use high quality lenses.

These numbers are a bit surprising when you consider that sensors only measure one color per “pixel” and thus lack information compared to screen pixels (see Bayer mosaic). But the camera industry is quite good at reconstructing the missing color information using fancy demosaicing algorithms. It also helps that our eyes are not especially good at seeing sudden color changes unless they coincide with sudden brightness changes. So even when viewed at “100%”, camera pixels can look surprisingly sharp.

But wouldn’t we need more pixels for larger prints such as A2 paper? Not necessarily: if you view big prints from a larger distance in order to see the entire composition, the required resolution saturates at the (angular) resolving power of our eyes.

You will be hard-pressed to buy a modern SLR camera with less than 12 MPixels (see Figure 3), so those extra MPixels allow you to crop your images (“digital zoom” during post-processing,again assuming your lenses are top-notch) – and to impress your male friends.

Figure 3. Launch dates versus MPixels. The lines show the various Canon series. Click to enlarge.

Figure 3 shows how MPixel values evolved over time. The vertical axis thus corresponds to the general public’s rather inaccurate view that MPixels mean image quality. This view can be tested by comparing Figures 2 (image quality) and 3 (MPixels). For example, take the yellow Canon G-series: between the G10 and G11, the resolution was actually reduced from 14.7 to 10 MPixels while the image quality went up. These new 10 MPixel models (G10, G11 and their respective twins, the S90, S95) were well received by the photographers looking for a small extra pocket camera.

But Having Too Many MPixels Doesn’t Hurt Either

More MPixels imply larger image files and obviously slow down processing and file transfers. But the good news is that extreme MPixel counts do not necessarily harm image quality – despite some tenacious claims to the contrary.

The reason for this is that when you scale down to a lower resolution (often automatically done when you print or view the results), the resulting noise and Dynamic range are equivalent to what you would have gotten if you had started off with a sensor which had the required target resolution.

Let’s look at this more closely – but without scaring you off with actual formulas.

Figure 4. Impact of pixel size on noise level. Click to enlarge.

Figure 4 shows an analogy: measuring the rate of rainfall by collecting rain in measuring cups. We could measure the rainfall with a single large bowl. Or, alternatively, we could use 4, 16 or 64 smaller cups. In all these cases the effective area used for catching drops is kept the same[11].

In the case with 64 cups, I exposed these cups to a simulated rainfall that caused each cup to get on average 5 drops of rain during the exposure. For visual clarity I used really big drops (hailstones) or really small cups. However, for the signal-to-noise ratio the size of the cups doesn’t matter. Due to the statistics ( Poisson distribution with “λ=5″, in the jargon), on average only 17% of the cups will contain exactly 5 drops of rain. Some will have 4 drops (17% chance) or 6 drops (15% chance), but some (4%) may even contain 9 drops or stay empty during the measurement interval (0.7%).

This phenomenon explains a major source of pixel noise (“photon shot noise”[12]) which is unavoidable and especially noticeable with small pixels, in dark shadows and at high ISO settings. The corresponding light level is shown projected as a gray-scale image below the cups: empty cups correspond to black pixels and full cups to white pixels.

Now let’s look at the array with 16 (instead of 64) cups. Each cup is 4× larger and will thus, on average, catch 20 drops instead of 5 drops. But, after scaling, the measurements obviously result in the same estimated rainfall[13]. Due to statistics, we may occasionally (9% chance) encounter 20 drops in cup, but we will likely also encounter 18 (8%), 21 (9%), and 25 (5%) drops. The chances of observing 4 or 36 drops are negligible – but non-zero. So, although larger cups will have slightly more variation in terms of drops than smaller cups, the variations expressed in uncertainty in the amount of rainfall/m2 will actually decrease as the cup size increases[14].

So the point is that when using smaller cups/pixels, proper scaling using all available measurement data allows us to get exactly the same signal and noise levels as when using bigger cups/pixels[15]. In terms of cups, a set of 4 cups will tell you exactly what a single bigger cup would have measured: just pour the content of 4 cups into one big cup.

Per-pixel Sensor Noise

Our cups-and-drops analogy gives a basic model[16] of pixel behavior when there is enough light. Real pixels in say a 12 MPixel APS-C Nikon D300 can hold in the order of 40,000 free electrons[17] knocked loose by those speedy photons. For compact cameras that number is lower because they have smaller photodiodes, for medium-format sensors that number can be higher.

λ=40,000 implies a noise level of 200 (= square-root of 40,000) electrons and thus a signal-to-noise ratio of 200:1 (“46 dB” in engineer-speak). This is under the best possible circumstances: it holds for the noise within an extreme image highlight at the camera’s lowest ISO setting. So instead of λ=5, λ=20, λ=80 and λ=320 as shown in Figure 4, actual sensors have values like λ=40,000. At λ=40,000 the basic principle and the math stays the same, although the noise levels can be imperceptible[18].

However, when parts of the image are exposed four stops lower (-4 EV, 6% gray) than the highlights, you catch 40,000 / (2×2×2×2) drops or λ=2,500. This gives a noise level of 50 drops. So the signal-to-noise ratio is now down to 50:1 (“33 dB”). That’s still pretty good, but you might be able to notice the noise. This is why you sometimes see noise in shadows even at 100 ISO.

If we make matters worse by boosting the ISO from say 100 to 3200 ISO, we are essentially underexposing by a massive 32×. You knew that ISO settings with digital cameras were ‘only’ underexposing, and brightening the results by analog amplification or digital scaling, didn’t you? So exposing our dark 6% gray at 3200 ISO, leaves us with an average signal level of just 78 electrons, with a noise level of at least 9 electrons – resulting in a highly visible signal-to-noise ratio of 9:1.

It is worth noting that, except for the number 40,000 electrons for the “full well capacity”, none of this can be changed by smart engineers or negotiated about by their managers. It’s just math.

But… Per-Pixel Noise Is Not Very Relevant

This gets us back to “smaller pixels give higher noise levels per-pixel”. But per-sensor-pixel noise is the wrong metric for prints (or, for that matter, any other way to view an image in its entirety). Printing implies scaling (let’s assume down) to a fixed resolution. If the resolution scaling is done carefully, it exactly cancels out the extra per-pixel noise which you get by starting off with smaller pixels.

So the following options for reducing image resolution – according to this basic model – give you the same signal levels and the same[19] noise levels:

  1. Starting off with a sensor which has large pixels (low resolution) with the same total light-sensitive area.
  2. Using a higher resolution sensor, but combining the analog quantities before going digital. This is like pouring 4 small cups into a bowl before measuring (“analog binning”).
  3. Using a higher resolution sensor, measuring the output per pixel and then scaling the results down by averaging (“digital binning”[ 20]).
  4. Using a higher resolution sensor, capturing all the information in a file, and letting a PC do the downscaling.

An example: this means that a 60 MPixel sensor in a Phase One P65+ camera back should[21]give the same print quality and the same DxOMark Sensor score as:

  • a hypothetical 15 MPixel sensor with the same medium-format sensor size
  • an image that is downscaled within the camera to 15 MPixels
  • an image that is downscaled during post-processing to 15 MPixels

By coincidence (as I later heard from a DxO expert) the benchmarking guys had actually tested the second scenario for the P65+ digital back: in its “Sensor+” mode with 15 MPixel Raw output files, it gets the same DxOMark Sensor score as in its 60 MPixel native mode. This helps reassure us of the usability of the model use for scaling noise when the resolution is scaled.

Resolution and DxOMark Sensor Score

As discussed above, the DxOMark Sensor score is “normalized” to compensate for differences in sensor resolution. To summarize: the DxOMark Sensor benchmark doesn’t “punish” high-resolution sensors for having lots of small pixels that are each individually noisier. And similarly, the benchmark doesn’t favor using large pixels despite their lower per-pixel noise. This is not some kind of ideology: it is just estimating the resulting noise level when viewing the entire image.

Figure 1d. The correlation between DxOMark Sensor score and the amount of MPixels.

OK. Let’s go back to the data shown in Figure 1d. Despite all the theory which explains why MPixels shouldn’t impact image-level noise, Figure 1d does show a trend that higher-resolution sensors produce higher DxOMark Sensor scores -which essentially means “less noise”.

Question: So why don’t we find 10-16 MPixel sensors with top DxOMark Sensor scores?

Answer: Technically it can be done, but it’s not a commercially interesting product. To make one, you use a large sensor (like the D3x) or even larger, and fill it with say 12 MPixels. But, as we explained above, this hypothetical 12 MPixel D3x-lite should perform just like a real D3x whose output images were downscaled to a lower resolution. So there is no major benefit of designing such a hypothetical D3x-lite compared to a D3x – and you would lose the option of using the high-resolution mode

Question: If high-resolution is painless, why not provide say 50 MPixel APS-C sensors?

Answer: The pixel pitch would drop down to about 2.5 µm. At that resolution, lenses are generally the bottleneck -so you won’t see much improvement in resolution. And for extremely small pixels, the assumed idealized scaling (with an assumed constant fill factor and constant quantum efficiency) may no longer hold: four 2.5×2.5 µm sensors together would capture less light than one 5×5 µm sensor (wiring gets in the way, mechanical tolerances on filters, “fill factor”, etc). This increase in noise at some point would reduce the DxOMark Sensor score.

Impact of larger sensor on our lenses

It should be clear by now that larger sensors (rather than larger pixels!) can produce less noisy images. This is simply because a larger sensor area can capture more light – and for reasonable resolutions this is pretty independent of the amount of MPixels the sensor’s surface has been divided into.

But to capture more light within the same exposure time, you need a proportionally larger lens. An example:

  1. Take a 105 mm f/2.8 lens on a full-frame camera as reference.
  2. And now we compare it to a medium-format camera with twice the sensor surface area of a full-frame sensor.
  3. If we try to use the 105 mm lens, it may not properly fill the 1.41× larger image circle. And if it did, we would have an increased field of view – which is not a fair comparison. So we use a 150 mm lens with a suitable image circle instead of the 105 mm full-frame lens.
  4. If the 150 mm lens is also f/2.8, we would get the same exposure times. But f/2.8 at 150 mm requires the effective diameter of the front lens to be 141% larger than a 105 mm f/2.8 lens.
  5. This means that the diameter of the front lens has increased proportionally with the diagonal of the image sensor. And that the area of the front lens has increased proportionally to the surface area of the sensor[22].

Which sounds sensible: bigger sensors require bigger glass if you want the same shutter speeds. Alternatively, you can use a 150 mm f/4 lens. Either you underexpose your image 2×, and get no noise level improvement over the original full-frame sensor. Or you expose twice as long, using a tripod if needed. But then it would have been fairer to benchmark against a 105 mm f/4 lens as well.

Q: Why couldn’t I overexpose the full-frame camera to catch more light just like the medium-format camera?
A: Just like film, silicon saturates at a particular level of photos per unit area. To avoid that, you have to close the shutter before the highlights have reached that level[23].

In this final part, we examine how the DxOMark Sensor score relates to three more basic metrics.

So What Were We Measuring Again?

The DxOMark Sensor score is itself computed using (measured and then resolution-normalized) figures for:

  1. Noise levels: what is the highest ISO level that still gives a specific print quality?
  2. Dynamic Range: ability to simultaneously render highlights and dark shadows under good lighting (low-ISO) conditions
  3. Color Sensitivity or “color depth”: how much color (“chroma”) noise is there, particularly in the shadows under good lighting (low-ISO) conditions All this data (and more!) is measured and provided by DxOMark on their website.

The 3 metrics are shown in Figures 5, 6 and 7.As DxOMark’s vice-president of marketing, Nicolas Touchard, explained during a telephone interview:

The DxOMark Sensor score is under normal conditions a weighted average of noise, dynamic range and color sensitivity information. But some nonlinearities are deliberately included in the algorithm to avoid clear weakness in one area from being hidden by clear strengths in one of the other areas.

It is worth noting that these three underlying measurements are to some degree interrelated because they are all tied to sensor noise: Dynamic Range is the ratio between the brightest signal and the background noise (at low ISO). Color sensitivity or Color Depth represents whether small color differences are masked by chroma noise. And Low-light ISO tells you what ISO levels give equivalent noise levels on different cameras.

Although this means that some degree of correlation between the three underlying measurements is inevitable, different cameras do come out on top for each sub-benchmark. This confirms that we are not just getting to see the same data presented in three different ways.

DxO at some point tried to link the metrics to different types of photography, but DxO is fortunately starting to deemphasize this as the mapping between measurement and use cases was not very helpful. Here were the mappings:

Metric Assumed lighting Use-case name Discussion
Dynamic Range Enough-light = low ISO “Landscape” This metric assumes that you use a tripod if needed. Many non-landscape photos can also have a large contrast: architecture, portraits, night photography, weddings. A higher Dynamic Range also allows you to make larger exposure errors.
Low-light ISO Challenging = high ISO “Sport” This metric assumes you are forced to go to higher ISO. This is relevant for many other types of photography: street, wildlife, news, weddings, night, concerts, and family. Most photographers need to resort to high-ISO settings regularly. And some need it on a daily basis.
Color Depth Challenging = high ISO “Portrait” This metric assumes you have enough light but may be a fair indication of what you would get with little light. Essentially it measures choma noise in the dark parts of a low-ISO image. Portraits may not be especially critical as chroma noise could be filtered out (at the cost of resolution) or you may be able to increase your lighting levels.

So all-in-all, I indeed wouldn’t take the names Landscape, Sport, and Portrait too seriously. At best they are nicknames, and particularly “Portrait” is the least accurate of the bunch.

We will discuss how the 126 cameras perform on these three metrics below.

Dynamic Range at Low ISO

Figure 5. Dynamic Range is currently best in certain recent APS-C cameras.

Here is DxOMark’s definition for their Dynamic Range metric:

Dynamic Range corresponds to the ratio between the highest brightness a camera can capture [..] and the lowest brightness [..] when noise is [as strong as the actual signal].

So far, this is a pretty standard definition. It tells you how many aperture stops of light (EV = bit = factors of two) can be captured in a single exposure. It is analogous to asking how much water a bucket can hold, expressed in units that represent the smallest reliably measurable volume.

Hunting a bit more through the documentation you find that the Dynamic Range value (in “Print” mode) is

normalized to compensate for differences in sensor resolution.

This scaling normalizes to a resolution of 8 MPixel. The choice to use 8 MPixels is irrelevant: it only gives an offset (in EV) in the Dynamic Range scores. And you will find that the Dynamic Range used in the overall benchmarking is the maximum Dynamic Range as

measured for the lowest available ISO setting [typically between 50 and 200 ISO].

Today’s sensor with the highest Dynamic Range score (the Pentax K-5) spans 14 stops at 80 ISO. DxOMark’s Dynamic Range plot for the K-5 shows that its Dynamic Range drops by almost one 1 EV each time the ISO is doubled. The ISO setting for the K-5 thus corresponds closely to an ideal amplifier that amplifies both signal level and noise level equally without adding noise of its own. That is nice.

Various other cameras like Canon’s 5D Mark II shows hardly any Dynamic Range improvements when you decrease the ISO from 800 to 100. This indicates significant background noise[24] in the 5D2 that has been largely avoided in the K-5 or Nikon D7000.

The data in Figure 5 confirm that larger sensors tend to have a larger Dynamic Range than smaller ones, but there is still a very significant variation within any sensor size. The exceptional Dynamic Range figures for the K-5 and D7000 will likely be exceeded by next generation full-frame and medium-format cameras.

The Dynamic Range scores of the FujiFilm FinePix S3 and S5 models are worth pointing out here because they have exceptional Dynamic Ranges, especially considering that they were introduced back in 2004/2006. This was achieved by combining large and small photodiodes on the same sensor. The small photodiodes capture the highlights, while the larger ones simultaneously capture the rest of the image.

Exercise: If you want to play with the data a bit, you can look up (under DxOMark’s tab “Full SNR”) the gray level at which the signal-to-noise ratio drops to 0 dB for the 80 ISO curve. For the K-5 this is a near-black with only 0.008% reflectivity. The brightest representable shade is 100%. So the ratio is 100/0.008 = 12500:1 which gives log(12500)/log(2) = 13.6 stops.

But we are not done yet: the “Full SNR” values in that particular DxO graph are not resolution-normalized. So we still need to scale from 16.4 MPixels down to 8 MPixels. This is a resolution ratio of roughly 2:1. The noise scales with the square root of this ratio, thus giving an extra 0.4 stop [ sqrt(16.4/8)-1 ] of Dynamic Range when scaled to 8 MPixels. The value listed by DxOMark for their normalized Dynamic Range should thus be roughly 13.6+0.4=14.0. The actual listed value is 14.1. Apart from proving that we still kind of understand how the benchmark works, this exercise shows that a twofold difference in resolution corresponds to 0.4 EV difference in Dynamic Range.

Low-Light ISO Score

Figure 6. The 10 models with highest ISO performance are all full-frame models.

Here is DxOMark’s definition for their low-light ISO score:

Low-Light ISO is then the highest ISO setting for the camera such that the Signal-to-Noise ratio reaches this 30dB value [32:1 ratio at 18% middle grey] while keeping a good Dynamic Range of 9 EVs [512:1 ratio] and a Color Depth of 18 bits [roughly 64×64×64 colors].

This is a rather complex definition with multiple built-in non-linearities: you are essentially supposed to increase[25] the ISO value until you exceed any one of the three rules. Due to this definition, the outcome can be anywhere in the ISO range[26] -not just values normally considered to be high ISO.

Again, Low-Light ISO is normalized to an arbitrary reference resolution of 8 MPixels.

The general idea behind this Low-Light ISO metric is simple: it tests which ISO level still gives acceptable image quality using a semi-arbitrary criterion for what “acceptable” means. As Figure 6 shows, the best camera on this particular benchmark is the Nikon D3s (not to be confused with the D3x). Note that the 10 best ranking models on this benchmark all happen to have full-frame sensors.

The gray scaling line in Figure 6 shows how other sensor sizes would score if they performed just as well as the Nikon D3s – but with an estimated handicap to reflect differences in sensor size. Thus a Four-Thirds sensor has a 4× smaller sensor area than a full-frame sensor, and thus would require 4× more light falling on this 4× smaller area in order to achieve the same signal-to-noise ratio. Indeed, some cameras like the Panasonic FZ28, the Canon S/G-series, the FujiFilm S100fs, the Panasonic GH1 and two new APS-C models perform close to this scaling line.

But the slope of the scaling line also predicts that a typical medium-format sensor should be able to deliver “acceptable” (according to the semi-arbitrary definition) images at 6400 ISO. This is 5-10 times better than the actually measured performance for medium-format sensors. Although commercially it may not be a big deal because these SUVs of the camera world are generally used on tripods or in studios with sufficient lighting, I don’t have a technical explanation yet for this performance.

Similarly, I hadn’t expected that the smallest sensors would quite manage to reach these scaled noise levels. This doesn’t mean these sensors have very low noise. On the contrary: they have to be used at e.g. 200 ISO to get the same print quality as the leading full-frame sensor at 3200 ISO. But given this unavoidable phenomenon, some actually do an admirable job[27].

Exercise: If you want to play with the data a bit, you can look up (under “Full SNR”) the ISO setting at which 18% gray gives a 30 dB (5 EV) signal-to-noise ratio. You should get a value for the K-5 around 600 ISO. To get the more relevant resolution-normalized ISO value, you have to replace the 30 dB criterion by 26.7 dB to compensate for resolution normalization. This should result in a score close to the 1162 ISO in DxOMark’s own results.

Low-ISO Color Sensitivity

Figure 7. Color Sensitivity seems to be best in the largest sensors.

Here is DxOMark’s definition for their Color Depth score:

Color Depth is the maximum achievable color sensitivity, expressed in bits. It indicates the number of different colors that the sensor is able to distinguish given its noise.

The metric thus looks at local color variations caused by noise. It does not cover color accuracy – presumably because that can be corrected in post processing and maybe because it opens an eXtra Large can of worms.

The benchmark values for Color Depth are again normalized with respect to sensor resolution. And, again, the phrase “maximum achievable” means that this is the Color Sensitivity at the lowest (e.g. 100) ISO settings.

As shown in Figure 7, larger sensors clearly have a larger Color Depth score. This is largely explainable by their lower noise at 100 ISO as shown with Figures 4 and 6. But color noise also depends on the choice and performance of the microscopic color filters that allow the photodiodes to measure color information (not shown in Figure 4). If less saturated color filters (“pink instead of red”) were used, the different color channels would respond only marginally differently to different colors. This would lead to higher general sensitivity of the camera, but would introduce more noise when converting to a standard color space.

For more information on the role of the “color response” of color filter arrays, see this white paper where DxO points out the impact of differences in color filter design between the Nikon D5000 and the Canon 500D[28].

A Color Depth value of 24 bit incidentally means that there is a total of 24 bits of information in the three color channels[29].

So How Fair is the DxOMark Sensor Score?

There is no simple objective answer to this important question. Probably every image quality expert would have a somewhat different personal preference for a benchmark like this. But my impression is that the benchmark is pretty useful: I analyzed the model and the data, but didn’t find any serious flaws. Furthermore, results like Figure 2 appear to be pretty consistent with traditional hands-on reviews: models that were stronger [weaker] than state-of-the-art when they were introduced (such as the Canon 40D [50D]) show up as expected in the DxOMark data. And, again, having a pretty solid metric by an independent party is better than endless discussions about what an ideal metric might look like.

The list of critical notes, suggestions and open issues that I ran into so far are all relatively subtle:

  • Complexity.
    Undoubtedly complexity is a fact-of-life when you design sensors. And to DxOMark’s credit, they allow you use just a single figure score to compare camera body image quality. But say you have a difference of 5, 10 or 20 points: I found it very difficult to figure out what to look for in a series of real-world test photographs to confirm the difference. In fact, Theuwissen’s parameterized model for sensor noise suggests that one should be able to characterize key sensor behavior in fewer graphs, measurements and numbers.
  • Undocumented formula.
    Documentation about the way the final DxOMark Sensor score is computed from Dynamic Range, Color Sensitivity and Low-light ISO scores is not currently available. I don’t know if some manufacturers have access to this information or have figured it out by themselves. But I would prefer to level the playing field by publishing the (probably simple compared to what we already know) formula to compute DxOMark Sensor score from the 3 lower-level metrics (that are documented well enough for most purposes).
  • Fixed Pattern Noise treatment.
    FPN is caused by physical or electrical non-uniformities in the sensor and can be largely corrected – although many cameras (like my own 5D2 don’t do this at normal exposure intervals). DxOMark does not attempt to distinguish between FPN noise (that can be subtracted away in say Photoshop) as opposed to irregular (“temporal”) noise. So if a camera would automatically corrects for FPN, it scores well on the test[30].
  • How important is Dynamic Range? – Photographers run out of Dynamic Range once in a while: usually in terms of “burnt” or “clipped” highlights. What DxOMark measures is more subtle: if you make an exposure series, what quality level will the best image have? In photographer-speak, what shadow noise do you get if you do an ideal “expose to the right” exposure. A high Dynamic Range sensor is good, but chances are that you can’t print or even view this without special software. The Landscape/Sport/Portrait terms can easily confuse people who take this literally. I am tempted to interpret the 3 metrics as Dynamic Range (as DxO does), Luminance Noise (instead of Low-Light), and Chroma Noise (instead of Color Sensitivity). Those are quantities you find more often in reviews.
  • Why measure Color Depth at low ISO? – I doubt people can actually see color noise at low ISO. It’s hard enough to spot regular noise at low ISO, and chroma noise is even harder to see. High-ISO chroma noise seems more relevant. I suspect that the choice to use low-ISO Color Depth is an artifact of originally trying to define a metric that matched studio portrait conditions.
  • Metric measureable per ISO setting? – It might have been clearer to have a single “perceived image quality” metric that could be measured at different ISO levels. This is particularly relevant because some cameras excel in high ISO conditions (requires a low noise floor) while others excel in low ISO conditions (requires large sensor).
  • Sensor size visualization – DxOMark’s online graphs allow you to plot scores with MPixels along the horizontal axis. It would be nice to add a setting that shows sensor size instead of MPixels. This would (just like in this article) cluster comparable products together. Representing sensor size in all graphs using color might also be a worthwhile improvement because photographers tend to consider different sensor sizes as different kinds of cameras (unlike MPixel ratings).


[1] The repeatability of the score can be estimated by comparing the scores for virtually identical cameras. Thus, for example, the database contains a pre-production Canon 550D as well as the actual production model. Similarly, the Canon S95 and G12 models are also believed to have the same technology in a different housing.

[2] This is the preferred way to visualize things when the ratio between numbers is more meaningful than the difference between the numbers.

[3] The scale is a continuous color gradient (Matlab-style colormap). If you want to use the same coloring convention formula to represent sensor size, contact me for help.

[4] Sony calls this “translucent”, but this is technically not a very appropriate term. Frosted glass is translucent. Using the right term keeps Ken Rockwell happy ;-)

[5] 70% of the light reaches the sensor. That is equivalent to loosing 0.5 stop of light. 15 points was 1 stop according to DxO, so photographing through Sony’s pellicle mirror (or through a 0.5 EV gray filter) should cost about 8 DxOMark Sensor points. Adding 8 points to the Sony Alpha 55′s score (73) brings the camera on par with the Nikon D7000 (80) and Pentax K-5 (82) which are believed to use a very similar Sony sensor.

[6] Because Canon is pretty much the only supplier in the 1.6× APS-C and 1.3× APS-H categories, you should compare these against e.g. 1.5× APS-C.

[7] Canon essentially created the mass-market for D-SLRs and had set an aggressive initial pace for innovation and price decreases.

[8] Some people say we are seeing Sony overtake Canon in sensor quality rather than seeing Nikon overtake Canon: Canon makes its own image sensors and Nikon reportedly buys its SLR sensors from Sony. This view is credible given that Sony’s α55 and Pentax’ K-5 (officially known to use Sony sensor) are also both best-in-class in terms of actual sensor performance. So it is quite possible that such companies will start to become serious competition for Canon and Nikon (at least in terms of sensor quality) in the coming years.

[9] The Pentax 645D has three times more pixels than the Pentax K-5. But as will be discussed later, this may not be as important for image quality as it may seem.

[10] 5 MPixel for A3 (with a bit of border) corresponds nicely to the 180 DPI lower limit recommended in Luminous Landscape’s in From Camera To Print – Fine Art Printing Tutorial .

[11] As sensor folks say, they have the same “fill factor” or as chip designers say “it’s an optical shrink”. The bowl and cup shapes share here are horizontally scaled versions of each other, thus leading to identical fill factors.

[12] If you have the time and courage to dive deeper, there is a tutorial series at www.harvestimaging.com that quantifies numerous sources of sensor noise. It is by Albert Theuwissen, a leading expert on image quality modeling. I created a kind of synopsis of this 100page series in another posting .

[13] Expressed in millimeters, or in water volume per unit of area.

[14] Cups that on average catch λ drops during the exposure to rain will on average have a standard deviation of sqrt(λ) drops. To estimate the rainfall ρ we get ρ = λ× drop_volume / measurement_area. The expected value of ρ is independent of cup size. And the variation of ρ decreases when larger cups are used. In Figure 4, ρ would be the depth of the water in the cups if the cups had been cylindrical. So as λ is increased (bigger cups or longer exposure), the Signal-to-Noise ratio improves. But ultimately we care about how hard it rains, rather than caring about droplets per measuring cup. If you measure rainfall with a ruler to see how deep the puddles are, you will get a result that doesn’t depend on cup size, and the noise due to drop statistics will decrease for larger cups.

[15] If you still don’t believe this, go read DxO’s white paper “ Contrary to conventional wisdom, higher resolution actually compensates for noise ” as punishment.

[16] To make the model more complete, you could:

  • Measure the amount of water in the cup by weighing each cup. If you don’t subtract the weight of the empty cup, you have a significant “offset”. If you do subtract the weight of empty cups, the correction will not be perfect.
  • Assume some random errors when measuring the amount of water per cup. This “temporal” noise has a fixed standard deviation, and has most impact when the cups are nearly empty.
  • Assume that the cups are not perfectly shaped (“Fixed Pattern Noise”). Maybe rows or columns of cups came from the same batch and have correlating manufacturing deviations (“row or column Fixed Pattern Noise”).
  • Drill a hole near the top of each cup so that excess water from one cup doesn’t overflow into neighboring cups. The holes will have slight variations in their location or size: “saturation or anti-blooming non-uniformity”.
  • Place the cups in a tray of water. If the cups are slightly leaky (unglazed flower pots), you will get some water leaking in from the surroundings into the cups (“dark current or dark signal”). Not all cups will leak equally fast (“dark signal non-uniformity”). And at higher temperatures, you will see a bit faster leakage (sorry, it would be too tricky to emulate the exponential temperature dependency without some really fancy materials).
  • Break a few cups or their measurement scales (“defective pixels”).

The above covers all the noise sources in the PTC tutorial on www.harvestimaging.com.

[17] For info on the value of λ or “the full well capacity”, see Roger Clarke’s website. See http:// www.clarkvision.com/articles/digital.sensor.performance.summary/#full_well.

[18] You would get the same statistics when you measure rain using 2 liter pans. Two liters correspond to about 40,000 drops.

[19] Note that although this scaling story holds for photon shot noise and dark current shot noise, other noise sources don’t necessarily scale in the same way. In particular, some very high-end CCDs can use a special analog trick (“charge binning”) to sum the pixels, thus reducing the amount of times that a readout is required. This would reduce temporal noise by a further sqrt(N) where N is the number of pixels that are binned. Apart from the fact that only exotic sensors have this capability (Phase One’s Pixel+ technology), DxOMark’s data suggest that this extra improvement doesn’t play a significant role.

[20] Some cameras like the Canon 5D Mark II do this digitally. Canon calls these Raw modes SRaw and they have strange MPixel ratios like 5.2 : 10.0 : 21.0.

[21] The above does not mean that you will get exactly the same resolution-normalized results for any down-scaling scenario. It just says basic scaling laws tell us it should be possible to get close.

[22] Actually a quick search showed that the Phase One’s 150mm f/2.8 lens and Nikon’s 105 mm f/2.8 lens weigh the same and the Phase One has an only slightly larger filter size. But the Nikon is a macro lens and the Phase One isn’t. So maybe these two designs are internally too different or one is especially optimistic about its aperture.

[23] In some cases you can increase the dynamic range by taking N identical noisy exposures and averaging out the noise afterwards. This improves the SNR of temporal noise by sqrt(N) but is generally not a very attractive technique.

[24] According to the theory, this could be either “temporal” (normal) noise or “fixed pattern” (nonuniformity) noise in the sensor. Fixed pattern noise can be corrected via various computational or calibration tricks.

[25] The benchmark doesn’t depend on the actual steps (e.g. 1.0 stop or 1/3 stop) in which a user can adjust the ISO setting. Intermediate values are generated by interpolation.

[26] Strictly speaking, the definition doesn’t allow you to express the Low-Light ISO behavior of a camera with a small enough sensor if the camera fails to meet one or more of the three criteria at its base ISO setting. But one of the tested models (Panasonic DMC FZ28) actually has a Low-Light ISO rating that falls below the (both nominal and actual) ISO range of the camera. So apparently this benchmark accepts extrapolated results.

[27] Arguably the Canon S90 is the best low-light camera in the database – at least when we take its limited size into account. In fact, creating an array of about 20 identical S90 sensors would result in a full-frame sensor which would, at least in theory, slightly outperform the reigning Nikon D3s! And (again assuming one could do the tiling seamlessly and could handle all the resulting data) would result in a 200 MPixel übersensor. Or a larger 400 MPixel medium-format sensor that outperforms all current medium-format sensors. Actually this may put Canon’s 120 MPixel “proof-of-concept” APS-H sensor (August 24th 2010) into perspective: when scaled from to full-frame, it would also have 200 MPixels.

[28] In particular, DxOMark’s analysis is that Color Filter Array colors that have too much overlap in their transmission spectra increase chroma noise. Too little overlap decreases chroma noise at the cost of more luminance noise. This is an example how the details of a benchmark can impact design choices.

[29] It doesn’t mean that each channel is sampled at 8 bit: each channel is typically sampled at 12-16 bit. The actual formulas for Color Depth reflect the amount of noise in each channel and are too complex to explain here (integrals).

[30] This is more or less fair because that this is what the user would like to happen. But the camera may have modes to turn this on (for 1+ second exposures) or the user could bother to take a reference exposure with the lens cap on, and then perform the compensation in software. In such cases, the noise figures from DxOMark are too high. If you really want to manually subtract a “dark frame”: make sure you use the same exposure time and ISO setting and temperature as the real image. Note that you don’t need a tripod for this. But you do want to avoid light leakage – particularly for light coming via the lens.

iPad Camera Connectivy Kit

I tested moving photos from a Canon 5D Mark II to an iPad. This can be useful if you need some extra storage, want to view the images on the iPad, or want to mail them.

The 5D2 uses Compact Flash memory cards, so the iCCK’s  SD card reader (right side of photo) won’t work. But you still have two options using the iCCK’s USB adapter (left side of photo):

  1. Plug a Compact Flash card reader into the iPad via the iCCK. This didn’t work for me: the iPad complained that it couldn’t handle the power drown by the reader.
  2. Connect the camera via its miniUSB-to-USB cable, via the iCCK to the iPad. This kind of works. After reading a several files, the program crashes.

Apple states on its website, that repeating the process helps: the import restarts where the previous one failed, and you will ultimately reach the end. The number of files you can read between crashes is variable: sometimes one or two, sometimes dozens.

So the good news is that the iPad can reads heavy-duty RAW files from a Canon 5D Mark II and display these (although you can’t zoom in all the way). But the crashing Photo application and Apple’s workaround are not very user-friendly. This problem was known back in April and is still an issue in iOS 3.2.2.

Save the image highlights!

A friend pointed out an interesting background article on ISO in digital cameras by Richard Butler, a reviewer and technical writer for DPReview. It deals with ISO, dynamic range and the options that camera makers have to avoid blown highlights. The article may be interesting if you have an engineering degree, a camera fetish or both.

Blown highlights

The wedding dress still looks OK (Canon's Highlight Tone Priority feature on). Note that the image was taken on an overcast day.

The problem

By default, a CMOS sensor – unlike film – clips highlights abruptly : when the analog-to-digital converter exceeds its 10- to 14-bit range in a bright part of the image, you no longer see variation in brightness and color. Essentially you hit a hard ceiling on the output: there is no visible difference anymore between bright and brighter-than-bright. So snow, wedding dresses and clouds become flat blobs of white.

Photographers hate blown highlights (unless it gives a fully white background) more than they hate noise or featureless shadows. You may even get corrupt colors if the sky is bright, but not quite bright enough: one or two of the color channels may clip before the remaining channels, leading to an unnatural-looking cyan sky. Essentially blown highlights are comparable to a nasty form of distortion called clipping in the audio world: the equipment can’t generate more output on loud sounds and thereby introduces readily audible artifacts.

Limited dynamic range is one of the main remaining weaknesses of digital imaging nowadays – especially now that many cameras have more megapixels than is good for them. The topic is not getting a lot of marketing attention because the available improvements are limited, and the benefits is these solutions are still too difficult to explain to average consumers.

In photography terms: we are quite aware that the required exposure of our photos can easily vary by say 20 stops (1/8000s at f/16 in bright sun down to 1 second at f/1.4 by candle light). But people are less aware that the range of brightness inside a single image can easily vary more than the 10 stops – being the typical limit of a modern sensor or film.

The goal

So the trick is to change linear response (rising line that suddenly turns horizontal) into an S-shaped response curve. Like a car whereby pressing the pedal at high speeds increases the speed slightly to make you feel you are still in control, but not more than the car or driver can handle.

A sensor-to-image version of the story

The story by Richard Butler starts more or less from the end result and works backward. An engineer tends to work from the fundamental problem towards the solution. So this is what happens if you turn the story inside out…

The ISO settings used for a photo essentially tells you nothing about the sensor (for digital). It mainly tells you about the postprocessing in camera or PC.

So a sensor’s fundamental output (expressed in charge or milliVolt) is only a function of light brightness (aperture, scene) and exposure time (shutter speed). Manually or automatically varying the ISO on cameras is essentially an indication of how the sensor output is scaled to reach normal on-screen or print brightness levels.

So to get to higher ISO values (=take pictures with less light), there are two basic options:

  1. you increase the amplification before converting the analog voltage to digital using an (analog) amplifier. This increases the signal seen by the analog-to-digital (A2D or ADC) converter, but also amplifies noise. In audio terms, you are increasing the volume when there isn’t enough signal. The dynamic range or signal-to-noise ratio will be low because the original signal was low. Amplification (in the best case) increases noise and signal by a similar ratio.
  2. you can alternatively digitally “amplify” by multiplying the output of the A2D. This gives roughly the same result as long as the A2D is accurate enough compared to the original noise level. Hence the interest in 14-bit A2D converters: they allow the processing to be moved to “the digital domain”. Note that it doesn’t really make a difference if you do the scaling within the camera (done for JPEG) or in post-processing (an option in RAW) as long as you don’t lose information on the way.

Modern cameras do a mix of both. A variable gain amplifier before the A2D converter can boost the signal (typically in steps of 2x) without adding significant noise of its own. This reduces the need to make 16-bit high-speed A2D converters (increasing A2D resolution tends to increase the measurement time, and this is a problem when you have a lot of pixels to sample). Smaller steps in ISO are typically done in the digital domain and lead to a fraction of  a stop loss in dynamic range. I doubt this small loss in dynamic range is measurable in normally detectable.

So both options look equivalent, and the trade-off seems to be mainly a subtle optimization of sampling speed versus cost versus noise around the A2D converter.

BUT when you do digital amplification, you can easily make the digital amplification non-linear: you can use the amplification to create an S-shaped response curve by multiplying more at low- and mid tones than in the highlights. This reduces the chance of clipping in the final result and thus stretches the dynamic range.

To some degree it is normal to tweak this tonal response curve to compress the response curve to a nice S-shape that avoids clipping in the shadows or clipping in the highlights. The second option just allows you do this a bit more when you need it.

How does this relate to the DPReview article on Olympus?

The quoted article by Richard Butler shows that the Olympus E-620 SLR normally underexposes by 1 stop and compensate by boosting the output (except for the high tones). This gives an S-shaped response curve. The point of the article is that DPReview notices that this doesn’t occur at 100 ISO: the dynamic range is one stop less, and the 100 ISO behavior is closer to previous Olympus cameras. The point is that at 100 ISO, the sensor gets enough light to utilize the full intrinsic range of the photo sites. So at 100 ISO, you have more risk of losing the highlights than at 200 ISO or above. In fact, it is suggested that if you need 100 ISO (e.g. to get a slow shutter speed) you may be better off using the 200 ISO’s response curve and decreasing the light intensity (you needed 100 ISO didn’t you) using a 2x gray filter.

Or you can go manual

Alternatively, you can still use 100 ISO on the Olympus (assuming that’s what you need) and underexpose by say 1 stop. That uses the sensor in the same range as if you use 200 ISO, and you can then brighten the dark picture manually using whatever response curve you feel like in post-processing. With the underexposed picture, you can actually have fun solving the puzzle yourself on the PC. In say Adobe Lightroom, you can increase the exposure (gain), or tweak tone curve related parameters. Unfortunately, Lightroom has a lot of overlapping sliders you can play with and no single graph showing what they do except for the rendered image and the histogram. So you can play with “fill-in light” or alternative can turn up the dark tones, but only the latter will show a change in the tone curve.

Canon’s Highlight Tone Priority

Canon has a feature you can enable and disable called Highlight Tone Priority. Again, it underexposes, and compensates in the camera by boosting the non-highlight output levels.
Again it doesn’t work at 100 ISO. This actually has as side effect that Canon cameras, with HTP enabled and in auto-iso mode will avoid 100 ISO and select settings of 200 ISO or higher. This can be a good thing because, unless you have a very bright scene, the extra quality you get at 100 ISO doesn’t outweight other improvements you can get if you if you make the sensor look twice as sensitive: you can opt for 2x less motion blur or reduce lens aberrations by moving away from full aperture. So in my case, I tend to leave HTP on all the time and essentially avoid too extreme “expose to the right behaviour”.

Nikon has a similar feature called D-Lighting (pronounced “delighting”) which also doesn’t work at 100 ISO.