DxOMark now reviews lenses

DxO Optics Pro

DxO‘s products are interesting – especially considering the modest size of the company. Their best known product, DxO Optics Pro,  is a sophisticated raw converter. It automatically corrects lens errors (distortion, vignetting, chromatic aberration) and provides sensor noise reduction based on measured noise characteristics of the sensor. It thus provides (since 2004) the lens correction features that Adobe just introduced in Lightroom 3 and Adobe Camera Raw 6.1.

It also provides a wide range of common image enhancement functions – notably including “DxO Lighting” which does automatic tone mapping on a single raw image to make it fit the limited dynamic range of a JPG, screen or print. In more photographic terms, you might use this is as automatic highlight recovery. Again this overlaps with Lightroom 3, and in fact the user interfaces of both products look a bit similar (in this case Lightroom was first).

DxOMark for raw review data

DxO also provides a free web-based service called DxOMark whereby they provide detailed image quality measurements of (SLR or high-end compact) cameras. You can see DxOMark as a “raw” version of a product review: DxO does the geeky part of measuring camera performance in a well-defined and objective way. DxO leaves the “back-end” of the review process to either its readers or to other review sites. Thus, in DxOMark you won’t find the subjective but useful opinions like “how solid does it feel?”, “are the buttons in the right place?”, “is it good value for money” or sometimes even “here is what the shutter sounds like”.

Why DxOMark?

I expect that DxOMark data is just a spin-off of the measurements which the company does to enable its DxO Optics Pro product. Incidentally not all equipment supported by DxO Optics Pro is currently in the DxOMark database. DxO writes:

Initially covering about 540 cameras-lens combinations, with dozens to follow each month

So although the extra costs of the service may be relatively limited if you have the data anyway, the business model of giving high quality unique data away for free is still not quite clear to me. The collected information goes both to high-end readership and to websites that use the data in their reviews (but requires the source to be clearly identified).

Possibly DxOMark is partly intended as a way to advertise DxO because it generates web traffic and brand awareness. I certainly think it is a healthy move for a company that hasn’t gotten too much attention in the past years. The actual terms of use of DxOMark are available via this link.  They indicate conditions for fee of charge for what they define as “Fair Use” and they offer forms of partnership if you need more. As far as I can tell, my usage of some of their graphs below falls under “Fair Use” because it is non-systematic, because I don’t modify the content, and clearly attribute the data to DxO. But if I would create an entire review site using their data (even while acknowledging DxO) that would be considered “systematic” use of the data and would require a special license.

DxOMark Sensor versus Lens-with-camera

Previously DxOMark only covered the sensor and some aspects of the image processing pipeline. This mainly provided noise data (important for low-light situations) and dynamic range (high-contrast situations).

But they recently added a second service to view performance of lenses. In accordance with DxO tradition, DxO measures each lens on multiple camera bodies. This is reasonable because that is ultimately what the user cares about. And furthermore the resolution of a lens, for example, depends on the “analog” lens quality and the “digital” sensor resolution. This also applies somewhat to vignetting, but not to distortion. The drawback for DxO is that they need to perform more measurements. DxO tries to maintain a balance between making all this data available to advanced readers, while trying to summarize the data (at the cost of some accuracy) in simple metrics.

Lens measurements and data complexity

DxO measures pretty much the same information about lenses as www.PhotoZone.de (which is in English despite being based in Germany). This means that it shows measurements for

  • resolution (c, f, A, r)
  • distortion (c, f, r)
  • chromatic aberration (c, f, A, r)
  • vignetting (c, f, A, r)

whereby c means “per camera body”, f means “per focal length = zoom setting”, A means “per aperture setting”, and r means “depending on distance from center of the sensor”.

The ways of representing all this data different between both sites. DxO has colorful interactive graphs, and the numbers from both websites are not directly comparable.

One representation of lens sharpness ("resolution") at the DxOMark.com site.

The above example graph shows that, at 1/3 of the distance between image center and image corner, this lens/camera combination performs pretty well (“greenish”). That applies even when used wide open (f/4). At extremely small apertures (f/22) one can see lens performance decreasing (thanks to basic physics, “diffraction”). And one can see that this lens is a bit softer at f/4 towards the 105 mm end of the zoom range.

So, admittedly this particular representation is a bit nerdy and may remind some of your school days. But there are also simpler views (color representing resolution across the 24x36mm sensor) as well as significantly nerdier ones (MTF graphs for both vertical and horizontal resolution).

Visualization of resolution (for f=24mm) at www.PhotoZone.de

The PhotoZone.de representation of the same lens on the same body shows comparable information, but as a 2-D bar chart per focal length. The results along the vertical axis are, however, in terms of line-pairs displayable across the long dimension of the sensor. So if we divide the PhotoZone.de results by 36 (mm for a full frame sensor) you get numbers between roughly 50 and 100 and lp/mm. For basic lenses (Canon EF-S 18-55mm f/3.5-5.6 II), the numbers drop down to 800 or 800/22.3mm or 35 lp/mm in the corners.

Although film-based photography has a history of lines/mm rather, values for lines/image can (in theory) be directly compared across different digital sensor sizes as long as they have the same aspect ratio (like full-frame versus APS-C).

Can’t I just get a single rating number?

To DxOMark’s credit, users can get as much or as little information as they like. In fact, in an “Overview” tab, those that want to see a single number, get a single “simple” number. For the lens/camera in the figure above, the answer is “59” (line pairs per mm). Simplicity, however, often means that the complexity has just been moved out of sight: the value “59”  is an average of the resolution over focal length and aperture. And multiple such averages are then averaged (after weighing) to account for the final dimension: differences in resolution between center and edge of the image.

A quick attempt to derive a comparison number from PhotoZone.de results in roughly 78 lp/mm (2800/36). This 30% different compared to 59 is enough to assume that the two measurement techniques are not directly comparable. This is to be expected: the measurements are complex. Thus, for example, as the resolution of the test target goes up, the contrast of the resulting image gradually decreases (see DxO’s MTF representation). This alone could cause a numerical difference. And two different copies of lenses will also give somewhat different results when measured in an identical way.

Transmission benchmark

DxO provides one extra metric that PhotoZone doesn’t have. DxO calls it “transmission”.

This tells you, for example, whether a 50mm f/1.4 lens is really going to give you f/1.4 speed. Thus Canon’s EF 50mm f/1.4 USM and Nikon’s AF Nikkor 50mm f/1.4D are both about 1/3 stop slower than a f/1.4 lens should be according to DxOMark. Thus “transmission” is thus comparable to vignetting: vignetting shows how much darker the edges of the image are than the center; “transmission” shows how much darker the center is than specified. Likely, transmission also impacts depth-of-field slightly.

Note that the sensor benchmark of DxOMark also shows whether a camera’s sensor has the right ISO value. Sometimes (read: generally) a manufacturer overrates its ISO value by a fraction of a stop. I am not sure how DxOMark distinguishes between a discrepancy in the ISO calibration (“exposure is slightly dark”) and aperture calibration (“exposure is slightly dark”).

High-level fun with Peak Score

DxOMark resolution scores for 103 full-frame lens/body combinations

The above graph shows 103 lens/body combinations in the DxOMark database that all have full-frame bodies. The vertical axis shows “Peak Score” which is supposed to represent how much visual “information” (resolution and bit/pixel) the camera captures. It is called “Peak” because the value corresponds to the most optimal setting of focal distance and aperture. From left to right, the four columns correspond to:

  • a mix of the Nikon D700, D3 and D3s: 12.1 MPixel
  • the old Canon 5D: 12.7 MPixel
  • a mix of the Canon 5D Mark II (21 Mpixel) and Canon 1Ds Mark III (21.1)
  • a mix of the Nikon D3x (24.5) and Sony A850/A900 (24.6) – these all use the same Sony sensor, and both Sony’s are virtually identical

As the mix of lenses across the brands is more or less comparable, we conclude that the 12 MPixel Nikons in the first column have the same image sharpness as the older 12.7 MPixel Canon 5D. The 6% difference in resolution is small compared to other factors that cause spread (such as difference between Canon and Nikon lenses). The 21 and 24.5 MPixel sensors clearly outperform the 12 MPixel sensors, but the score doesn’t scale linearly with the MPixel value. DxO writes in the explanation of “peak score”:

[Peak score] grows linearly with resolution, for a perfect optic and a noise-free signal

so apparently at 20+ Mpixel, the lenses can no longer be considered ideal, and the PeakScore(Mpix) curve is already saturating. This is consistent with the fact that, for example, the Canon EF 24-105mm f/4L IS lens shows chromatic aberration at such resolutions – albeit that Peak Score deliberately flatters the lens by measuring its performance at the setting where it performs best (e.g. 70mm f/8).

Interestingly, the two 21 MPixel Canon models match the Peak Scores of the 24.5 MPixel Nikon and Sony models. The 16% difference in resolution doesn’t really show up. This is not because the sensor vastly outperforms the lens (24.5 MPixels sounds like a lot, but these are still 5.9×5.9 µm pixels – compare that to the 4.2×4.2 µm pixels of a Canon 7D).

The DxOMark site also allows you to put the price of the lens on the horizontal axis. Or alternatively the price of the lens/body combination. Those graphs stress that the 1Ds Mark III and especially the D3x are much more expensive than some of the alternatives. This doesn’t necessarily mean that they are overpriced: there are obviously many other relevant parameters than just resolution – and in fact, maybe for your application, extreme resolution may not have the highest priority.

DxOMark versus PhotoZone.de

Based on my own exchanges with both sites in the past, I basically trust both. In particular, both have a healthy safety margin of technical expertise (in-house in the case of DxO, or partly by using 3rd party software – Imatest – and a few expert friends in the case of PhotoZone). You can also tell (in the case of PhotoZone) by looking for remarks like “this looks strange, but I got similar results when I retested”. Or by reading the white papers about their measurement approach and what it all means at DxOMark.com.

As stressed at the start, PhotoZone is real lens review site: it tells you an engineer/photographer’s opinion of the lens. This includes equal parts “hard data” (a few bar charts), sample photos,  and hand-written text (about one page).

DxOMark, on the other hand, is not really a classic review site: there is no real hand-written text explaining what is good or bad and when/why you should care. DxO tries to fulfill the need for simple answers by using clear color schemes (green is good) and generating a handful of simple ratings so that you can see “who wins” and by how much. They even generate ratings (for what it’s worth) for the suitability of the lens/body combination for landscapes, portraits, journalism, sports and family use. These ratings are based on specifications and measurements only: they do not include value-for-money, or other subjective things like “build”, “bokeh” and “focus speed”. Or the inevitable statements about weather-sealing or what does/doesn’t move when you zoom/focus. DxO knows this, and has reserved space on their site to link to “human-written” reviews.

In PhotoZone you will find hundreds of different lenses, but tested on a minimum number of camera bodies each. The limited number of camera bodies has benefits (less data, less work), but also drawbacks (readers would prefer to see the results for the body they would use). Thus in PhotoZone you will find 3rd party lenses (Sigma, Tamron), famous specialty brands (Zeiss, Leitz), but also exotic lenses like a Russian fish-eye or Korean wide-angles. In DxOMark you will find most relevant camera bodies, but currently “only” about 14 lenses per major camera brand. In contrast, PhotoZone has over 100 Canon-mount lenses (from Canon or others). It is hard to say which strategy on allocating testing budget/effort is better – they are just different. I suspect that the kind of person that does research before choosing a lens will end up using both sites for complementary information.

This entry was posted in Photo Technology and tagged , , , . Bookmark the permalink.

3 Responses to DxOMark now reviews lenses

  1. pvdhamer says:

    The DxOMark peak score graph was updated to show some new lenses that were added to the DxOMark database. It now shows 103 lens/body combinations for full-frame sensors.

  2. pvdhamer says:

    Argggh! Why didn’t anybody warn me that the final picture no longer displaying correctly. Fixed. On the positive side, DxOMark now has a link to this page and that’s generating a trickle of traffic to this page.

  3. Pingback: DxOMark Sensor for benchmarking cameras | Peter.vdHamer.com

Leave a Reply

Your email address will not be published. Required fields are marked *