DxOMark now reviews lenses

DxO Optics Pro

DxO‘s products are interesting – especially considering the modest size of the company. Their best known product, DxO Optics Pro,  is a sophisticated raw converter. It automatically corrects lens errors (distortion, vignetting, chromatic aberration) and provides sensor noise reduction based on measured noise characteristics of the sensor. It thus provides (since 2004) the lens correction features that Adobe just introduced in Lightroom 3 and Adobe Camera Raw 6.1.

It also provides a wide range of common image enhancement functions – notably including “DxO Lighting” which does automatic tone mapping on a single raw image to make it fit the limited dynamic range of a JPG, screen or print. In more photographic terms, you might use this is as automatic highlight recovery. Again this overlaps with Lightroom 3, and in fact the user interfaces of both products look a bit similar (in this case Lightroom was first).

DxOMark for raw review data

DxO also provides a free web-based service called DxOMark whereby they provide detailed image quality measurements of (SLR or high-end compact) cameras. You can see DxOMark as a “raw” version of a product review: DxO does the geeky part of measuring camera performance in a well-defined and objective way. DxO leaves the “back-end” of the review process to either its readers or to other review sites. Thus, in DxOMark you won’t find the subjective but useful opinions like “how solid does it feel?”, “are the buttons in the right place?”, “is it good value for money” or sometimes even “here is what the shutter sounds like”.

Why DxOMark?

I expect that DxOMark data is just a spin-off of the measurements which the company does to enable its DxO Optics Pro product. Incidentally not all equipment supported by DxO Optics Pro is currently in the DxOMark database. DxO writes:

Initially covering about 540 cameras-lens combinations, with dozens to follow each month

So although the extra costs of the service may be relatively limited if you have the data anyway, the business model of giving high quality unique data away for free is still not quite clear to me. The collected information goes both to high-end readership and to websites that use the data in their reviews (but requires the source to be clearly identified).

Possibly DxOMark is partly intended as a way to advertise DxO because it generates web traffic and brand awareness. I certainly think it is a healthy move for a company that hasn’t gotten too much attention in the past years. The actual terms of use of DxOMark are available via this link.  They indicate conditions for fee of charge for what they define as “Fair Use” and they offer forms of partnership if you need more. As far as I can tell, my usage of some of their graphs below falls under “Fair Use” because it is non-systematic, because I don’t modify the content, and clearly attribute the data to DxO. But if I would create an entire review site using their data (even while acknowledging DxO) that would be considered “systematic” use of the data and would require a special license.

DxOMark Sensor versus Lens-with-camera

Previously DxOMark only covered the sensor and some aspects of the image processing pipeline. This mainly provided noise data (important for low-light situations) and dynamic range (high-contrast situations).

But they recently added a second service to view performance of lenses. In accordance with DxO tradition, DxO measures each lens on multiple camera bodies. This is reasonable because that is ultimately what the user cares about. And furthermore the resolution of a lens, for example, depends on the “analog” lens quality and the “digital” sensor resolution. This also applies somewhat to vignetting, but not to distortion. The drawback for DxO is that they need to perform more measurements. DxO tries to maintain a balance between making all this data available to advanced readers, while trying to summarize the data (at the cost of some accuracy) in simple metrics.

Lens measurements and data complexity

DxO measures pretty much the same information about lenses as www.PhotoZone.de (which is in English despite being based in Germany). This means that it shows measurements for

  • resolution (c, f, A, r)
  • distortion (c, f, r)
  • chromatic aberration (c, f, A, r)
  • vignetting (c, f, A, r)

whereby c means “per camera body”, f means “per focal length = zoom setting”, A means “per aperture setting”, and r means “depending on distance from center of the sensor”.

The ways of representing all this data different between both sites. DxO has colorful interactive graphs, and the numbers from both websites are not directly comparable.

One representation of lens sharpness ("resolution") at the DxOMark.com site.

The above example graph shows that, at 1/3 of the distance between image center and image corner, this lens/camera combination performs pretty well (“greenish”). That applies even when used wide open (f/4). At extremely small apertures (f/22) one can see lens performance decreasing (thanks to basic physics, “diffraction”). And one can see that this lens is a bit softer at f/4 towards the 105 mm end of the zoom range.

So, admittedly this particular representation is a bit nerdy and may remind some of your school days. But there are also simpler views (color representing resolution across the 24x36mm sensor) as well as significantly nerdier ones (MTF graphs for both vertical and horizontal resolution).

Visualization of resolution (for f=24mm) at www.PhotoZone.de

The PhotoZone.de representation of the same lens on the same body shows comparable information, but as a 2-D bar chart per focal length. The results along the vertical axis are, however, in terms of line-pairs displayable across the long dimension of the sensor. So if we divide the PhotoZone.de results by 36 (mm for a full frame sensor) you get numbers between roughly 50 and 100 and lp/mm. For basic lenses (Canon EF-S 18-55mm f/3.5-5.6 II), the numbers drop down to 800 or 800/22.3mm or 35 lp/mm in the corners.

Although film-based photography has a history of lines/mm rather, values for lines/image can (in theory) be directly compared across different digital sensor sizes as long as they have the same aspect ratio (like full-frame versus APS-C).

Can’t I just get a single rating number?

To DxOMark’s credit, users can get as much or as little information as they like. In fact, in an “Overview” tab, those that want to see a single number, get a single “simple” number. For the lens/camera in the figure above, the answer is “59” (line pairs per mm). Simplicity, however, often means that the complexity has just been moved out of sight: the value “59”  is an average of the resolution over focal length and aperture. And multiple such averages are then averaged (after weighing) to account for the final dimension: differences in resolution between center and edge of the image.

A quick attempt to derive a comparison number from PhotoZone.de results in roughly 78 lp/mm (2800/36). This 30% different compared to 59 is enough to assume that the two measurement techniques are not directly comparable. This is to be expected: the measurements are complex. Thus, for example, as the resolution of the test target goes up, the contrast of the resulting image gradually decreases (see DxO’s MTF representation). This alone could cause a numerical difference. And two different copies of lenses will also give somewhat different results when measured in an identical way.

Transmission benchmark

DxO provides one extra metric that PhotoZone doesn’t have. DxO calls it “transmission”.

This tells you, for example, whether a 50mm f/1.4 lens is really going to give you f/1.4 speed. Thus Canon’s EF 50mm f/1.4 USM and Nikon’s AF Nikkor 50mm f/1.4D are both about 1/3 stop slower than a f/1.4 lens should be according to DxOMark. Thus “transmission” is thus comparable to vignetting: vignetting shows how much darker the edges of the image are than the center; “transmission” shows how much darker the center is than specified. Likely, transmission also impacts depth-of-field slightly.

Note that the sensor benchmark of DxOMark also shows whether a camera’s sensor has the right ISO value. Sometimes (read: generally) a manufacturer overrates its ISO value by a fraction of a stop. I am not sure how DxOMark distinguishes between a discrepancy in the ISO calibration (“exposure is slightly dark”) and aperture calibration (“exposure is slightly dark”).

High-level fun with Peak Score

DxOMark resolution scores for 103 full-frame lens/body combinations

The above graph shows 103 lens/body combinations in the DxOMark database that all have full-frame bodies. The vertical axis shows “Peak Score” which is supposed to represent how much visual “information” (resolution and bit/pixel) the camera captures. It is called “Peak” because the value corresponds to the most optimal setting of focal distance and aperture. From left to right, the four columns correspond to:

  • a mix of the Nikon D700, D3 and D3s: 12.1 MPixel
  • the old Canon 5D: 12.7 MPixel
  • a mix of the Canon 5D Mark II (21 Mpixel) and Canon 1Ds Mark III (21.1)
  • a mix of the Nikon D3x (24.5) and Sony A850/A900 (24.6) – these all use the same Sony sensor, and both Sony’s are virtually identical

As the mix of lenses across the brands is more or less comparable, we conclude that the 12 MPixel Nikons in the first column have the same image sharpness as the older 12.7 MPixel Canon 5D. The 6% difference in resolution is small compared to other factors that cause spread (such as difference between Canon and Nikon lenses). The 21 and 24.5 MPixel sensors clearly outperform the 12 MPixel sensors, but the score doesn’t scale linearly with the MPixel value. DxO writes in the explanation of “peak score”:

[Peak score] grows linearly with resolution, for a perfect optic and a noise-free signal

so apparently at 20+ Mpixel, the lenses can no longer be considered ideal, and the PeakScore(Mpix) curve is already saturating. This is consistent with the fact that, for example, the Canon EF 24-105mm f/4L IS lens shows chromatic aberration at such resolutions – albeit that Peak Score deliberately flatters the lens by measuring its performance at the setting where it performs best (e.g. 70mm f/8).

Interestingly, the two 21 MPixel Canon models match the Peak Scores of the 24.5 MPixel Nikon and Sony models. The 16% difference in resolution doesn’t really show up. This is not because the sensor vastly outperforms the lens (24.5 MPixels sounds like a lot, but these are still 5.9×5.9 µm pixels – compare that to the 4.2×4.2 µm pixels of a Canon 7D).

The DxOMark site also allows you to put the price of the lens on the horizontal axis. Or alternatively the price of the lens/body combination. Those graphs stress that the 1Ds Mark III and especially the D3x are much more expensive than some of the alternatives. This doesn’t necessarily mean that they are overpriced: there are obviously many other relevant parameters than just resolution – and in fact, maybe for your application, extreme resolution may not have the highest priority.

DxOMark versus PhotoZone.de

Based on my own exchanges with both sites in the past, I basically trust both. In particular, both have a healthy safety margin of technical expertise (in-house in the case of DxO, or partly by using 3rd party software – Imatest – and a few expert friends in the case of PhotoZone). You can also tell (in the case of PhotoZone) by looking for remarks like “this looks strange, but I got similar results when I retested”. Or by reading the white papers about their measurement approach and what it all means at DxOMark.com.

As stressed at the start, PhotoZone is real lens review site: it tells you an engineer/photographer’s opinion of the lens. This includes equal parts “hard data” (a few bar charts), sample photos,  and hand-written text (about one page).

DxOMark, on the other hand, is not really a classic review site: there is no real hand-written text explaining what is good or bad and when/why you should care. DxO tries to fulfill the need for simple answers by using clear color schemes (green is good) and generating a handful of simple ratings so that you can see “who wins” and by how much. They even generate ratings (for what it’s worth) for the suitability of the lens/body combination for landscapes, portraits, journalism, sports and family use. These ratings are based on specifications and measurements only: they do not include value-for-money, or other subjective things like “build”, “bokeh” and “focus speed”. Or the inevitable statements about weather-sealing or what does/doesn’t move when you zoom/focus. DxO knows this, and has reserved space on their site to link to “human-written” reviews.

In PhotoZone you will find hundreds of different lenses, but tested on a minimum number of camera bodies each. The limited number of camera bodies has benefits (less data, less work), but also drawbacks (readers would prefer to see the results for the body they would use). Thus in PhotoZone you will find 3rd party lenses (Sigma, Tamron), famous specialty brands (Zeiss, Leitz), but also exotic lenses like a Russian fish-eye or Korean wide-angles. In DxOMark you will find most relevant camera bodies, but currently “only” about 14 lenses per major camera brand. In contrast, PhotoZone has over 100 Canon-mount lenses (from Canon or others). It is hard to say which strategy on allocating testing budget/effort is better – they are just different. I suspect that the kind of person that does research before choosing a lens will end up using both sites for complementary information.

Lightroom 3 review

Adobe released Lightroom 3.0 on June 8th, after eight months of public beta testing.

I simply kept using LR 2.7 during the beta testing period as the beta version didn’t allow you to easily import LR 2.x databases, and I didn’t want to run any risk with my existing catalog data. So, even though official releases can still have some bugs, I upgraded to LR 3.0 as soon as the final version was available.

For an overview of what’s new in Lightroom 3.0, see for example Adobe’s own site.

Upgrade process

Lightroom 3.0 installs itself alongside the LR 2.x version. When the software is first run, a new LR v3 catalog (=metadata database) is generated alongside the old version 2 catalog. This means you can go back to the old version if necessary. For my catalog containing 25000 images (and over 100,000 keywords), the conversion took about 15 minutes.

Lens correction modules

Lightroom 3 has the option to correct vignetting, lens distortion and lateral chromatic aberration for

  • Canon (26 lenses and 2 point-and-shoots),
  • Nikon (7 lenses and 1 point-and-shoot),
  • Sigma (2 APC-C lenses and 3 full-frame lenses)
  • Sony (Sony DT 18-200mm only), and
  • Tamron (Tamron DI 28-75mm only)

This can be seen as a simple equivalent to DxO’s Optics Pro lens correction modules: the program automatically corrects these defects based on calibration data provided by Adobe (in some cases with support from the lens maker). I still hope that Adobe will acquire DxO’s technology – but this seems less likely now that Lightroom 3.0 does the low-hanging fruit part of what DxO does.

There are Canon lenses supported at present (2 of my 3 lenses; 100mm f/2.8 macro missing):

Canon lens support in Lightroom 3

The first lens is incidentally the Canon PowerShot G10/G11 point-and-shoot camera, but can also be used for the Canon PowerShot S90. The number of supported lenses will likely continue to grow: modules can be provided by Adobe, third parties, or even by end-users (Adobe provides software for this).

Support for managing video files

My directory tree containing all my pictures also contain a few dozen short HD video fragments made using my Canon 5D Mark II. It is a good idea to run “Synchronize Folder…” on the file system because this allows Lightroom 3.0 to find and import these videos. In my case, it also picked up some JPGs that my daughter had made with her camera and had manually placed in the directory tree. The support for videos is currently pretty basic: you can see a thumbnail, can view it using an external application (e.g. Windows Media Player), can add keywords, and can (obviously) export the file – which in this case just means copying the file as-is.

This isn’t much, but should be enough for now to prevent the following scenario that was easily possible with Lightroom 2 with a newer model camera:

  1. you take hundreds of pictures (JPG or Raw) with your DSL, but also a video (.MOV)
  2. you use Lightroom 2.x to import the pictures from your flash card. It warns that there are some movie files, but it doesn’t do anything with them.
  3. you are eager to see your pictures, so you start running Lightroom (adding keywords, deleting the weak images, etc.).
  4. you put your flash card back in the camera and… reformat the flash card: the video files are now lost.

Lightroom 3, in contrast, imports both the pictures and any file format that it recognizes as videos (I have seen .mov, .mp4, .avi work).

The 2003 versus the 2010 “process”

Adobe’s original image improvement “flow” or “process” was getting a bit out of date (it hasn’t fundamentally changed since Lightroom was introduced in 2003). So the Lightroom engineers needed a way to improve this without causing old photo’s to suddenly start looking slightly different. Thus by default, Lightroom 3 still uses the “2003 process” for existing images in catalogs and uses a new “2010 process” for anything that is newly imported. You have full control over which of process you want – these are just the defaults. The main improvement in the 2010 process is supposedly the handling of high-ISO images.

Below is a 100% crop of a raw image taken with a Canon 5D Mk II using a 24-104mm f/4L IS USM lens. The lens is good, but not great, so we can see some lens artifacts when we zoom in all the way.

Image EXIF data: ISO 200 with Highlight Tone Protection enabled (essentially underexposed!), 32mm, f/6.7, 1/250, tripod, raw @ 21 MPixels. In addition, the original image was by 1 stop too dark (probably due to spot metering). The HTP and the underexposure together mean that the dark parts of the image exhibit chroma noise – even at 200 ISO. Warning: the differences between the images is very small. I will point them out, but if you want to compare them, you can download the files and compare them in a slideshow-like tool.

100% crop of a 21 MPixel image using Lightroom 3.0 using 2010 process without lens correction

In this image (remember that you are seeing only 1.1% of the surface area of the full image – the full image is 10× wider and 10× higher), look for:

  • the purple fringe at border between sleeve and Leigh’s arm. This is chromatic aberration. It is not too visible here because we are not too far from the center of the image and the image quality would have been visibly worse at 24mm.
  • the purple color noise in the lady’s gray sweater and my blue sweater. Surprising in a 200 ISO image, but this is again because of the HTP setting and accidental underexposure.
  • the moiré in the striped pink blouse. A resolution of 21 MPixel may sound more than high enough, but it is actually not too high by modern standards: it corresponds to the same pixel pitch as an 8 MPixel APS-C camera.
Test image using Lightroom 3.0, 2010 process and lens correction

This image is very similar to the previous one. But look for:

  • There is less/no purple fringe at border between sleeve and arm. The lens correction module for the 24-105mm lens has automatically fixed this.
  • When you compare this image to the one without lens correction, you see slightly different cropping on the left side. Check out the pearl necklace. This is due to the circa 1% distortion: you lose a few pixels.
Test image using Lightroom's 2003 process with lens correction

This image is again quite similar to the previous two – but things to look for:

  • there seems to be less purple color noise in the lady’s gray sweater and my blue sweater.

Adobe obviously intends the 2010 process to outperform the 2003 process. In this case, the 2003 process does a better job (argggh; the first comment by a reader incidentally seems to confirm this). Adobe demonstrates the differences between the two processes mainly using high-ISO images.

Test image using DxO v6.2 with lens correction (and black level and exposure tweaking to match the overall appearance of the Lightroom output)

The same file processed using DxO Optics Pro version 6.2. DxO specializes in noise reduction and correction of lens aberations. Using default settings, I would say it has gotten rid of the chroma noise, but at the cost of detail in the sweaters. This essentially means that when your image is too noisy, you use spatial low-pass filtering to reduce the noise – at some loss of detail. You can tune all these settings in both Lightroom and DxO, so you may be able to fix this by moving away from the default settings – after all it is a critical trade-off. Note also that the aliasing in the pink blouse is less than in the Lightroom images, suggesting more effective de-mosaicing filtering.

Lens correction benefits

The cropped image used above does show too much lens correction. So let’s look at another image shot using a Canon EF 24-105mm f/4L lens. This image with lots of straight lines and indirect lighting incidentally shows a hall where Belgian coal miners used to shower after their shift.

ISO 200, 24mm, f/7.1, 1/100, full image (move mouse over image to see result of lens correction)

If you compare the two images, you see clearly that the lens has quite some distortion at 24mm (and with a full frame sensor). The Photozone.de website even calls this 4.3% distortion “massive“. A direct comparison also shows that the light fall-off of 1.5 stops in the corners (this is at f/7.1; it would have been more at f/4).

100% crop of a corner (move mouse over image to see result of lens correction)

The cycan and magenta fringes are clearly visible (both around the roof and between the tiles) and are largely corrected using the lens correction module. In the crop, both images are distortion corrected for practical reasons. It is worth noting that although all of this is pretty advanced stuff, you only need to click on a checkbox to activate lens correction.

iPhone support

The iPhone 3G and 3GS are both supported with respect to the camera inside these phones. On the one hand these are popular camera’s and undoubtedly have medium quality optics. Possibly Adobe added this as a bonus for people with fancy cameras who also use iPhones. Anyone using the iPhone as a main camera probably doesn’t care too much about image quality.

I am also pretty sure that the Canon PowerShot S90 is also supported. In fact, the Adobe software silently did distortion correction for this model without informing the user or giving users the option to enable or disable the feature. This was a design decision by Canon: correct residual lens aberrations in Canon software and where possible also in major 3rd party software.

Tethering your camera

Connecting your camera via a USB cable to a laptop or desktop is easy and can be useful. Every picture you take is sent over the USB cable and shows up in Lightroom pretty much immediately. The picture that shows up in Lightroom can automatically be given some keyword or preset or get the same adjustments as the previous image.

You can either trigger the camera’s shutter using the camera’s shutter button(s), or trigger the shutter from the computer (using a mouse or keyboard). You can also see important camera settings like ISO/aperture/shutter_speed/white_balance, but you cannot adjust these from the computer. I didn’t manage to start a video this way, but that may not be terribly useful anyway.

So the tethering works (at least with a hand full of recent cameras) and is easy to use. But don’t expect the ability to really control the camera from your armchair.

Oddities and bugs

  • Lens correction and image resizing. The lens correction module fixes distortion, resulting in a warped picture that is then automatically cropped back to a rectangle. You lose some pixels at the edges. The resulting image size in pixels is, however, identical to the original image size. I guess that choice is ok for casual users, but how about demanding users?Another way to explain what I mean: if you straighten the image using Lightroom (e.g. rotate it by 1 degree), you get a different image size and different aspect ratio. Distortion correction is somewhat comparable, but behaves differently.
  • File count in the keyword hierarchy. I have a keyword hierarchy that includes Locations > Europe (1) > Belgium (32) > Brussels (61) > Manneke Pis (3). The numbers indicate the number of pictures. In Lightroom 2.x the number following Europe would show the total number of images classified as Europe, including images that had only Belgium as keyword. This is clearly no longer the case. I probably only have 1 image that is labeled Europe that is not attributed to any specific country. I can click to get all Europe images (11017), but how do I get the odd one? If I cannot find it easily, why show me the number?
  • Tethered shooting. Turning off the camera while it was still connected to the computer for tethered shooting, gave a weird colored animation on the LCD on the back of the Canon 5D Mark II: essentially the camera thought it still needed to store a file. The animation looked like it might be designed for the WiFi adapter (which I don’t have) because something similar happens when the camera starts messing with FTP and HTTP protocols to push files to a nearby server.
  • Editing video capture time. Lightroom 3 doesn’t claim to be able to edit the capture time of videos, but this is a bit of an inconvenience. I had previously (LR 2.x) adjusted the times of a set of images to match the local timezone in which they were shot. Now I wanted to do the same for the videos in LR 3.0. It can’t. So now my videos show up in the wrong locations when the files are sorted based on capture time. I guess the capture time can be edited: the time is stored somewhere in/with the file itself. And if absolutely necessary, you could edit just the capture time as stored in the database (and risk losing the change if you resynchronize metadata).
  • Lens correction and freedom of choice. By default, lens correction will use the lens the image was taken with. Surprisingly, Lightroom 3 also lets you use the correction models for other lenses. Even lenses from other brands and for lenses that don’t have the focal length you are using. This is nice for playing around with (“what if I select a fish-eye”) or to use a similar modular if the one you need is not available. But there is no warning if you select a “wrong” module: it gets stored in your catalog.
  • Deletion. When a single image is viewed within a directory, and the image is deleted, Lightroom loses track of where you were within the directory. This applies for both Development view and Library view. This behavior is different than Lightroom 2, and a bit of a nuisance.