DxOMark @ 269 cameras

DxOMark score versus (a) sensor size, (b) price, (c) launch date, (d) MPixels. Click on image to view larger.

Here is another update about camera sensors in cameras as tested by DxO Labs (www.dxomark.com). DxO’s test essentially covers the noise and dynamic range of cameras – it doesn’t cover resolution, focus speed, ease-of-use, etc.

Modern cameras span a range of over 70 points in this pretty rigorous benchmark. In reality the range is larger because DxO doesn’t test camera phones and other good-enough-for-a-selphie models. A 3 point difference is barely visible to specialists, 10 points is readily visible, 30 points tends to be noticed by everyone.

Since my previous posting, 18 new cameras have been tested by DxO Labs. I also rescaled the graphs to allow for scores above 100 and extended the timeline somewhat. For an in-depth explanation of what you are seeing here, check my January 2013 article at either DxO Labs or Luminous Landscape.

Some of the highlights hidden within the newer benchmark results:

  • Epic Dragon scores 101 points
    The 19 MPixel Epic Dragon video camera can take better stills than any existing still camera. This is especially usual given that the Epic Dragon has a moderate sensor size. Because it is only a prototype, DxO did not include the Epic Dragon in its normal test result database. After all, you can’t buy it.
    Note that the Epic Dragon is a component of a modular video system: without a lens, we probably wouldn’t recognize it as a camera.
    But its score demonstrates that today’s APS-H (1.3x) and larger sensors can break the psychological 100 point DxOMark barrier. So it is only a matter of time before we see commercial cameras scoring over 100 points in this benchmark. This may be asking too much for the expected Canon 7D Mark II with a 1.5x sensor, but could be achievable for one of the new 50 MPixel medium format cameras with a Sony sensor (Hasselblad, Phase One, Pentax).
  • Mirrorless
    Some of the top models (highest performance in its price class) are now mirrorless cameras. Somehow these are all Sony models (A3000, A5000, A6000, A7, A7r). Sony is currently the leading sensor manufacturer. Manufacturers like Nikon use Sony sensors mainly to boost the image quality of their existing SLR product lines.
  • Sony A7s
    The Sony A7s has a very low resolution (12 MPixel) for a full-frame camera launched in 2014. Its score is a bit low for a modern-full frame sensor (except for its High ISO subscore). From a still camera perspective it is not obvious why Sony introduced a sensor with such large pixels: it already supplies higher image quality at with higher resolution sensors (e.g. A7 and A7r). The answer to the puzzle may be probably related to the A7s unique 4K video capabilities. I wouldn’t be surprised if the A7s is also partly intended to test the market.
  • Nokia Lumia Pureview
    The Nokia’s Pureview smart phone models caught quite some attention with their 41 MPixel resolution. They score somewhat lower than cameras with a similar sensor size (e.g. Fujifilm X10). This is likely because the pixel size of just over 1 μm gives fill factor issues: a non-negligible percentage of the sensor area is lost as overhead and not used for light gathering.


DxOMark @ 251 cameras

DxOMark Sensor scores for 251 cameras (click to view larger)

Here is another update about new cameras tested by www.dxomark.com. The test only looks at the noise and dynamic range performance of cameras – it doesn’t cover resolution, speed, ease-of-use, durability, etc.

Since my previous posting, 5 new cameras have been tested by DxO Labs. Modern cameras span a range of over 60 points. A 3 point difference is barely visible to specialists, 10 points is readily visible, 30 points tends to be obvious even when someone is not paying attention to image quality at all:

  • Sony A3000/A5000 (78 and 79 points).
    The pricing of APS-C system cameras with a state-of-the-art sensor has dropped below US$ 500 with the introduction of the Sony A3000 and A5000. Despite the Alpha branding, these are basically NEX models (Sony has dropped the usage of the NEX brand). They thus have E lens mount (as used in the NEX series) rather than the A-mount (as used in the Alpha 77).
  • Leica S medium format (76 points).
    The Leica S medium format camera, despite its $28,000 price,  does not really have a state-of-the-art sensor. It “still” uses a CCD sensor technology, although  recently medium format models with a Sony-built CMOS sensor have been recently announced (by Hasselblad, Phase One and Pentax). CMOS sensors should manage to make medium format cameras more all-round cameras again. Arguably, because medium format cameras are often used in studios or tripods, they historically had more emphasis on resolution, color fidelity and lens quality than on low light or high dynamic range.
  • Leica X Vario (78 points).
    Leica also gives you the option of buying the X Vario which actually performs similarly to the Sony A3000 or A5000, but at a Leica price.
  • Olympus Stylus 1 (51 points).
    The Olympus Stylus 1 scores surprisingly low for a new camera with its SLR-like looks. But looks are misleading here. If you look carefully at the specs, it turns out to have a very small sensor with a 4.66x crop factor. This puts in in the same league as the Canon Powershot S120. The Stylus 1 (51 points) is outperformed by the more compact S120 (56 points).
Canon Powershot S120 next to the Olympus Stylus 1 (www.camerasize.com)
Canon Powershot S120 and the Olympus Stylus 1 have the same sensor size (www.camerasize.com)

State of the DxOMark (camera nerdiness)

DxOMark scores as of Nov 2013 (click to view larger).

I updated my overview chart with available high-end cameras. See http://www.dxomark.com/Cameras/Camera-Sensor-Database for detailed benchmark results for specific cameras. As usual, there is a lot of interesting information in such an overview:

  • Assuming you care about low light and high dynamic range performance, the best cameras have full-frame sensors (the blue dots). You knew that – right? Well, surprisingly full-frame sensors beat even larger (purple, pink, red) sensors. So don’t bother spending big money on a medium format camera unless you really need the super-high resolution. Or need it show that your equipment is clearly on a price class of its own.
  • The so-called APS-C cameras with 1.5x or 1.6x sensors have improved. Examples: the Nikon D5200 and D5300.
  • The Sony NEX-5R mirrorless (which is 1.5x) has a slightly higher price than an APS-C SLR, but the body is smaller and the performance is competitive. Mirrorless models should have the same performance as an SLR with a comparable sensor. A mirror doesn’t add image quality – it just makes a click sound like ke-lick.
  • Canon still has a long way to go to catch up with its APS-C sensors (1.6x). The Canon 70D performs slightly better than the old Canon 7D, but a comparison to Nikon or Sony tends to be embarrassing.
  • Recent Micro Four Thirds cameras (Olympus & Panasonic) have improved and are even ahead of Canon’s APS-C (1.6x) models.
  • The Sony RX100 and RX100-II are still doing fine – at least considering their small sensor size (2.7x or 1/1″ ). The Nikon Series 1 is technically not state-of-the art, but nice if you like white or pink gear: it targets a young Asian lifestyle market.
  • The premium pocket cameras have improved. Especially the 1/1.7″ sensor models such as the Canon Powershot S120 and G16 and their Nikon equivalents.
  • The best deals if you need a high quality model can be found at the top edge of the cloud in diagram “b”: you get the highest quality in that price range. Note that the prices shown are official prices at introduction, and will differ from current street prices. These deals include:
    • The Nikon D600 and D610. These are essentially the same camera, but the D610 resolves a dust issue.
    • The new Sony A7R mirrorless. Note that this model uses Sony E-mount lenses, but actually requires new Sony full-frame E-mount lenses called “FE”. So it will take a while until there are enough lens options.
    • The Sony RX1 and RX1R. These look overpriced (and probably are – although I ordered one myself), but their price does include an excellent 35mm Zeiss f/2.0 lens. On the other hand, they do not come with an optical or electronic viewfinder. These cost about 500 US $ or Euro extra. Lens hood pricing is  joke (so look into the Photodiox accessories).
    • The Nikon D5200 or D5300. Both have a 24 MPixels state-of-the-art sensor, but the newer one gives sharper images (no AA filter) if your lenses are up to the challenge.
    • The Nikon D3200. Also 24 MPixels with state-of-the-art sensor technology.
    • The Pentax K50 and K500. A somewhat overlooked brand.
    • The Nikon Coolpix P330. A “take me everywhere” camera at a lower price point than the excellent Nikon Coolpix A or FujiFilm’s X-100s models.

Note that some major new camera models are not shown because DxO Labs simply hasn’t tested them yet. These include:

  1. The new full-frame Nikon Df (with the professional Nikon D4’s 16 Mpixel sensor). It should score about 89 (D4) for $3000 – nice, but not sensational unless you insist on a retro look and feel.
  2. Most FujiFilm X-Trans models have not been tested. Tests may be delayed because they have a non-standard color filter array (complicating raw conversion). The CFA design allows the sensor to work without a low pass filter. Alternatively, the missing tests may be because FujiFilm is not enthusiastic about their cameras’ DxOMark scores (pure speculation on my part, but the FujiFilm X-100 didn’t score exceptionally well).  FujiFilm high-end cameras are getting a lot of attention from serious photographers who prefer small, unobtrusive cameras with a classic mechanical feel.
  3. The Sony A7. Many people wouldn’t really benefit from 36 MPixels (Sony A7R) without an image stabilizer or a tripod or high-end lenses.

For a detailed explanation of what the benchmark itself means, see http://www.luminous-landscape.com/essays/dxomark_sensor_for_benchmarking_cameras2.shtml. Note that the number of tested cameras has meanwhile increased from 183 to 236 models.


Magic Lantern for the Canon 5D Mark II

What is Magic Lantern?

If you own a Canon 600D, a Canon 60D, Canon 5D Mark II, or certain of their predecessors, you might be interested to hear that you can extend the capabilities of your camera for free (although a donation is requested). This is not by replacing the camera’s internal software by a newer version (recommended, but this mainly fixes bugs), but by adding software from a bunch of non-Canon developers. This Magic Lantern software extends the existing Canon software with many new features that target technically inclined videographers and photographers.

Features for photographers

Magic Lantern was originally created mainly for those who use Canon DSLRs for serious video work. I don’t know much about video, so I will only describe features that help photographers.

The features are somehow largely centered around Liveview and likely benefit photographers most who sometimes need to do “slow” photography: they use a tripod, use tethering in a studio to check focus, have a complex setup or simply want to have maximum control. Having said that, Magic Lantern states that it has benefits as well for photographers that are in a constant hurry: it gives you the option of putting certain options that you use a lot under a particular button.

A few of the key features:

  • focus peaking – whereby the Liveview image displays which parts of the image are in focus. Useful when you want to carefully control what is in focus. This can be seen as an alternative to tethering your camera to a computer via USB in the studio.
  • exposure clipping – the Liveview image can show which parts of the image will be too light and too dark using overlaid zebra stripe patterns.
  • more on-screen data – for example the current main camera mode (e.g. M), focal length and focus distance.
  • focus loupe – you can see part of the image zoomed in 2x or 3x to check sharpness. This feature is fancier that Canon’s counterpart and can even simulate what a split screen focus aid used to look like.
  • interval timer – you can take 100 pictures at 60 second intervals to show a flower opening. Or 1000 pictures at 1 hour intervals of a construction site – all providing you can get your battery to last.
  • triggering exposures – the shutter can automatically fire if the scene brightness or content changes significantly. Essentially a makeshift motion sensor.
  • automatic HDR – not only can the camera take a series of images at different exposures automatically, but it can take the entire series at one press of the button. It can even determine how many exposures are required automatically (or manually) and give you a rough preview of the merged image. Pretty cool. Essentially this gives your 5D2 a feature found in the 5D3, but without the artsy options: you do your real HDR merging afterwards on a computer.
  • improved mirror lockup – flip up the mirror a few seconds before taking the picture to reduce vibrations. The Canon equivalent is relatively tedious to operate.
Display when shooting video using Magic Lantern

The actual list of features is about as long as the list of features that your camera originally came with. So some people only use 2 or 3 of the new features. Others actually do read the software manual and experiment around (takes an evening – just like Canon’s firmware).

Installation and risk

There are risks involved in tinkering with complex equipment. My feeling is that the risk is comparable to opening up PCs to upgrade memory. If you never did something similar, you can get someone else to install Magic Lanterns (ML) and show you the basics.

The risk is lower than you might expect because ML doesn’t simply overwrite Canon’s software: it runs as an add-on and (in most cases) you will not see changes to the menus provided by Canon. There is a simple procedure to uninstall ML entirely.

This is essentially how ML works under the hood:

  1. A minor modification to Canon’s software makes the camera Magic Lantern aware. Comparable to a boot loader on a PC. ML is incidentally not the only party that does this (there seems to be a USB remote controller that uses the same trick to extend Canon’s software).
  2. Whenever you activate the camera, the firmware first checks for the presence of special non-image files on your flash card. If found, it loads Magic Lantern from the flash card. This does not visibibly delay camera operation. The ML software sits alongside the Canon software in camera memory (RAM). If the ML files are not found on the flash card (or you hold  down a button while turning it on), Magic Lantern is not loaded and you get  unmodified camera behavior. Alternatively, you can choose to carry memory cards with and without ML.
  3. Running ML,
    • the optical viewfinder information display is unchanged
    • the LCD viewfinder for LiveView displays significantly different information
    • Canon’s own menus (Menu button) are for 99% unchanged
    • you can view ML’s own menus by pressing the Erase button while in Liveview mode
  4. Whenever you make changes to ML settings this is written to the flash card for the next session. Some changes are also stored in the camera’s non-volatile memory (e.g. when ML menu’s interact with existing Canon features?)
  5. The ML files stay on the flash card, even if you erase the card using the camera. Actually ML formats the card and then writes the ML files back from memory. If you erase or format the card entirely using a PC, you need to reinstall the ML files onto the card. Until then, you will be operating without ML when you use that card.

Quality and stability

I cannot give you hard numbers, but since version 2.3 the stability seems to be close to that of Canon’s own software. Both have occasional bugs and both try to fix these bugs as soon as possible. ML is an open source project, so anyone with (considerable) programming skills can contribute.

All this doesn’t mean you can never run into a problem: ML software adds complexity to the entire setup, and strange combinations of features may give strange results. But if you stick to mainstream usage of the features (= use them more or less as documented) you should be alright.

Some features are clearly marked as “for very advanced users”. One example is the ability to take pictures in a low-res format while in Liveview mode without any shutter motion or sound whatsoever. A bit weird, and it actually seems to work, but you won’t be using this unless you are a video technician or are motivated enough to figure out how to deal with these “422” encoded frames.

A final example is a menu item called “Don’t press this”. The user manual just says not to press it. Actually it probably doesn’t do any harm (otherwise why give it such a tempting name), but I don’t want to press it just yet. I suspect it contains a game that is totally not camera related. After all, your camera is just a computer with an industrial strength webcam attached as a peripheral (at least that is how geeks tend to see it).


So far, things are going well with my own use. And ML has thousands of heavy users who rely on it on a daily basis. The documentation is actually pretty good – including the description of the risks involved. But…

  • It will only install on the latest version of Canon’s firmware. So you need to upgrade a 5D2 to v2.12 before you can install ML. A sensible choice by ML to minimize risk.
  • Running ML will slightly increase battery drain. Essentially because it gives the ARM processor more work to do because of extra features. It will increase batter drain a lot if you start using Liveview more than you previously did.
  • ML increases overall system complexity somewhat: it is like upgrading from a 5D Mark II to a 5D Mark III – more features which you may or may not use.
  • ML is not available on all currently Canon cameras (notably not the 7D or 5D Mark III so far). ML is written by volunteers and all this is a lot of work.
  • Something could go wrong. But the manual explains how to get the camera up and running again in the more common cases. As far as I can tell, the risk of loosing images stored on the flash card is absent, but there is a risk that you may need to briefly remove the batter to recover. A quote from the Magic Lantern FAQ:

In practice, we are doing our best to prevent these situations, and thousands of users are enjoying it without problems. However, this does not represent a guarantee – use it at your own risk.

Canon 5D2 – black dot plague and its cures

The black dot problem

The Canon 5D Mark 2 was announced on Sep 17th 2008 and many early orders were fulfilled around Christmas. Unfortunately, the camera turned out to have a defect that was particularly noticeable in pictures featuring Christmas lights. The problem shows up as black dots directly to the right of small bright lights. Here “to the right” assumes landscape mode and this translates to top (or bottom) in portrait mode. The problem shows up on RAW images and presumably in JPG images as well.

I have only photo showing this, taken during a Winter evening (Canon 70-200mm f/4L IS, 800 ISO, 1 stop underexposed, f/4, 1/60 s, 21 MPixel Raw, camera firmware 1.0.6).

The black-dot problem is exhibited in the Christmas lighting along the edige of the roof
The black-dot problem is exhibited in the Christmas lighting along the edige of the roof
100% crop of the image of the inn
100% crop of the image of the auberge

Pixel peeping to the max

If you would plot the intensity scanning from left-to-right through one of these small highlights, you expect to see pixel brightness rise to  the maximum measurable intensity, a plateau at this maximum intensity (255 on an 8-bit scale), a decrease down to background intensity. BUT (see actual scan at the end of the article), the ramp down “overshoots” and forms a small black dot.

Enlarge (11x) crop of same lamps, but showing too bright and too dark pixels.
Enlarge (11x) crop of same lamps, but showing too bright and too dark pixels.
Enlarge (11x) crop of same lamps, but showing too bright and too dark pixels.
Enlarge (11x) crop of same lamps, but showing too bright and too dark pixels.

You normally only see these black dots when viewing at 100% and can miss them unless you are looking for them. They appear to be a digital processing artifact. One clue is the phrase “to the right of..”. The lens itself has axial symmetry, so will not know what we users consider to be the right side of the image. The sensor itself (at the photosite level) also cannot behave like this. So the problem apparently lies in analog or digital signal processing inside the camera.

Speculations on the cause

One likely culprit seems to be the “highlight tone priority” feature which attempts to avoid blown highlights. This presumably gives a local HDR-like treatment: the area around a highlight is digitally underexposed to compress the scene’s dynamic range. This helps keep the bride’s bright white wedding dress from showing burnt out highlights.

If the “highlight tone priority” algorithm works left-to-right, you could imagine that a bright spot will result in a local adjacent dark spot – just like you are temporarily blinded by the headlights of an oncoming car at night: the feedback loop in your vision which reduces the pupils and probably sensitivity of the retina itself needs some time to adjust to the “normal” darkness again.

But this assumes that “to the right” is somehow associated with “later” in time. A digital filter is normally design to work symmetrically: the information needed to compensate in all directions (left, right, up, down) is available once you have a bitmap stored in memory. So the “black dot to the right” is reminiscent of analog processing whereby the photo sites are read out left-to-right and first processed using analog circuitry. The fundamental reason for this is that an analog filter which fed with a time-dependent signal can only react to the present (current signal) or past (previous signal values) and not to the future (the still to be processed pixels). A simple analog gain control loop (e.g. used to regulate audio levels) shows such behaviour: after a strong signal, it may temporarily be blinded.

Firmware fix

Fortunately, speculations about the detailed cause of black dots are no longer relevant. Canon supplied a firmware upgrade (1.07) in early January which fixes the problem.

The fact that it could be fixed by a digital modification may suggests that it is a digital algorithm problem (you can design algorithms that have this blinded-to-the-right phenomenon simply by emulating an analog filter), or that the analog processing is digitally controlled or that the problem can be masked by digital means. Current collective Internet opinion suggests that critical users seem satisfied with the patch. Unfortunately the firmware patch has the side-effect that the raw convertors created for the Canon 5D2 now need software upgrades but this is a one-time inconvenience for early adopters. So either this patch is fixing the problem rather than masking it, or the masking simply works well enough.

Since I took the above picture, I have upgraded to the 1.0.7 firmware and will hopefully not run into the problem again.

The details are even weirder

Intensity scan done manually at 11x magnification in Lightroom 2.2
Intensity scan done manually at 11x magnification in Lightroom 2.2 (using Adobe Camera Raw version 5.2)

The graph shown above shows the intensity of the Red, Green, and Blue channels when I scanned manually (Lightroom) from left to right throught the topmost (of the the two) Christmas lights. The basics are obvious: the light has a width of about 16 pixels. Blue is a little less intense on the left side, leading to the yellow color. There is indeed a dip around X=2 which is clearly lower than that around X=16. The peak at X=6 is due to the proximity of a second light source.

But there are some puzzles:
  1. The dip at X=2 is not convincingly lower than the intensity at X=-18 or X=13. This might be explainable by strong linearity: you firstly need to combine R/G/B into a single number which might just be lower at 2 than at 13. And you apparently need to subtract a black level from that. Some have reported that the dip is actually lower than the black level you subtract, leading to a negative light intensity.
  2. The RGB readout of Lightroom shows the dip at X=2, while the darkest spot on the screen is clearly at X=0. This is likely a bug in Lightroom 2.2, but needs validation.
  3. The RGB readout of Lightroom (at 11x) shows 11×11 pixel squares. Strangely the Lightroom cursor seems to read out varying intensity within a square (as if you could read out the intensity at sub-pixel resolution). So the actual values will tend to vary a bit if you repeat the experment.

Canon 5D2 – mode dial and target audiences

The Canon EOS 5D Mark 2 doesn’t have the traditional mode-dial settings for Sports, Landscape, Macro, etc. These settings mainly impact shutter- versus aperture trade-offs and possibly burst mode settings.

Prosumer market?

Apparently you are supposed to know what you are doing if you purchase a camera like this. After all, the camera caters to advanced amateurs willing to lug the extra weight around, buy fancy (and heavy) lenses and pay the price premium for all this. The Canon 5D2 and its successor, the Canon 5D, were seen as the camera of choice for professional wedding photographers, professionals on a budget, or even simply as backup cameras for professionals with a top-of-the-line Canon 1D or 1Ds. It is also suitable for the enthusiasts with enough time to worry about equipment details and enough commitment to save up quite some money to make their peers envious with at fancy equipment. All these groups are supposed to know what they are doing, and know how the camera is intended to be used. So you also won’t find a pop-up flash (unlike the comparable Nikon D-700).

Or anybody who can afford it?

But  interestingly, the mode-dial also has two separate “fully automatic” modes for dummies: Full Auto (green, classic) and the new Creative Auto. In the latter mode you get a slider for “light or darker picture” and even one for “blurring the background”. In addition, unsafe settings which may cause problems are locked. So apparently Canon’s product managers didn’t want to rule out users who don’t even want to bother learning about shutter speeds and apertures.

One explanation might be that some people have enough money to buy a camera like this. But don’t have the interest to learn how to use it. So if I owned a large yacht, I guess I would be unhappy if I bought an expensive quality camera but I couldn’t get it to take a decent picture. Let’s hope the salesman was smart enough to tell me to buy a flash unit (although that is still a nuisance to charge and mount).

Interestingly “dummy modes” are conceivably useful if more than one family member shares the same camera. But on the other hand, this may not be the kind of equipment you put in the hands of kids. So what is “creative mode” good for on a camera like this?

Canon Picture Styles and the 5D Mark 2

Picture Styles is a feature is present in Canon models sold in the past year or two. It impacts subtle post-processing settings: 

  • sharpness,
  • contrast,
  • saturation,
  • tone.

The current opinion on the Internet appears to be that Picture Style settings do not change the pixels stored in RAWimages but are available as metadata for use by Canon’s post processing software, DPP. Lightroom and other post-processors currently seem to ignore the selected Picture Style (although the information is stored somehow in the EXIF data). Instead, Lightroom, for example, has presets which aim to provide much the same result as the Canon Picture Styles. These are, however, not identical to Canon’s Picture Styles because everyone uses slightly different algorithms.

It would be nice if Canon could explained this a bit better. They do explain how the Picture Styles impact the sharpening, contrast, saturation, and tone parameters. Canon says that all goes well automatically when you use Canon’s own DPP software o process JPG files. But many people use other software – especially for cameras in this price range.
The latest Adobe Camera Raw 5.2 (used by Photoshop and more to read Raw files) also emulates Canon’s Picture Styles. Interestingly, users seem to start believing that Picture Styles from Canon have some magical authority, compared to say other presets you can find from Adobe, DxO or users.

See manual p59.
See brief discussion on RAW format and pictures styles from Chuck Westfall.