Lightroom 4′s PV2012 image enhancement algorithm

In 2009 I wrote a small article here about a new class of image enhancement algorithms. Such algorithms made pictures look sharper by added local contrast and brought out details in both shadows and highlights. And they did this without adding halos around high-contrast edges such as the transition between foreground and sky. The article focused on a research paper by Farbman, Fattal, Lischinski and Szeliski (FFLS, 2008).

The brand new Lightroom 4 (and the associated versions of Adobe Camera Raw 6.7) has now incorporated similar technology that is based on a newer research paper entitled “Local Laplacian Filters: Edge-aware Image Processing with a Laplacian Pyramid” by Paris, Hasinoff and Kautz (PHK, 2011).

The implications of this new Lightroom feature for photographers are significant enough that I will gradually add more details here as I upgrade to Lightroom 4 (possible since March 5th) and get hands-on experience with it.

What’s so important about this technology?

On superficial examination, the two mentioned research papers (and there are many more where that came from) seem similar enough. They are both trying to give users the ability to make details in images more striking (local contrast enhancement) without creating undesirable artifacts such as halos.

The weird thing about this research is that they don’t distinguish between minor image tweaking (e.g. typical raw converters sharpen images to some default degree) and image modifications like HDR which also sharpen the image but can lead to unnatural results if overused. This is probably because there is no sharp boundary between enhancing a little and going overboard with your the settings.

Thus many of the example images in the research papers look like what we photographers would consider HDR photography: in HDR photography, you want to show a broad range of lightings without making the local contrast look flat, and without making the overall picture look artificial. Probably the authors would answer that HDR is an application area where this kind of algorithm is needed – but they can also be used for “normal” images shot with a single exposure in a Normal Dynamic Range situation.

Something similar applies to the Lighting module in DxO Lab’s “DxO Optics Plus”. It probably uses a similar type of approach to boost local contrast. I tend to consider it an HDR-like technique.

Alternative algorithms

A claimed key benefit PHK as chosen by Adobe is that the algorithm is simpler and thus takes less processing time. In fact, the paper’s introduction starts off by saying that the widely known (in the right circles) Laplacian Pyramid technology was underrated, leading previous researchers (including FFLS) to develop more complex algorithms to compensate for its shortcomings.

It is worth noting that the first author in PHK works for Adobe. But these are scientifically reviewed papers published in respectable conferences – meaning the authors have to be very rigorous about their claims, when these hold and what evidence and even counter-evidence there is for these claims.

Where do I find this in Lightroom 4?

Should we care?

Posted in Photo Technology, Software | Tagged , , , | Leave a comment

Exposition (2 of 2)

This is the poster (designed by Ria van der Ploeg) for an overview exhibition of four friends with a somewhat comparable “pure” or classical style of photography from our local photo club:

The exhibition will be opened on Sunday, January 29th 2012 by our photography mentor, Hans Zoete. The exhibition is at an art gallery in Eindhoven and lasts 6 weeks. The exhibition website (http://www.puur-2012.nl) is in Dutch but largely consists of images.

Posted in Peter, Photos | Tagged , , , , | Leave a comment

Exposition (1 of 2)

This is the poster (made by Ariejan van Twisk) for the yearly exposition of my local photography club “Fotogroep Waalre”. The poster shows a picture by Aad Schoenmaker that I really like. Although the picture may look like it was created in Photoshop, it is essentially an as-taken photo. I deliberately won’t explain how Aad managed to get himself warped into this cyberzone, and leave it to you to guess (or ask Aad).

Posted in Temporary | 2 Comments

Fuji’s Organic Sensor Technology

In 2009, Fujifilm was granted a patent on an organic layer to improve CMOS image sensor performance. The thin organic layer converts incoming light to electrical charge (like a solar cell) that is then detected by more or less conventional CMOS image sensor circuitry.

Fujifilm’s “organic sensor” technology started to receive attention recently due to speculation that it would be featured in a future high-end compact camera (expected in early 2012; possibly called the “LX10“). This camera is expected to have an interchangeable lens and target the same market as the highly successful Fujifilm X100.

Update: the Fujifilm X-Pro1 (March 2012) turned out to not contain this particular technology. Instead it has a color filter array layout which avoids the need for an anti-aliasing filter.

Although Fujifilm has not confirmed the use of their organic conversion layer in the forthcoming camera, a Fujifilm executive claimed that the interchangeable lens camera (possibly with an APS-C sized sensor) will outperform the current generation of full-frame sensors in terms of noise.

The technology

Fuji sensor with "panchomatic" organic layer

Fuji sensor with "panchromatic" organic layer (taken from Fuji's 2009 white paper)

A color sensor using this technology uses a conventional color filter array to make individual pixels sensitive to green, red and blue light. In a 2009 white paper, Fuji states that the layer is insensitive to infrared light. Therefore no IR absorption filter is needed, which is in itself a minor benefit.

The organic layer is labeled “Panchromatic Photoelectric Conversion Layer” in the above diagram by Fuji. The layer is 0.5 μm thick, and converts almost all visible light to an electrical charge (electrons and holes). The organic layer, which closely resembles a solar cell technology, is sandwiched between a negatively charged transparent electrode (like in LCDs) and an array of positively charged square “Pixel Electrodes” that form the actual imaging pixels. The latter are not transparent.

Electron microscope image of the upper surface of an array of 3 μm (aka um) pixel electrodes. The small circle within (below) each square the metal conductor that connects each pixel electrode to the transistors below.

In the 2009 prototype the pixels are spaced 3 μm apart. This would result in a resolution of roughly 40 MPixels for an APS-C size sensor or 22 MPixels for a Four-Thirds size sensor. Due to the gaps between the pixel electrodes, 85% of the surface area is used to capture charge. The possible loss of 15% of the incoming light can probably be neglected anyway, but these gaps could be further reduced if Fuji wants to and may not play any measurable role at all if the free electrons generated above the gaps reach the pixel electrodes anyway due to the applied electric field.

So a key benefit of this technology is that it is efficient compared to alternative ways to make arrays of relatively small pixels. With small pixels in a conventional CMOS sensor, the gaps between pixels (or the inverse “fill factor”) becomes a problem, especially because the wiring needing to access the pixels lies on the upper (outside) layers of the sensor. Backside illumination (BSI) addresses the latter by grind down the chip and illuminating the sensor from the back.

Fuji stresses that their sensor doesn’t need a micro lens array to funnel the light to the pixels. Apart from a reduction in component count, this avoids color errors at the edges of the sensor. This occurs with small pixels due to crosstalk between the color channels when light hits the sensor at an extreme angle.

The Benefits

One key benefit of this sensor technology should be its sensitivity to light: Fuji measured a quantum efficiency of 65% in 2009 (for green light at 550 nm) but stated that this can be improved by adding anti-reflection coatings. The quantum efficiency figure seems competitive compared to relatively expensive backside-illuminated sensors as used in recent high-end sensors with small pixel dimensions (such as the Nikon P-100, Canon S100, iPhone 4 and iPhone 4s).

This translates to a slight sensitivity gain – and would behave like removing a very light gray neutral-density filter. Using the available data, I am unable to calculate how big this improvement is expressed in ISO. Note that the prototype sensor described in 2009 will obviously differ from Fuji’s current technology anyway.

A more interesting question is what impact the technology change has on dynamic range. The ability to funnel more light to the sensor helps sensitivity and noise. But dynamic range depends on the ratio between the maximum amount of charge the pixel can hold (at saturation) compared to the noise level. Given the high expectations set by Fuji for the 2012 camera, the benefits of the organic layer must somehow translate to a dynamic range improvement (as differences in quantum efficiency between modern sensors tend to be limited).

Impact

It is a bit early to speculate on the impact of an unannounced camera of only leaked images seem to exist. Especially when we can only guess whether it utilizes Fuji’s organic photoelectric conversion layer. But what would happen if the technology delivers? In particular, would it really close the image quality gap between say full frame and APS-C sensors?

The key question for strategists is thus whether an organic layer would improve all sensor sizes and sensor resolutions to the same degree. If so, it could simply improve the quality level of what could be manufactured, but doesn’t change the landscape of Four-Thirds versus APS-C versus Full Frame versus Medium Format sensors. Fuji’s white paper starts off with

We proposed a new CMOS image sensor with a thin overlaid panchromatic organic photoelectric conversion layer as the best candidate for sensors with reduced pixel size.

and its title also mentions reduces pixel size as the highlight of this technology.

This suggests that the organic layer helps increase the design of high-resolution sensors regardless of their size, by improving their light gathering efficiency. This implies improved performance of tiny compact camera sensors with mere 1 μm pixels, and would provide an alternative to the cumbersome back-side illumination technology used in some compact cameras and camera phones.

It could also pave the way for 20 MPixel Four-Thirds sensors and 40 Mpixel APS-C sensors, and 80 MPixel full-frame sensors. Such (relatively speaking) extreme resolutions when using 3 μm pixels might be overkill for many lenses, but would indirectly increase image sharpness by promoting cameras designed without an anti-aliasing filter. Such an AA filter softens the image to get rid of details that are too fine to be resolved. But in doing so, AA filters also blur the finest details which the sensor can capture (in essence an anti-aliasing filter is a sheet of very finely frosted glass).

Because extreme resolution is not always needed and is sometimes even useless, a high-end camera could conceivably provided modes whereby the full pixel resolution can be downscaled within the camera. For example a full-frame camera 80 MPixel camera might normally record at 40, 20 or 10 MPixel resolution. But can be set to resolve in 80 MPixels if the user wants this. Current Canon SLRs already have a similar resolution scaling feature that Canon calls SRAW, but the option is not used very much because it gives loss of detail without shrinking the file size much. Note however that this route should give fine image quality, but likely costs more energy than the equivalent lower resolution sensor (more samples to digitize, more digital operations).

[ last modification: 26-Nov-2011 ]

Posted in Photo Technology | Tagged , , , | 5 Comments

Fotogroep Waalre 8-Jun-11

2011-Rotterdam-024 (100 ISO, 135mm, f/7.1, 1/400)

This composition is essentially about diagonal lines in modern architecture.

The Toren op Zuid (aka KPN Gebouw) by Renzo Piano has a façade with an overhang of 6°. The single external pole is for the required extra support. The foreground on the left is part of the nearby Erasmus Bridge.

2011-Rotterdam-015. Click to view larger (100 ISO, 70mm, f/10, 1/400)

Nearby there is a new high-rise residential building (“New Orleans”) and cranes for the next high-rise project (“De Stad Rotterdam”). Some viewers complained about the distracting cloud “coming out of” the building. The Dutch Photography Museum is incidentally housed somewhere below the green loft on the right.

Overview of Wilhelminaplein in Rotterdam showing the relationship between both previous images.

Posted in Peter, Photos | Tagged , | 2 Comments