HarvestImagingPTC

Notes on a tutorial about image sensor noise

During late 2009 and all of 2010, Albert Theuwissen published an 26-part series of postings on image sensor noise on his Harvest Imaging website. The series explores various sources of image sensor noise and their relationship to signal strength. The series targets sensor designers, and those who use sensors in challenging applications. Probably many of the Harvest Imaging readers know Prof. Theuwissen from his courses, workshops and conferences.

The series centers around a proprietary simulation model (written in Matlab/C?) where Theuwissen selectively isolates each noise source encountered in a sensor to show its impact on overall image noise. Every installment of the series centers around a graph (James Janesick’s Photon Transfer Curve, PTC) that plots sensor noise against sensor signal. The graph is thus closely related to the signal-to-noise ratio, but the graph stresses how the ratio varies

  • as the sensor is exposed to darkness for varying durations, and
  • as the sensor is exposed to light for varying exposures.

If you are not a sensor expert, you can try to use the series to learn about sensor behavior – provided you can handle a bunch of basic formulas and are willing to learn the associated terminology (which is is not entirely consistent across postings). Hence these notes which try to follow the terminology used in later postings (where up to 15 noise sources needed to be distinguished).

Incidentally, the word “Harvest” in the domain name of Albert’s website is after the title of a Neil Young album: Prof. Theuwissen is somewhat of a Neil Young fan.

Abbreviations

  • CDS = correlated double sampling (a kind of self-calibration technique with differential amplifiers, see Wikipedia)
  • DN = digital number (simply the digital value read out after measuring the analog signal)
  • DSNU = Dark Signal Non-Uniformity (differences in dark current signal build-up due to variations between individual pixels)
  • FPN = fixed pattern noise (small pixel-to-pixel deviations that don’t change over time)
  • k = gain in DN/electron
  • PRNU = photo-response non-uniformity (differences is pixel sensitivity to light)
  • PTC = photon transfer curve = signal versus noise graph
  • RTS = random telegraph signals (random jumping between fixed output levels)

Overview of sources of dark noise (see also posting p=358)

The following table lists all the noise sources that occur in absolute darkness. They also occur when there is light (but with light there are extra noise sources). They are discussed in more detail below.

All noise sources are measured by resetting the pixels, and then reading out the pixel after a short or longer delay. I classified the noise sources based on their time behavior (the table columns) and their source (the table rows). The central message of the Harvest Imaging series is that you can distinguish these noise sources in actual measurements by analyzing noise build-up over time (to distinguish the table columns) and by different ways of averaging the individual pixel measurements (to isolate fixed sensor line and column pattern noise).

The conclusion of the series is that you can distinguish many of the noise sources by appropriate measurements on a sensor. And that the estimated parameter values can be pretty accurate.

Fixed-Pattern Noise Non-uniformity
Noise
Temporal Noise Shot Noise (Poisson statistics)
Scales with Constant value time or dark current
Constant std dev Sqrt(time) or Sqrt(dark signal)
Pixel-level Pixel defects (p=329) & RTS (p=344) & Pixel FPN (p=358) Dark Signal Non Uniformity (p=84) Temporal
pixel noise
(p=211)
Dark current shot noise (p=48)
Output-level Amplifier offset (p=154) Output amplifier (p=263)
Row-level Row FPN (p=243) Temporal
row noise
(p=243)
Column-level Column FPN (p=229) Temporal
column noise
(p=229)

The Amplifier offset (p=154) has such a bad effect on low signal measurements that it is assumed to be corrected away in most of the PTC graphs.

A noise source not listed above, Saturation Non-Uniformity (p=142), is only relevant for severely overexposed pixels. This can happen during normal exposures, but this part of the dynamic range is normally hidden from the user because of its non-linearity and non-uniformity.

Careful: there may be multiple definitions of Temporal Pixel Noise: including or excluding temporal row/column noises. When you just measure the pixel noise, you get “including”, but when you do a lot of analysis or create a synthetic model, you get “excluding”. A similar problem may exist for pixel-level FPN.

The sample sensor used in all computations

Calculations are done on a hypothetical 160×120 pixel sensor. Given the assumed full-well capacity of 17,500 electrons, the pixels may have a pitch of around 3-5 μm. So the data would correspond to a small section of a larger sensor with pixel dimensions that are likely between compact camera pixels and SLR camera pixels. The physical dimensions are not directly relevant for any of the calculations.

1. Dark Current Shot Noise (p=48)

  • Dark Current is what a sensor sees during long exposures with no light at all.
  • Signal proportional to exposure time (leakage?)
  • Noise = sqrt(dark signal) ; Poisson
  • measurement of PTC can tell you k of the system (here 0.15 DN/e-)

2. Dark Signal Non-Uniformity (p=84)

  • Noise to pixel having varying levels of dark current (non-uniformity)
  • Noise = fraction of Dark Signal (here 15%)
  • careful: the terminology switches back and forth between FPN and DSNU
  • later in the series this is simply known as DSNU

3. PTN Curve and Temperature (p=120)

  • Dark Signal and its noise both scale exponentially with temperate
  • The PTN graph shows their ratio, and is thus temperature independent

4. Anti-blooming (p=142)

  • Anti-blooming is a kind of safety value for pixels that overexposed.
  • It causes non-linear response above what the sensor considers to be white.
  • The threshold for anti-blooming introduces significant FPN
  • In the anti-blooming range, noise drops because temporal noise is clamped.

5. Amplifier Offset (p=154)

  • Offsets “in analog circuitry” can mess up readings, especially of low DN values.
  • They simply need to be (accurately) compensated

6. Pixel Noise (p=211)

  • At zero integration time, there should be 0 dark current signal and 0 DSNU noise.
  • In reality, there is signal due to imperfect offset compensation.
  • And noise because there is temporal noise in the analog chain.

7. Column FPN and Temporal Noise (p=229)

  • Columns (by def.) share the same bias and readout circuitry. Can thus have FPN.
  • Column FPN will exceed dark current noise for short enough (e.g. 1 second) integration times
  • Can isolate Column noise by averaging all pixel FPN across all pixels in a column

8. Row Noise (p=243)

  • Defined as row-to-row variation in the average of all pixels in a row.
  • Row FPN noise may repeat very N (e.g. 16) columns. No explanation. Timing?
  • Can be measured by calculating Fourier transform of noise-versus-row#.

9. Amplifier Noise (p=263)

  • Gives non-zero (e.g. 1.2 DN) noise at zero exposure time.
  • Differential amplifier has “correlated double sampling” trick.
  • CDS should reduce offset and low-frequency noise sources.

10. Defective Pixels (p=329)

  • Defective pixels are “stuck at” 0 or 1 and give extreme spikes in the image
  • Are normally compensated for by digital processing (e.g. replaced by estimate)
  • After compensation, their impact on statistics should be small

11. RTS Noise (p=344)

  • Random Telegraph Signals pixels are pixels that hop slowly between fixed levels.
  • Their cause is not really understood.
  • e.g. 5x more RTS pixels than stuck-at pixels
  • RTS noise hardly visible in the PTC curve. Why?

12. Integral Fixed-Pattern Noise (p=358)

  • isolate FPN by averaging enough frames, thus getting rid of temporal noise
  • uncorrected, defective pixels dominate the (short integration time) FPN
  • can measure Row and Column contributions by averaging over pixels & frames
  • can isolate Pixel FPN noise by subtracting (squared) Row & Column FPN noise
  • noise parameters can be estimated from the simulated images (about 10% off)

13. Integral Temporal Noise (p=386)

  • standard deviation is calculated per pixel (by using 100 images)
  • this removes all FPN effects
  • at long integration times, dark current shot noise dominates
  • at short integration times, noise from pixel/row/column electronics dominates
  • not directly possible to split temporal noise in pixel/row/column components

14. Dark image data files are available (p=417)

  • due to public demand, Theuwissen provided a 162 MByte file with all images generated by the simulations (at different integration times in the dark)
  • the download does not include the simulation model that generated the images

15. Let there be light (p=430)

  • initial simulations are without any noise sources on the silicon
  • signal is proportional to integration time (until close to saturation)
  • photons result on average in e.g. 0.3 free electron (quantum efficiency)
  • noise is shot noise and proportional to sqrt(time) or sqrt(photons)

Additional noise sources due to light

Fixed-Pattern Noise Non-uniformity
Noise
Temporal Noise Shot Noise (Poisson statistics)
Scales with Constant value time or photons Constant std dev Sqrt(time) or Sqrt(photons)
Pixel-level Pixel-level FPN (p=501), Pixel Defects (p=527), RTS Defects (p=546) Photo-response non-uniformity (p=454) Photon shot noise (p=430)
Output-level Output-level FPN (p=501)
Row-level Row-level FPN (p=501)
Column-level Column-level FPN (p=501)

Note that the Fixed-Pattern Noise column are not new noise sources: they exist (presumably with the same magnitude) in the absence of light but are measured again below in the presence of light. PRNU and Photon Shot Noise, in contrast, are really new noise sources that increase noise when photons fall on the sensor.

In addition, photons obviously also cause a signal component (which looks like dark current, but is proportional to the photon flux) which is what the sensor is meant to measure in the first place. It is not shown in the above table because it is not a noise source.

16. Photon shot noise (p=430)

  • exposure times here 0.6 s (compared to 60 s for dark current measurements)
  • linear signal response (to time=photos=electrons) until pixel saturation is reached
  • noise is shot noise (Poisson = sqrt(signal)) when all sensor noise is eliminated
  • PTC graph can be used to estimate system-level gain k

17. Photo-response non-uniformity (PRNU p=454)

  • cause is non-uniformities such as varying pixel size or varying quantum efficiency
  • noise is proportional to signal (e.g. 3%)
  • when the signal saturates, PRNU continues to increase (artifact of model?)
  • I checked with Theuwissen: due to saturation FPN

18. Combining both sources of non-uniformity (PRNU & DSNU p=472)

  • Dark signal and photo response non-uniformity look pretty much the same
  • At high light intensity (short integration times) PRNU dominates e.g. by 100x
  • And photon shot noise dominates the dark current shot noise

19. Combining both sources of non-uniformity (PRNU & DSNU p=486)

  • At low light (long integration times) DSNU becomes comparable to PRNU
  • This causes fixed-pattern noise to increase
  • To distinguish the two you need to vary light intensity
  • Turning on both shot noise sources increases the noise level and signal level
  • So the PTC curve doesn’t shift: if you know the signal, you known the shot noise – regardless of whether the signal is dark current, photons or a mix.

20. Pixel/Row/Column FPN noise (p=501)

  • FPN Noise is a sum of Pixel/Row/Column FPN noise
  • Row and Column data can be isolated by suitable averaging
  • Pixel noise can be calculated by subtracting squared row and column values
  • Looks like the Row-level noise value has a typo (not a big deal, I reported this)
  • This posting doesn’t distinguish between dark current FPN and photo-related FPN, but this could be done by varying the light intensity.

21. Temporal Pixel/Output/Row/Column noise (PORC, p=516)

  • adding Pixel/Output/Row/Column temporal noise dominates very short exposures, but is negligible at longer exposures
  • it is probably not feasible to isolate Pixel/Row/Column noise sources
  • Output noise might be measurable by measuring noise far into saturation

22. Stuck-at Pixel Defects (p=527)

  • Story very similar to the defective pixels in the dark
  • uncorrected, defective pixels dominate very brief exposures
  • they can be corrected efficiently using a processor
  • once corrected, they hardly have an impact on fixed pattern noise

23. RTS Defects (p=546)

  • RTS defects had little impact in the dark, so the same applies with light
  • Still unclear why an RTS pixel contributes at least one order of mag less noise than a defective pixel. I posted the question.

24. FPN noise estimation using the complete model (p=559)

  • 15 noise sources are included. The underlined ones are relevant when measuring fixed-pattern noise:
  1. overall amplifier offset *
  2. dark current shot noise
  3. Dark Signal Non-Uniformity
  4. saturation non-uniformity *
  5. Pixel temporal noise
  6. Output amplifier temporal noise
  7. Row temporal noise
  8. Column temporal noise
  9. FPN for pixels *
  10. FPN for rows *
  11. FPN for columns *
  12. PRNU *
  13. photon shot noise
  14. defective pixels*
  15. RTS pixels
  • until corrected, offset dominates low exposure FPN
  • the splitting of Pixel/Row/Column noise is done just like p=501
  • and results in very similar numbers (no typo this time)
  • the sources that could be estimated are marked with a *
    • DSNU would look just like PRNU in the PTC, but should be negligible
    • RTS should be negligible

25. Temporal noise estimation using the complete model (p=578)

  • 15 noise sources are included. The underlined ones are relevant when measuring temporal noise:
  1. overall amplifier offset
  2. dark current shot noise Medium
  3. Dark Signal Non-Uniformity
  4. saturation non-uniformity
  5. Pixel temporal noise Low
  6. Output amplifier temporal noise Low
  7. Row temporal noise Low
  8. Column temporal noise Low
  9. FPN for pixels
  10. FPN for rows
  11. FPN for columns
  12. PRNU
  13. photon shot noise Medium
  14. defective pixels
  15. RTS pixels
  • At low exposure to light, the sources marked Low dominate. They cannot be distinguished using the PTC. Added together they are called temporal pixel noise, but this is ambiguous.
  • At higher exposure to light, photon shot noise dominates. Dark current shot noise should be negligible for these short integration times.
  • At high exposure the pixels go into saturation and there is little noise left (you might be able to estimate the output amplifier noise if you really cared)

26. Lit image data files are available (p=588)

  • Due to public demand, Theuwissen provided a 141 MByte file with all images generated by the simulations (at different exposure times)
  • The download does not include the simulation model that generated the images
  • This is the counterpart to the dark data files provided in p=417
  • The images should look pretty much the same as the dark images (because the ones in the dark are at much longer integration times). I haven’t checked.

4 thoughts on “Notes on a tutorial about image sensor noise

  1. Pingback: DxOMark Sensor for benchmarking cameras | Peter.vdHamer.com

  2. Jack Hogan

    HI Peter, an excellent article. I also enjoyed the one you wrote for LuLa, for which I could not find a feedback button. In that one, in note [36] you write “At 100% gray, 100 ISO gives 43.3 dB. This is lower than the expected 46 dB due to curvature in the graph at high light intensities (likely anti-blooming or controlled overflow circuitry)”.

    I believe that the lower SNR value should instead be mostly attributed to PRNU, which you so aptly describe above.

    Cheers,
    Jack

    Reply
    1. pvdhamer Post author

      Thanks Jack. There is a marginally newer version of the LuLa article on DxOMark itself (but probably requires Google to find it).

      At the moment I am not following Prof. Theuwissen’s blog as well I used to (time constraints). But I am still following the camera industry itself very closely.

      Peter

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>