Google Pixel 4A Camera Noise Model
Introduction
Noise in image sensors stems from a vast array of non-idealities in the readout circuit chain of a CMOS image sensor. Whether temporal or fixed-pattern noise, the result is a lack of clarity, and ultimately, a degradation in the final resultant image. To help explain and characterize this phenomenon, we will begin by first outlining and explaining the relevant noise sources in the standard readout circuit model. This will serve to give a high-level overview as to where the relevant noise sources originate from. After building such a foundation, we will describe and outline our method of characterizing the noise performance of a well-known CMOS image sensor used in today’s most anticipated smartphones, the IMX363 used in the Google Pixel 4A. The results of such a characterization will then take advantage of the ISETCAM software [6][7] to build a model of the sensor’s performance and highlight preliminary results making broad assumptions about the smartphone’s post-processing pipeline.
Noise Sources and the Readout Circuit Model
Most of the noise inherent to the readout circuit chain of a CMOS image sensor can broadly be described as either temporal or fixed pattern. Temporal, much like its name suggests, is non-deterministic and varies randomly across time. The sources of this type of noise tend to be attributed to random movement of some form of carrier such as the random collection of photons on an image sensor (Photon Noise), thermal agitation of electrons in a semiconductor (Thermal Noise, Flicker Noise), or random carrier movement across any sort of potential barrier (Shot Noise) [1][5].
Fixed-Pattern noise (FPN) is the result of spatial variations in pixel output values whether under uniform illumination or not. The cause of such variation can be attributed to device parameter mismatches across the image sensor itself. Since fixed-pattern noise exists in the presence of an illuminant or not, two metrics for the characterization of the FPN exists as the following:
Photo-Response Non-Uniformity (PRNU) or Light FPN:
As a uniform light source falls on an image sensor, each pixel should output the same output voltage. However, due to variations in the circuits themselves (Threshold Voltage Mismatch, Cell Size Variation, etc.), this is not the case as each pixel will output a different voltage value. This is known as the PRNU and is the gain variation between each pixel across a sensor array. It can be shown that this source of variation can be made worse with a large exposure time and a small gain camera setting [2].
Dark Signal Non-Uniformity (DSNU) or Dark FPN:
In the absence of light, photodiode leakage and sensor read circuitry can manifest itself as an offset in the output of each pixel. This variation of such is known as the DSNU.
Other sources of characterized noise that play a critical role in the performance of the CMOS image sensor is the Dark Current Rate and the Read Noise [3]. Referring the Fig. 1, the Dark Current Rate is the mean gain between the pixels without an illuminant caused by leakage current inherent to the electronics that is indistinguishable from external photons. The largest culprit of such is the reverse-biased diode in the standard 4T-pixel array. This leakage is mostly process dependent and is attributed to interface and material defects in the semiconductor. Read Noise, however, is largely caused from the sampling of the pinned-photodiode in the 4T-pixel and the resulting quantization of the output voltage from the Analog-to-Digital converter at the output of the readout circuitry.
Redadout Noise
Referring the Fig. 2, one can see the typical model of the Readout Circuit chain. Starting from the left, we can see the model of the 4T-Pixel cell. It consists of the photo-element, a sampling capacitor, and a readout source follower. Common sources of noise in this cell consist of the Shot Noise from the pinned-photodiode, Thermal and Flicker Noise from the MOS Source Follower, and gain mismatch in the Source Follower stage dependent on the Threshold Voltage of the MOS device. The next cell, the Programmable Gain Amplifier (PGA) introduces noise in the form of Thermal and Flicker Noise, and much like the Source Follower stage, gain variation dependent on device mismatch. The Sample-and-Hold cell that follows contributes kTC noise largely mitigated through the use of a technique known as Correlated Double Sampling. Lastly, we find ourselves at the Analog-Digital Converter, which offer its own noise source in the form of Quantization Noise and variations in its performance (INL/DNL) due to device mismatch.
In a CMOS sensor, each of these pixels and its corresponding readout electronics are laid out in a grid. In a typical sensor, each pixel has its own amplifier with associated gain and offset noise. Assuming an operational amplifier model, the distribution of this error should approach a Gaussian across all of the pixels. Additionally, each column of pixels is converted from the analog digital domains (via ADCs), which can also have its own amplifier. A typical pixel array is shown in Fig. 3 [INSERT REF - https://andor.oxinst.com/learning/view/article/understanding-read-noise-in-scmos-cameras]. These readout electronics cause an additional gain/offset error due to the amplifiers and quantization error introduced by the ADCs that vary on a column-by-column basis. Such noise also approaches a Gaussian distribution. Therefore, the sum of all of these readout electronics introduce noise both on a pixel and a column basis.
IMX363 Noise Characterization
To appropriately characterize the noise performance of the IMX363 sensor, various experiments were done to create carefully curated image collections to help characterize the following noise sources: PRNU, DSNU, Dark Current Rate, and the Read Noise. In this section, we will briefly outline how each collection was developed and the methods that were undergone to extract the necessary noise measurement, as well as, the final figures for each.
PRNU and DSNU:
To appropriately characterize the PRNU and DSNU, raw images were taken that range from dark with very short exposures to bright with long exposure times. For each exposure time, five frames were taken. This process was then repeated across six unique ISOSpeed settings. All images were taken using an integrating sphere to make sure that the rays arriving at the camera spanned all angles uniformly. As such, any non-uniformity in the acquired images should be due to the image formation by the lens, an effect discussed below.
To first prepare the data, all the raw images taken with the same ISOSpeed and exposure time, were averaged to reduce the presence of any temporal noise in the capture. Doing so enables us to better estimate the Fixed Pattern noise in the images. As both PRNU and DSNU relate to the non-uniformity in the pixel output across the image, whether in the presence of an illuminant or not, one can estimate the PRNU by looking at the variation in the gain of each pixel and the DSNU by exploring the variation in the offset [4]. The image below plots the mean intensity value across the entire image versus exposure time for each of the unique ISOSpeed settings.
As a first step for measuring both the PRNU and DSNU, a linear fit of the pixel output versus exposure time was taken across the entire image. The resultant DC component, or offset, and the slope were stored in separate matrices. This was done for each separate ISOSpeed setting post-normalization according to the following gain table.
Each of these separate matrices, the offset and gain-slope, were then partitioned according to channel. The IMX363 makes use of the standard Bayer Color Filter, and as such, there exists two separate green channels (Gr and Gb), a red channel (R), and a blue channel (B).
It was apparent after looking at the intensity map of an image that the lens shading was creating a low frequency gradient from the top corner of the sensor to the center. As such, the analysis was done at two separate locations within the image to compare the relative measurements. For each of the locations, a local histogram and intensity map was used to get a reasonable idea of the uniformity between each of the channels. From the histograms below, it can be readily seen that the green channels exhibit the largest gain relative to the blue and red channels. As far as the variance, both the red and green channels appear to have an obviously wider spread than that of red.
- PRNU
-
Fig. 6 : PRNU vs. Sensor Location -
Fig. 7 : Slope Histograms per Channel
To measure the value of the PRNU from these results, the variation of each of the presented regions was taken over the mean. This resulted in a PRNU percentage for each ISOSpeed. For the DSNU, the highlighted methodology was used, however, the offset matrix rather than the slope matrix (from the linear regression of the individual pixel output vs. exposure time) was used. The resultant DSNU was taken as the variation in the offset for each pixel in Digital Number. A conversion to mVRMS was then through ADC transfers function where 1 LSB = 448 uV. The highlighted results of for both the PRNU and DSNU may be seen below.
- DSNU
-
Fig. 8 : DSNU vs. Sensor Location -
Fig. 9 : Offset Histograms per Channel
Dark Current Rate:
For the Dark Current Rate (DCR) measurement, the dataset and data analysis followed the same methodology as the PRNU and DSNU measurements. The only difference here is that all of the images were captured without the presence of a uniform light source and we only consider the slope of the linear regression of each pixel. Doing so allows us to take note of the amount of “photons” captured per second in pure darkness. This “photon” capture is the result of leakage in each of the pixel cells from the reverse biased pinned photodiode. The sensor has no way to determine that this is not actual light, and as such, it is necessary that it is characterized. As mentioned earlier, this leakage is largely process dependent and is highly material dependent. The results are summarized below.
Read Noise: To isolate and measure the read noise of the sensor, all other sources of noise must be normalized, so that we can be sure any deviation in pixel value is due to the electronics, rather than any other source of fixed-pattern or temporal noise.
IMX363 Model and Results
Conclusions
References
[1] “Imager Circuit Noise,” Teach RSS. [Online]. Available: http://scarlet.stanford.edu/teach/index.php/Imager_Circuit_Noise. [Accessed: 17-Nov-2020].
[2] Basler AG, “Sales & Support,” Basler AG, 17-Nov-2020. [Online]. Available: https://www.baslerweb.com/en/sales-support/knowledge-base/frequently-asked-questions/what-is-prnu/14988/. [Accessed: 17-Nov-2020].
[3] “Understanding Read Noise in sCMOS Cameras - Andor Learning Centre,” Oxford Instruments. [Online]. Available: https://andor.oxinst.com/learning/view/article/understanding-read-noise-in-scmos-cameras. [Accessed: 17-Nov-2020].
[4] Harvest Imaging Blog. [Online]. Available: https://harvestimaging.com/blog/?p=916. [Accessed: 17-Nov-2020].
[5] R. J. Baker, CMOS circuit design, layout, and simulation. Hoboken, NJ: IEEE Press, 2019.
[6] J. Farrell and B. Wandell, ISETCAM, (2020), GitHub repository, https://github.com/iset/isetcam/wiki
[7] A simulation tool for evaluating digital camera image quality (2004). J. E. Farrell, F. Xiao, P. Catrysse, B. Wandell . Proc. SPIE vol. 5294, p. 124-131, Image Quality and System Performance, Miyake and Rasmussen (Eds). January 2004
Appendix
The written code for this project may be found at the following GitHub repository :
- IMX363 Noise Characterization : https://github.com/JYurgie/IMX363_Noise_Characterization
Notable Scripts :
- "sensor_model.m" : Builds and simulates a model using the extracted noise parameters.
- "PRNU_DSNU_PixelWise.m" : Reads in the images and calculates the slope and offset matrices for each pixel. PRNU and DSNU result structures are created.
- "DarkCurrentRate_PixelWise.m" : Reads in the images and calculates the slope for each pixel. DCR result structure is created.
- "GenerateDataStructures_**.m" : Will generate structures for all of the datasets (PRNU/DSNU, DCR, ReadNoise) either at the image or pixel level.