Google Pixel 4A Camera Noise Model
Introduction
Noise in image sensors stems from a vast array of non-idealities in the readout circuit chain of a CMOS image sensor. Whether temporal or fixed-pattern noise, the result is a lack of clarity, and ultimately, a degradation in the final resultant image. To help explain and characterize this phenomenon, we will begin by first outlining and explaining the relevant noise sources in the standard readout circuit model. This will serve to give a high-level overview as to where the relevant noise sources originate from. After building such a foundation, we will describe and outline our method of characterizing the noise performance of a well-known CMOS image sensor used in today’s most anticipated smartphones, the IMX363 used in the Google Pixel 4A. The results of such a characterization will then take advantage of the ISETCAM software [6][7] to build a model of the sensor’s performance and highlight preliminary results making broad assumptions about the smartphone’s post-processing pipeline.
Noise Sources and the Readout Circuit Model
Most of the noise inherent to the readout circuit chain of a CMOS image sensor can broadly be described as either temporal or fixed pattern. Temporal, much like its name suggests, is non-deterministic and varies randomly across time. The sources of this type of noise tend to be attributed to random movement of some form of carrier such as the random collection of photons on an image sensor (Photon Noise), thermal agitation of electrons in a semiconductor (Thermal Noise, Flicker Noise), or random carrier movement across any sort of potential barrier (Shot Noise) [1][5].
Fixed-Pattern noise (FPN) is the result of spatial variations in pixel output values whether under uniform illumination or not. The cause of such variation can be attributed to device parameter mismatches across the image sensor itself. Since fixed-pattern noise exists in the presence of an illuminant or not, two metrics for the characterization of the FPN exists as the following:
Photo-Response Non-Uniformity (PRNU) or Light FPN:
As a uniform light source falls on an image sensor, each pixel should output the same output voltage. However, due to variations in the circuits themselves (Threshold Voltage Mismatch, Cell Size Variation, etc.), this is not the case as each pixel will output a different voltage value. This is known as the PRNU and is the gain variation between each pixel across a sensor array. It can be shown that this source of variation can be made worse with a large exposure time and a small gain camera setting [2].
Dark Signal Non-Uniformity (DSNU) or Dark FPN:
In the absence of light, photodiode leakage and sensor read circuitry can manifest itself as an offset in the output of each pixel. This variation of such is known as the DSNU.
Other sources of characterized noise that play a critical role in the performance of the CMOS image sensor is the Dark Current Rate and the Read Noise [3]. Referring the Fig. 1, the Dark Current Rate is the mean gain between the pixels without an illuminant caused by leakage current inherent to the electronics that is indistinguishable from external photons. The largest culprit of such is the reverse-biased diode in the standard 4T-pixel array. This leakage is mostly process dependent and is attributed to interface and material defects in the semiconductor. Read Noise, however, is largely caused from the sampling of the pinned-photodiode in the 4T-pixel and the resulting quantization of the output voltage from the Analog-to-Digital converter at the output of the readout circuitry.
Readout Noise
Referring the Fig. 2, one can see the typical model of the Readout Circuit chain. Starting from the left, we can see the model of the 4T-Pixel cell. It consists of the photo-element, a sampling capacitor, and a readout source follower. Common sources of noise in this cell consist of the Shot Noise from the pinned-photodiode, Thermal and Flicker Noise from the MOS Source Follower, and gain mismatch in the Source Follower stage dependent on the Threshold Voltage of the MOS device. The next cell, the Programmable Gain Amplifier (PGA) introduces noise in the form of Thermal and Flicker Noise, and much like the Source Follower stage, gain variation dependent on device mismatch. The Sample-and-Hold cell that follows contributes kTC noise largely mitigated through the use of a technique known as Correlated Double Sampling. Lastly, we find ourselves at the Analog-Digital Converter, which offer its own noise source in the form of Quantization Noise and variations in its performance (INL/DNL) due to device mismatch.
In a CMOS sensor, each of these pixels and its corresponding readout electronics are laid out in a grid. In a typical sensor, each pixel has its own amplifier with associated gain and offset noise. Assuming an operational amplifier model, the distribution of this error should approach a Gaussian across all of the pixels. Additionally, each column of pixels is converted from the analog digital domains (via ADCs), which can also have its own amplifier. A typical pixel array is shown in Fig. 3 [3]. These readout electronics cause an additional gain/offset error due to the amplifiers and quantization error introduced by the ADCs that vary on a column-by-column basis. Such noise also approaches a Gaussian distribution. Therefore, the sum of all of these readout electronics introduce noise both on a pixel and a column basis.
IMX363 Noise Characterization
To appropriately characterize the noise performance of the IMX363 sensor, various experiments were done to create carefully curated image collections to help characterize the following noise sources: PRNU, DSNU, Dark Current Rate, and the Read Noise. In this section, we will briefly outline how each collection was developed and the methods that were undergone to extract the necessary noise measurement, as well as, the final figures for each.
PRNU and DSNU:
To appropriately characterize the PRNU and DSNU, raw images were taken that range from dark with very short exposures to bright with long exposure times. For each exposure time, five frames were taken. This process was then repeated across six unique ISOSpeed settings. All images were taken using an integrating sphere to make sure that the rays arriving at the camera spanned all angles uniformly. As such, any non-uniformity in the acquired images should be due to the image formation by the lens, an effect discussed below.
To first prepare the data, all the raw images taken with the same ISOSpeed and exposure time, were averaged to reduce the presence of any temporal noise in the capture. Doing so enables us to better estimate the Fixed Pattern noise in the images. As both PRNU and DSNU relate to the non-uniformity in the pixel output across the image, whether in the presence of an illuminant or not, one can estimate the PRNU by looking at the variation in the gain of each pixel and the DSNU by exploring the variation in the offset [4]. The image below plots the mean intensity value across the entire image versus exposure time for each of the unique ISOSpeed settings.
As a first step for measuring both the PRNU and DSNU, a linear fit of the pixel output versus exposure time was taken across the entire image. The resultant DC component, or offset, and the slope were stored in separate matrices. This was done for each separate ISOSpeed setting post-normalization according to the following gain table.
Each of these separate matrices, the offset and gain-slope, were then partitioned according to channel. The IMX363 makes use of the standard Bayer Color Filter, and as such, there exists two separate green channels (Gr and Gb), a red channel (R), and a blue channel (B).
It was apparent after looking at the intensity map of an image that the lens shading was creating a low frequency gradient from the top corner of the sensor to the center. As such, the analysis was done at two separate locations within the image to compare the relative measurements. For each of the locations, a local histogram and intensity map was used to get a reasonable idea of the uniformity between each of the channels. From the histograms below, it can be readily seen that the green channels exhibit the largest gain relative to the blue and red channels. As far as the variance, both the red and green channels appear to have an obviously wider spread than that of blue.
- PRNU
-
Fig. 6 : PRNU vs. Sensor Location -
Fig. 7 : Slope Histograms per Channel
To measure the value of the PRNU from these results, the variation of each of the presented regions was taken over the mean. This resulted in a PRNU percentage for each ISOSpeed. For the DSNU, the highlighted methodology was used, however, the offset matrix rather than the slope matrix (from the linear regression of the individual pixel output vs. exposure time) was used. The resultant DSNU was taken as the variation in the offset for each pixel in Digital Number. A conversion to mVRMS was then through ADC transfers function where 1 LSB = 448 uV. The highlighted results of for both the PRNU and DSNU may be seen below.
- DSNU
-
Fig. 8 : DSNU vs. Sensor Location -
Fig. 9 : Offset Histograms per Channel
Dark Current Rate:
For the Dark Current Rate (DCR) measurement, the dataset and data analysis followed the same methodology as the PRNU and DSNU measurements. The only difference here is that all of the images were captured without the presence of a uniform light source and we only consider the slope of the linear regression of each pixel. Doing so allows us to take note of the amount of “photons” captured per second in pure darkness. This “photon” capture is the result of leakage in each of the pixel cells from the reverse biased pinned photodiode. The sensor has no way to determine that this is not actual light, and as such, it is necessary that it is characterized. As mentioned earlier, this leakage is largely process dependent and is highly material dependent. The results are summarized below. The data visualization makes it clear that the Dark Current Rate is quite non-deterministic and does not seem to follow a clear trend like that of the PRNU and DSNU.
Read Noise:
To isolate and measure the read noise of the sensor, all other sources of noise must be normalized, so that we can be sure any deviation in pixel value is due to the electronics, rather than any other source of fixed-pattern or temporal noise. Therefore, the images in the dataset were taken in the dark (to limit photon arrival and thus shot noise) and with extremely small exposure time (to limit dark current). Also, the average was taken for all of the images for a given ISOSpeed to normalize for any temporal noise. Therefore, any resulting variance in an image's pixel values are be primarily due to the read noise.
The first analysis of the data was just comparing read noise at each ISOSpeed level, as shown in Figure 12. The read noise was calculated by taking the average standard deviation among all pixels in each image of a given ISOSpeed. The initial units of DN were converted to mV by multiplying by the full scale range of the sensor over the number of quantization levels. These initial results show a clear impact of ISOSpeed on the read noise, with higher ISOSpeeds resulting in higher read noise measurements. Although more data at various ISOSpeeds is needed to fully deduce the relationship, it appears to be a strong positive linear relationship. The benefits of greater sensitivity given by using a large ISOSpeed are consequently accompanied by a heavy cost in terms of noise. This relationship can qualitatively be seen in Figure 13 [8].
The next analysis done was on how the read noise varied per color channel, shown in Figure 14. Even though the average pixel value was 64 DN (regardless of color channel), the read noise for the green channels is greater than for the red or blue channels. This was measured by calculating the overall average standard deviation of each of the nonzero color arrays (given a Bayer filter) per ISOSpeed. Though the difference between color channels is small at lower ISOSpeeds, it is quite significant at higher ones. Though speculative, it is possible that the green channels are allowed more lenience in their read noise since they have two channels collecting photons. With twice as many pixels in green channels than in red or blue, green can have greater noise while still maintaining the same SNR.
The final analysis done was how read noise varied due to pixel location. Since the images were taken in the dark with very short exposure time, lens vignetting would not be a significant influence on the sensor image. However, since each column of pixels had its own amplifier and ADC, it would instead be more interesting to see how read noise changed on a column basis. The results of this analysis can be seen in Fig. 15. For each of the ISOSpeeds, the top plot shows the read noise per column (by taking an average standard deviation of each column in the image), and the bottom image shows a histogram of the distributions of column read noise. As expected, each histogram is roughly Gaussian. The trends within this data indicated read noise both on a pixel-by-pixel and column-by-column basis. For example, each ISOSpeed has a peak at a column number of approximately 2300. Since the same architecture is used for each of the ISOSpeed images, this indicates a column-specific read noise. On the other hand, the strong variation in distributions indicate that each pixel also contributes its own read noise.
IMX363 Model and Results
ISETCam's sensorSet() functions were used to create a model of the IMX363 imaging sensor analyzed in this project. Since our noise values were dependent on ISOSpeed, a model was created for each ISOSpeed measured. For each model, the overall values (not separated by color channel or location) for PRNU, DSNU, analog gain, dark current, and read noise were set as dictated by the results listed previously. Additional parameters corresponding to the Pixel 4A mechanical structure were set, including a focal length of 24mm, an f-number of 1.7, a pixel depth of 1.4 um, and a pixel array of 3024x4032.
For the comparison, an image of a uniform white field at an exposure time of 1.96ms was used at the various ISOSpeeds. Only demosaicking was done in the image processing in order to provide the most accurate comparison between the model and actual data.
The sensor model outputs and corresponding actual images are seen in Figures 18, 19, and 20, respectively. The leftmost images display the raw sensor output of the model (in terms of the RGB channels), the middle images display the model images after demosaicking, and the rightmost images display the corresponding images taken in a physical integrating sphere. A subset of the center of the images is displayed (rather than the whole image) to make the noise a bit more apparent. In the actual data, we see the effects of lens vignetting and non-idealities in the integrating sphere (present as a line across the image), which were not included in the model. No color-correction was included in the image processing, so the actual data appears more yellow than the modeled data From a qualitative perspective, the amount of noise in the model seems on par with that of the actual data.
The Bayer array on the left helps in visualizing the noise produced at the pixel level, and how that translates into the final image through the model. When increasing ISOSpeed from 55 to 798, we can see a significant difference in the pixel values in the model, which is represented by a substantial increase in image noise at a higher ISOSpeed.
Conclusions
Through this project, we were able to accomplish our goal of measuring PRNU, DSNU, dark current, and read noise. We compared noise of the overall image to its composite color channels, as well as to different locations of the pixel array to account for lens vignetting and column ADCs, and showed the impact that each of these factors had. Finally, we studied how all of these metrics varied according to ISOSpeed. Using the ISETcam model, we were able to create a representative pixel array, and the corresponding output image, at each ISOSpeed. This showed the significant impact that ISOSpeed has on the resulting image, and how that noise is introduced at the pixel level. For a better comparison between the model and actual datasets, access to the raw sensor output (rather than just the DNG or JPEG), is necessary. With that information, we anticipate that we would be able to isolate errors in our created model and ideally have a better representation of the Pixel 4A camera and sensor.
References
[1] “Imager Circuit Noise,” Teach RSS. [Online]. Available: http://scarlet.stanford.edu/teach/index.php/Imager_Circuit_Noise. [Accessed: 17-Nov-2020].
[2] Basler AG, “Sales & Support,” Basler AG, 17-Nov-2020. [Online]. Available: https://www.baslerweb.com/en/sales-support/knowledge-base/frequently-asked-questions/what-is-prnu/14988/. [Accessed: 17-Nov-2020].
[3] “Understanding Read Noise in sCMOS Cameras - Andor Learning Centre,” Oxford Instruments. [Online]. Available: https://andor.oxinst.com/learning/view/article/understanding-read-noise-in-scmos-cameras. [Accessed: 17-Nov-2020].
[4] Harvest Imaging Blog. [Online]. Available: https://harvestimaging.com/blog/?p=916. [Accessed: 17-Nov-2020].
[5] R. J. Baker, CMOS circuit design, layout, and simulation. Hoboken, NJ: IEEE Press, 2019.
[6] J. Farrell and B. Wandell, ISETCAM, (2020), GitHub repository, https://github.com/iset/isetcam/wiki
[7] A simulation tool for evaluating digital camera image quality (2004). J. E. Farrell, F. Xiao, P. Catrysse, B. Wandell . Proc. SPIE vol. 5294, p. 124-131, Image Quality and System Performance, Miyake and Rasmussen (Eds). January 2004
[8] "What is ISO? Your Camera's Sensitivity to Light Explained," [Online]. Available: http://davemorrisonphotos.com/photo-tutorials/iso-sensitivity-light.
Appendix
The written code for this project may be found at the following GitHub repository :
- IMX363 Noise Characterization : https://github.com/JYurgie/IMX363_Noise_Characterization
Notable Scripts :
- "sensor_model.m" : Builds and simulates a model using the extracted noise parameters.
- "PRNU_DSNU_PixelWise.m" : Reads in the images and calculates the slope and offset matrices for each pixel. PRNU and DSNU result structures are created.
- "DarkCurrentRate_PixelWise.m" : Reads in the images and calculates the slope for each pixel. DCR result structure is created.
- "GenerateDataStructures_**.m" : Will generate structures for all of the datasets (PRNU/DSNU, DCR, ReadNoise) either at the image or pixel level.
- "ReadNoise_Analysis.m" : Reads in ReadNoise output data structure and computes/plots results.
Work Breakdown :
Joey: Created scripts for generating structures from the datasets. Did analysis for PRNU/DSNU and dark current.
Melissa: Did analysis for read noise. Created script for the sensor model.