Evaluation Pipeline with GenAI-Assisted Algorithm Development for Virtual Image Denoising and Pixel-Defect Correction

Introduction

Motivation for an Evaluation Pipeline for Image Processing

AI-generated image-processing scripts vary widely in quality, making it difficult to determine which versions are reliable for real-world applications. As large language models (LLMs) become increasingly integrated into algorithm development workflows, the need for systematic evaluation becomes increasingly critical. Different prompts or model versions can produce inconsistent algorithm logic, resulting in reproducibility challenges that undermine confidence in AI-assisted development [1].

Standardized benchmarking improves the efficiency of comparing LLM-generated algorithms across various tasks, including denoising, pixel-defect correction, region of interest (ROI) reconstruction, and enhancement. Without a robust evaluation framework, researchers and engineers must rely on slow, manual inspection to validate algorithmic variants, which is a process that significantly extends development cycles and introduces subjective bias.

To measure true performance across diverse conditions, a robust evaluation pipeline must account for a variety of scenes, defect patterns, noise levels, and lighting conditions. This systematic approach accelerates development cycles by automatically validating and ranking algorithm variants, enabling data-driven decisions to determine which implementations merit further refinement or deployment.

Advantages of Generative Artificial Intelligence for Algorithm Development in Image Processing

Generative Artificial Intelligence (GenAI) may be particularly advantageous for developing algorithms to handle images with challenging noise conditions and complex patterns, as well as proposing context-aware methods to reconstruct ROI impacted by defective pixels. GenAI-assisted scientific programming using LLM can expedite the development of denoising and defect-correction image processing pipelines. The development of sophisticated algorithms in traditional image processing pipelines requires extensive denoising, calibration, and defect correction capabilities; a variety of factors, such as noise model selection, tuning and filtering parameters, and validation using image quality metrics, must be accounted for. Multiple algorithmic variants can be developed and tested in parallel to speed up the development phase and vet which models are most promising for the desired image processing application [2].

With appropriate prompts, LLM-aided code generation can facilitate sensor characterization by testing various denoising assumptions, simulating images impacted by different forms of defect pixels for a more diverse and larger sample size for testing, as well as executing extensive parameter sweeps to evaluate their influence on image quality metrics. The relatively widespread access to GenAI tools, such as ChatGPT, would enable a broad audience to conveniently use available LLM resources for improving image quality by the GenAI-assisted development of sophisticated image processing algorithms [3].

Applications that Benefit from a Reliable Evaluation Pipeline

In recent years, the photography and imaging industry has undergone a rapid transformation driven by the integration of artificial intelligence (AI) into both camera hardware and post-processing workflows. No longer limited to traditional image-signal-processing (ISP) pipelines or manual editing in desktop software, modern camera systems increasingly leverage neural networks, on-device NPUs, and deep-learning algorithms to enhance image quality, reduce noise, stabilize scenes, and even reconstruct detail; this is often executed in real time or shortly after capture. Camera makers, including Nikon, Canon, and Sony, all utilize AI autofocus systems in their latest mirrorless cameras, with features such as face and eye detection, sports autofocus, subject detection, and scene recognition [4].

The development of a robust evaluation pipeline serves multiple domains where image quality is critical. In consumer photography, reliable algorithms ensure consistent enhancement across diverse shooting conditions. Scientific imaging applications, ranging from microscopy to astronomical observation, require validated processing methods in which accuracy is critical for drawing research conclusions. Moreover, computer vision systems depend on high-quality input images for tasks such as object detection, segmentation, and scene understanding. As AI reshapes the entire imaging pipeline, from sensor readout to final edits, the ability for systematic evaluation and comparison of image processing algorithms is crucial for enabling practitioners to select methods appropriate for their specific requirements, as well as account for balancing factors such as processing speed, accuracy, and robustness to various degradation types.

Background

Significance of Key Scene and Camera Parameters on Image Quality

Luminance

Luminance corresponds to the area light scattered from an extended source in the oriented surface’s direction; it can be used to measure the surface brightness from a specific viewpoint as it appears to the human eye. Luminance relates to the number of photons entering the camera and can directly influence the signal-to-noise ratio (SNR) and therefore impact image metrics, such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) [5].

Exposure Control

Exposure settings must be set such that electron generation in the dark regions of an image will be sufficiently above the sensor’s noise floor, and the bright regions in the image will not exceed the full well capacity of the pixel. This is also crucial for ensuring that the dynamic range of the scene can fit within the dynamic range of the sensor. Therefore, optimizing exposure settings is crucial to minimize excessive noise or saturation. Both lens aperture and exposure duration are primary factors that influence exposure control. Very short exposure times risk insufficient collection of electrons in these dark regions, which can be beneath the noise level. A long exposure duration allows more time for photon collection and conversion to electrons. However, very long exposure times risk saturating the floating diffusion node, in which the scene intensities are recorded at the same maximum value. This occurs when electron generation in the bright region’s pixels is greater than the storage capability of the pixel. The exposure value depicted below defies how the aperture size and exposure time influence the image brightness. This relationship reveals that a shorter exposure time or a narrower aperture reduces the light reaching the sensor. Moreover, increasing the exposure value is necessary to prevent overexposure when the scene brightness increases [6].

$EV={\log }_2(\frac{T}{F^2})=2{\log }_2(F)-{\log }_2(T)$ [6]

where EV = exposure value, F = F-number, T = exposure time in seconds

F-number (f/#)

The F-number, as defined below, is the ratio of the focal length to the diameter of the aperture. It is a crucial parameter for thin lenses. A larger F-number for a given focal length corresponds to a smaller aperture diameter but increased depth of field; this would further limit the light that arrives at the sensor and be more resemblant of a pinhole camera. By comparison, a wider aperture enables the sensor to capture more photons [7].

$\text{F-number}=f/\mathrm{\#}=\frac{\text{focal length}}{\text{aperture diameter}}$

Fill factor

The fill factor, ranging from 0 to 1, corresponds to the portion of the photosensitive pixel area. This parameter sets the minimum possible noise level and constrains the SNR in low-light conditions. A higher fill factor increases the light sensitivity, since more of the pixel area is used for light detection. The architecture of stacked sensors provides separation of the photodiode layer from the readout and processing circuitry; this not only maximizes the fill factor for each pixel but also allows for advanced, lower-noise circuitry to be integrated into the sensor without the tradeoff in pixel size [6].

Significance of Photon and Read Noise on Image Quality

Photon noise, inherent to light measurements, is considered a dominant source of image noise in most conditions, except in low-light scenes where read noise dominates. Photon noise is also considered the least influenced by camera hardware technology improvement. The arrival of photons at each of the sensor's pixels and photon noise can be described by the Poisson distribution, in which the mean equals the variance, since within each small time interval, the probability of photon arrival is constant. When images are captured at a fixed exposure duration and aperture size, the number of photons collected by the sensor will be directly proportional to the ambient lighting level. Therefore, both the photon noise and SNR will increase proportionally to the square root of the ambient lighting level. At dim illumination, fewer photons are captured by the photodiode. The inherent Poisson noise indicates that the collection of fewer electrons will also result in a lower SNR. Although the absolute noise can be lower in the case of dim illumination, the SNR can worsen when certain pixels collect fewer electrons than the noise level, resulting in a noisy image [6], [8].

Read noise is independent of illumination and is the random electronic noise that arises during the readout process when the pixel’s stored charge is converted to a digital value. It can become the dominant source of noise that constrains image quality, especially in low scene illumination, in which fewer photons are captured by the sensor. There is a negative correlation between read noise and dynamic range, in which lower read noise will decrease the noise floor and correspond to an increased dynamic range [9].

Significance of Pixel Defects on Image Quality

Pixel defects, commonly categorized as dead, hot, or stuck pixels, pose a significant challenge to modern image sensor performance because they introduce localized impulse noise that disrupts both pixel-level accuracy and global perceptual quality. They appear as bright, distinctive dots on the image, and even at low defect densities, these anomalies can degrade key metrics such as PSNR, SSIM, and MTF by introducing intensity discontinuities that propagate through demosaicing, denoising, and compression pipelines; this ultimately leads to a reduction in edge fidelity and scene detail. Defective pixels severely impact sensors regularly exposed to high levels of light, electrical energy, or radiation, leading to high rates of pixel corruption [10]. As pixel sizes continue to shrink in advanced CMOS sensors, the relative influence of individual defective sites increases, making robust defect detection and correction essential for maintaining image quality in consumer and scientific imaging systems.

Evaluation of Image Quality Metrics: PSNR, SSIM, and MTF50

The PSNR is derived from the mean square error (MSE) and measures pixel-level error. It corresponds to the ratio between the maximum pixel intensity and the power of distortion. A higher PSNR value corresponds to higher image quality. As the MSE approaches zero, the PSNR value approaches infinity. By comparison, low PSNR values indicate large numerical differences between the reference and test images. The SSIM metric accounts for factors of luminance, contrast, and local image structure. For SSIM, pixel intensity patterns are considered structures after both luminance and contrast are normalized. It provides a local quality score that better aligns with human visual perception than the PSNR metric. Studies have shown that both PSNR and SSIM are particularly sensitive to noise degradation [11], [12].

$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{\sum _{m,n}{[I_1(m,n)-I_2(m,n)]}^2}{MN}$ [13]

$\mathrm{P}\mathrm{S}\mathrm{N}\mathrm{R}=10{\log }_{10}\left(\frac{R^2}{\mathrm{M}\mathrm{S}\mathrm{E}}\right)$ [13]

$\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}(x,y)=\frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)({\sigma }_x^2+{\sigma }_y^2+C_2)}$ [14]

The MTF50 (Modulation Transfer Function at 50% contrast) metric quantifies spatial resolution and sharpness by measuring the spatial frequency at which contrast drops to 50% of its maximum value, as defined in the ISO 12233 standard for electronic still-picture imaging resolution and spatial frequency responses [15].

Methods: Evaluation Pipeline for Image Processing

The evaluation pipeline developed in this project provides a comprehensive framework for systematically assessing image processing algorithms using ISETCam as the supporting package [16]. The pipeline incorporates sweeps of scene and camera parameters, varied noise models, and various pixel defect modes (hot, dead, and mixed) across a range of defect percentages. It generates trend plots for ∆PSNR and ∆SSIM, box plots, and percentage improvement metrics, SNR profiles for each RGB channel, MTF curves, and MTF50 trend plots to quantify spatial resolution performance. The system supports multiple trials per setting for robust statistical analysis and includes visualization capabilities such as SSIM error maps and GIF generation comparing the golden standard, defective, and corrected images. Additionally, the pipeline offers flexibility by supporting both user-defined scenes through file path selection and built-in standard testing charts, enabling evaluation across diverse image content and standardized targets. The two key phases used in establishing the evaluation pipeline in this investigative study are shown in Fig. 1.

Phase 1: Identification and Evaluation of Key Parameters and Challenges. A systematic study using ISETCam was conducted to assess how sweeping parameters listed in Table 1 influence image quality across different noise models and pixel defect types. Assessment metrics included PSNR to compare the rendered image to a noiseless or defectless reference based on MSE as well as SSIM, to evaluate the perceived visual quality and similarity between the images. MTF50 is also evaluated as a function of pixel defect percentage.

Phase 2: Testing Evaluation Pipeline. MATLAB Simulation case studies are performed with ISETCam to evaluate the performance of GenAI assistance in algorithm development for virtual image quality enhancement, with the objective of denoising and pixel defect removal. This image quality improvement would traditionally require a camera hardware upgrade or is inherently limited by environmental conditions during scene capture.

Fig. 1 Block diagram of evaluation pipeline development for virtual image enhancement

Table 1. Parameter sweep intervals for image quality evaluation

Parameter	Units	Sweep Interval [LSL USL]	Purpose
Luminance	cd/m²	[50 5000]	Lighting level (brighter or dimmer)
Exposure Time	seconds	[0.001 0.5]	Controls the number of photons captured by the sensor
F-Number (f/#)	-	[1.4 16]	Related to the optical aperture diameter for a given focal length
Fill factor	-	[0.2 0.8]	Photosensitive fraction of pixel area

Evaluation of the Influence of Parameter Sweep for Varied Noise Models and Pixel Defect Types on Image Quality Metrics of PSNR and SSIM

A MATLAB script was developed to perform a parameter sweep for luminance, exposure time, f/#, and fill factor. Each parameter sweep was conducted independently to analyze its effect on PSNR and SSIM for a Macbeth reflectance chart illuminated by D65 with 600 x 600 pixels. Parameter sweeps at three different noise models were evaluated by setting the noise flag to the following: (1) no noise, (2) photon noise only, and (3) the combination of photon and read noise. This evaluation allowed for a quantitative analysis to determine which imaging parameters have the strongest influence on image quality metrics across the different noise models. Moreover, the same parameter sweep analysis is performed across three pixel defect types (hot, dead, and mixed) at a fixed pixel defect percentage of 0.5%. In addition, MTF50 analysis is conducted to evaluate the impact of pixel defects on the spatial resolution of a slanted edge scene.

Denoising Algorithm Development through GenAI-Assisted Script Generation

Non-Local Means and Smoothing Filter Operations with RGB Channel Processing for Maximizing PSNR and SSIM

Prompts were given to GenAI through ChatGPT to generate MATLAB scripts for image enhancement, denoising images impacted by photon noise only or a combination of photon and read noise across varied parameter settings. The image files had file names that specified the parameter sweep condition and noise model type; the script references the specified folder with the original image files for the denoising process. Several iterations of prompt refinement were necessary to create a more robust denoising script that achieved better image quality. One of the refined prompts updated the algorithm’s range of smoothing values for denoising strength to account for edge preservation and prevention of image blurriness. The final denoising script also adapts the algorithm to each image’s unique conditions (e.g., parameter sweep settings and noise model type) to achieve the highest PSNR and SSIM.

As shown in the flow diagram in Fig. 2, a non-local means (NL-means) filtering was applied in the denoising algorithm with RGB channel processing, such that each pixel is filtered based on weighted averages of patches across the image with similar patterns. To preserve the image fidelity and spatial structure, each color channel is separately denoised to prevent tentative cross-channel color artifacts [17]. Moreover, the smoothing parameter is optimized to maximize the PSNR and SSIM relative to the noiseless reference image to ensure the optimal tradeoff between noise reduction and fine detail preservation. Increasing the degree of smoothing will heighten the aggressiveness of denoising but at the tradeoff of blurred details in the image. By comparison, decreasing the degree of smoothing will allow for better preservation of the edges and fine details but can result in poorer noise removal [18].

The denoising algorithm effectively produces a parameter-aligned, denoised image for each of the swept sensor settings. After the denoising procedure is applied, the script takes in the denoised images for each noise model (photon noise only as well as the combination of photon and read noise) and compares them against the noiseless reference for each swept parameter setting. ΔPSNR and ΔSSIM were used to assess the improvement between the ChatGPT-denoised and original noisy images.

Fig. 2. Optimized algorithmic logic and image quality metric evaluation flow with adaptive denoising of Macbeth reflectance chart for different noise models and parameter settings