Accelerating Denoising at the Speed of Light - Revision history

Samidhm: /* Conclusion and Future Work */

2024-12-13T16:02:44Z

Conclusion and Future Work

← Older revision		Revision as of 16:02, 13 December 2024
Line 158:		Line 158:

	We also study the 'R' values and obtain a strong correlation between the 512 app image and the denoised output. This suggests that such a denoiser could be used for fixed tolerance acceptance levels.		We also study the 'R' values and obtain a strong correlation between the 512 app image and the denoised output. This suggests that such a denoiser could be used for fixed tolerance acceptance levels.
	[[File:~~correlation_dataset~~.png\|600px\|thumb\|center\|Correlation of 512 spp v/s denoised values]]		[[File:correlation_denoised.png\|600px\|thumb\|center\|Correlation of 512 spp v/s denoised values]]

	== References ==		== References ==

Samidhm: /* Conclusion and Future Work */

2024-12-13T16:02:20Z

Conclusion and Future Work

← Older revision		Revision as of 16:02, 13 December 2024
Line 155:		Line 155:

	[[File:table_4.png\|800px\|thumb\|center\|Performance Metrics]]		[[File:table_4.png\|800px\|thumb\|center\|Performance Metrics]]


			We also study the 'R' values and obtain a strong correlation between the 512 app image and the denoised output. This suggests that such a denoiser could be used for fixed tolerance acceptance levels.
			[[File:correlation_dataset.png\|600px\|thumb\|center\|Correlation of 512 spp v/s denoised values]]

	== References ==		== References ==

Samidhm: /* Problem Definition */

2024-12-13T14:18:14Z

Problem Definition

← Older revision		Revision as of 14:18, 13 December 2024
Line 8:		Line 8:

	=== Problem Definition ===		=== Problem Definition ===
	Given a 64x64 image rendered with one sample per pixel, along with other features that can be obtained using similar computational resources, we propose a denoising framework capable of producing a 64x64 output image that closely matches the quality of a ground-truth image rendered with 512 samples per pixel. Our framework is evaluated primarily based on two key criteria:		Given a 64x64 image rendered with one sample per pixel, along with other features that can be obtained using similar computational resources, we propose a denoising framework capable of producing a 64x64 output image that closely matches the quality of a ground-truth image rendered with 512 samples per pixel. Our framework is evaluated primarily based on two key criteria- Quality and Performance.

	==== Quality ====		==== Quality ====

Samidhm: /* Roughness (3 channels) */

2024-12-13T12:58:50Z

Roughness (3 channels)

← Older revision		Revision as of 12:58, 13 December 2024
Line 115:		Line 115:

	==== Roughness (3 channels) ====		==== Roughness (3 channels) ====
	* Encodes surface texture to define how light scatters.		* Encodes surface texture to define how light scatters.
	* Crucial for realistic material rendering in PBR workflows.

	==== Albedo (3 channels) ====		==== Albedo (3 channels) ====

Samidhm: /* Conclusion and Future Work */

2024-12-13T12:58:33Z

Conclusion and Future Work

← Older revision		Revision as of 12:58, 13 December 2024
Line 146:		Line 146:

	== Conclusion and Future Work ==		== Conclusion and Future Work ==
	In summary, we trained a 1 layer UNet that successfully denoises an input image that was ray traced at 1 spp. This model takes the input image along with three features (albedo, roughness and normal) and produces a denoised output as shown in the two figures below. This model has been reduced to half precision and we used torch.compile() to perform code optimizations. The model performance characteristics are summarized in the table below. All inference latencies were measured on a batch of size 1260. The next steps would consist of exploring quantizing the model to int8 using the TensorRT library (PyTorch quantization library does not support GPU backend). We expect that this would achieve good performance due to the availability of fast integer matrix units. We observed that the denoised images from our model tend to have a blurry effect when compared to the ground truth. For this we intend on incorporating supersampled auxillary features as inputs to our model to improve the output PSNR of the model. Finally, an interesting direction would be to analyze the ~~sparsity characteristics~~ of the ~~model weights~~ to ~~exploit any structured/unstructured sparsity that~~ could be ~~present~~.		In summary, we trained a 1 layer UNet that successfully denoises an input image that was ray traced at 1 spp. This model takes the input image along with three features (albedo, roughness and normal) and produces a denoised output as shown in the two figures below. This model has been reduced to half precision and we used torch.compile() to perform code optimizations. The model performance characteristics are summarized in the table below. All inference latencies were measured on a batch of size 1260. The next steps would consist of exploring quantizing the model to int8 using the TensorRT library (PyTorch quantization library does not support GPU backend). We expect that this would achieve good performance due to the availability of fast integer matrix units. We observed that the denoised images from our model tend to have a blurry effect when compared to the ground truth. For this we intend on incorporating supersampled auxillary features as inputs to our model to improve the output PSNR of the model. Finally, an interesting direction would be to understand metrics other than PSNR to analyze the denoising construct. A study of the tolerance of whether an image is good to be considered as a true representation could also be done based on the application we plan to use the denoiser for. This study will greatly depend on the application if it is gaming, interactive simulations or robotic training data.

Samidhm at 12:55, 13 December 2024

2024-12-13T12:55:17Z

← Older revision		Revision as of 12:55, 13 December 2024
Line 132:		Line 132:
	==== Depth (1 channel) ====		==== Depth (1 channel) ====
	* Encodes distance from the camera to the surface in grayscale.		* Encodes distance from the camera to the surface in grayscale.
	* Key for spatial positioning and 3D scene reconstruction.

Samidhm: /* Feature ablation study */

2024-12-13T12:53:43Z

Feature ablation study

@@ Line 109: / Line 109: @@
 === Feature ablation study ===
 In this study, we attempt to understand the impact of each input feature in an effort to identify the minimum viable subset of features that would help attain a high PSNR while limiting the model's size, thereby improving inference latency. The set of available features includes the original image (3 channels), roughness (3 channels), albedo (3 channels), normal (3 channels), relative normal (3 channels), and depth (1 channels).
 The original input image is a requirement, therefore we run experiments by removing every other features one at a time. Furthermore, we also assess the impact of any normal related features, by removing both the normal and relative normal.
 Our findings establish the following feature importance order: albedo >> roughness > normal > relative normal > depth. Furthermore, we conclude that removing both forms of normal does negatively impact the model's performance.

Samidhm: /* Model Architecture */

2024-12-13T12:42:03Z

Model Architecture

← Older revision		Revision as of 12:42, 13 December 2024
Line 36:		Line 36:
	=== Model Architecture ===		=== Model Architecture ===

	We chose a simple UNet architecture [4] that consists of n encoder, n decoder and one bottleneck layers as shown in the ~~'Simple UNet~~ architecture' figure. Each encoder consists two sets of 2D convolutional layer, batch normalization and a ReLU activation stacked together. The bottleneck layer is chosen to be the same as an encoder layer. However, we can use other layers such as RNN, LSTM, etc as the bottleneck layer based on the application requirements. A decoder layer consists of two parts, a 2D convolutional transpose layer and a convolutional layer. The output of the convolutional transpose layer is concatenated with the corresponding output from encoder (skip connection) and passed into the convolutional layer.		We chose a simple UNet architecture [4] that consists of n encoder, n decoder and one bottleneck layers as shown in the architecture figure below. Each encoder consists two sets of 2D convolutional layer, batch normalization and a ReLU activation stacked together. The bottleneck layer is chosen to be the same as an encoder layer. However, we can use other layers such as RNN, LSTM, etc as the bottleneck layer based on the application requirements. A decoder layer consists of two parts, a 2D convolutional transpose layer and a convolutional layer. The output of the convolutional transpose layer is concatenated with the corresponding output from encoder (skip connection) and passed into the convolutional layer.

	[[File:unet_example.png\|600px\|thumb\|center\|Simple UNet architecture]]		[[File:unet_example.png\|600px\|thumb\|center\|Simple UNet architecture]]

Samidhm: /* Loss function and training schedule study */

2024-12-13T08:27:32Z

Loss function and training schedule study

← Older revision		Revision as of 08:27, 13 December 2024
Line 88:		Line 88:

	=== Loss function and training schedule study ===		=== Loss function and training schedule study ===
	In the first phase, the L1 loss is weighted by the parameter $\alpha$ and a corresponding $1 - \alpha$ is used to weight the HFEN loss. We spaced alpha in increments of 0.25 and observed that $\alpha = 1$ yields the best result. Furthermore, $\alpha = 0$ performs significantly worse than other runs. This implies that the HFEN loss does not significantly improve the model's performance, but L1 plays a crucial role.		In the first phase, the L1 loss is weighted by the parameter alpha and a corresponding (1 - alpha) is used to weight the HFEN loss. We spaced alpha in increments of 0.25 and observed that alpha = 1 yields the best result. Furthermore, alpha = 0 performs significantly worse than other runs. This implies that the HFEN loss does not significantly improve the model's performance, but L1 plays a crucial role.

	Next, we investigate the role of the 2nd phase of training. The L2 loss function is sensitive to outliers due to the squaring of errors, making it suitable for applications where minimizing large errors is crucial [6]. However due to this sensitivity, it is necessary to use L2 loss only once the model is in a relatively stable position in the optimization landscape, hence the model is first trained with L1 loss.		Next, we investigate the role of the 2nd phase of training. The L2 loss function is sensitive to outliers due to the squaring of errors, making it suitable for applications where minimizing large errors is crucial [6]. However due to this sensitivity, it is necessary to use L2 loss only once the model is in a relatively stable position in the optimization landscape, hence the model is first trained with L1 loss.

Samidhm: /* Model Architecture */

2024-12-13T08:26:57Z

Model Architecture

← Older revision		Revision as of 08:26, 13 December 2024
Line 36:		Line 36:
	=== Model Architecture ===		=== Model Architecture ===

	We chose a simple UNet architecture [4] that consists of $n$ encoder, $n$ decoder and one bottleneck layers as shown in the 'Simple UNet architecture' figure. Each encoder consists two sets of 2D convolutional layer, batch normalization and a ReLU activation stacked together. The bottleneck layer is chosen to be the same as an encoder layer. However, we can use other layers such as RNN, LSTM, etc as the bottleneck layer based on the application requirements. A decoder layer consists of two parts, a 2D convolutional transpose layer and a convolutional layer. The output of the convolutional transpose layer is concatenated with the corresponding output from encoder (skip connection) and passed into the convolutional layer.		We chose a simple UNet architecture [4] that consists of n encoder, n decoder and one bottleneck layers as shown in the 'Simple UNet architecture' figure. Each encoder consists two sets of 2D convolutional layer, batch normalization and a ReLU activation stacked together. The bottleneck layer is chosen to be the same as an encoder layer. However, we can use other layers such as RNN, LSTM, etc as the bottleneck layer based on the application requirements. A decoder layer consists of two parts, a 2D convolutional transpose layer and a convolutional layer. The output of the convolutional transpose layer is concatenated with the corresponding output from encoder (skip connection) and passed into the convolutional layer.

	[[File:unet_example.png\|600px\|thumb\|center\|Simple UNet architecture]]		[[File:unet_example.png\|600px\|thumb\|center\|Simple UNet architecture]]