David Gal - Revision history

imported>Dgal1: /* L3 Motivation and Background */

2011-03-21T02:13:29Z

L3 Motivation and Background

← Older revision		Revision as of 02:13, 21 March 2011
Line 19:		Line 19:
	L3 stands for local, linear, and learning. Indeed the L3 algorithm is all of these. There are two fundamental components to the L3 algorithm – a regression stage that builds libraries of filters and an application stage that utilizes these filters to perform image processing.		L3 stands for local, linear, and learning. Indeed the L3 algorithm is all of these. There are two fundamental components to the L3 algorithm – a regression stage that builds libraries of filters and an application stage that utilizes these filters to perform image processing.

	The regression stage involves taking several images with any CFA pattern at different illuminant levels. L3 categorizes 9 x 9 sections of an image known as patches into two fundamental types – flat patches and texture patches for smooth and edged surfaces respectively. Knowing what the ideal image should look like a novel regression algorithm creates an ideal filter set for various structural elements within an image. While the filter tap coefficients indeed change for novel CFA designs the mechanics of applying the filters do not. The burden of creating the optimal filter set is left to the regression algorithm, but is largely agnostic to the CFA design.		The regression stage involves taking several images with any CFA pattern at different illuminant levels. L3 categorizes 9 x 9 sections of an image known as patches into two fundamental types – flat patches and texture patches for smooth and edged surfaces respectively. Knowing what the ideal image should look like a novel regression algorithm creates an ideal filter set for various structural elements within an image. While the filter tap coefficients indeed change for novel CFA designs, the mechanics of applying the filters do not. The burden of creating the optimal filter set is left to the regression algorithm, but is largely agnostic to the CFA design.

	For every color element in the CFA pattern a filter library is created. In the case of a Bayer pattern there are four filter libraries – one for red, two for green, and one for blue. Figure 2 shows the contents of a filter library.		For every color element in the CFA pattern a filter library is created. In the case of a Bayer pattern there are four filter libraries – one for red, two for green, and one for blue. Figure 2 shows the contents of a filter library.

imported>Dgal1: /* Background and Motivation */

2011-03-21T02:07:05Z

Background and Motivation

← Older revision		Revision as of 02:07, 21 March 2011
Line 10:		Line 10:
	[[File:Bayer Pattern.jpg \| Figure 1 - Bayer Pattern]]		[[File:Bayer Pattern.jpg \| Figure 1 - Bayer Pattern]]

	The Bayer pattern is, however, not without its limitations and despite ~~it’s~~ wide adoption should by no means be considered the end all and be all of CFA design. Researchers have considered countless designs of CFA patterns in the past. Coupled with the advent of such pixel rich sensors new CFA patterns may well find their way into the consumer market soon. One such considered improvement to CFA design is the addition of a “white” pixel in the array. Such a pixel would filter at near infrared frequencies and provide a true luminance channel. With this additional channel low-light photography could occur with faster shutter speeds thus reducing blur. Additionally it’s possible that higher dynamic range images could easily be captured.		The Bayer pattern is, however, not without its limitations and despite its wide adoption should by no means be considered the end all and be all of CFA design. Researchers have considered countless designs of CFA patterns in the past. Coupled with the advent of such pixel rich sensors new CFA patterns may well find their way into the consumer market soon. One such considered improvement to CFA design is the addition of a “white” pixel in the array. Such a pixel would filter at near infrared frequencies and provide a true luminance channel. With this additional channel low-light photography could occur with faster shutter speeds thus reducing blur. Additionally it’s possible that higher dynamic range images could easily be captured.

	The trouble with novel CFA design is not, however, in the design of the array itself. It is in fact in the design – or rather lack of design – of the image-processing pipeline that transforms sensor data to an image on a screen. As aforementioned countless methods for demosaicing, denoising, and color transformation exist for the Bayer pattern, but for a novel CFA pattern there is no reason to assume such algorithms exist. The necessity of such algorithms can be a major bottleneck in novel CFA research. Without the requisite transformation pipeline there is no way to properly evaluate novel CFA design.		The trouble with novel CFA design is not, however, in the design of the array itself. It is in fact in the design – or rather lack of design – of the image-processing pipeline that transforms sensor data to an image on a screen. As aforementioned countless methods for demosaicing, denoising, and color transformation exist for the Bayer pattern, but for a novel CFA pattern there is no reason to assume such algorithms exist. The necessity of such algorithms can be a major bottleneck in novel CFA research. Without the requisite transformation pipeline there is no way to properly evaluate novel CFA design.

imported>Dgal1: /* Background and Motivation */

2011-03-21T02:02:14Z

Background and Motivation

← Older revision		Revision as of 02:02, 21 March 2011
Line 5:		Line 5:
	Moore’s law has resulted in unprecedented scaling of both the technology and economy of the semi-conductor industry. Among other advancements, semi-conductor scaling has enabled CMOS imaging sensors to reach resolutions of 10s of millions of pixels (megapixels). Traditional research in imaging technology has often focused on improving key metrics of these sensors such as SNR, power consumption, and cost. The abundance of pixels in today’s commercially available imaging sensors has become something of a marketing campaign and competition for leading camera manufacturers. At present both Canon and Nikon offer consumer cameras with more than 20 megapixels. Given a monitor with 72 dots per inch (DPI) such cameras are capable of producing images that at full size occupy in excess of 60” in a single dimension. As a result of what has become “overkill” for the typical consumer researchers have begun to consider other uses of pixels in imaging sensors. One such utilization of the pixels on a sensor is the realization of novel color filter arrays (CFA).		Moore’s law has resulted in unprecedented scaling of both the technology and economy of the semi-conductor industry. Among other advancements, semi-conductor scaling has enabled CMOS imaging sensors to reach resolutions of 10s of millions of pixels (megapixels). Traditional research in imaging technology has often focused on improving key metrics of these sensors such as SNR, power consumption, and cost. The abundance of pixels in today’s commercially available imaging sensors has become something of a marketing campaign and competition for leading camera manufacturers. At present both Canon and Nikon offer consumer cameras with more than 20 megapixels. Given a monitor with 72 dots per inch (DPI) such cameras are capable of producing images that at full size occupy in excess of 60” in a single dimension. As a result of what has become “overkill” for the typical consumer researchers have begun to consider other uses of pixels in imaging sensors. One such utilization of the pixels on a sensor is the realization of novel color filter arrays (CFA).

	The ubiquitous Bayer pattern seen in the figure below has served both the consumer and professional photographic community well. ~~It’s~~ design and consideration for the human visual system – namely high spatial sampling of green wavelengths – has rendered it a universally accepted standard in still camera design. Numerous papers have been published on the Bayer pattern’s merits as well as algorithms for demosaicing, denoising, and color transforming its resultant sensor data.		The ubiquitous Bayer pattern seen in the figure below has served both the consumer and professional photographic community well. Its design and consideration for the human visual system – namely high spatial sampling of green wavelengths – has rendered it a universally accepted standard in still camera design. Numerous papers have been published on the Bayer pattern’s merits as well as algorithms for demosaicing, denoising, and color transforming its resultant sensor data.

	<br>		<br>

imported>Dgal1: /* Background and Motivation */

2011-03-21T02:00:49Z

Background and Motivation

← Older revision		Revision as of 02:00, 21 March 2011
Line 3:		Line 3:

	= Background and Motivation =		= Background and Motivation =
	Moore’s law has resulted in unprecedented scaling of both the technology and economy of the semi-conductor industry. Among other advancements, semi-conductor scaling has enabled CMOS imaging sensors to reach resolutions of 10s of millions of pixels (megapixels). Traditional research in imaging technology has often focused on improving key metrics of these sensors such as SNR, power consumption, and cost. The abundance of pixels in today’s commercially available imaging sensors has become something of a marketing campaign and competition for leading camera ~~manufactures~~. At present both Canon and Nikon offer consumer cameras with more than 20 megapixels. Given a monitor with 72 dots per inch (DPI) such cameras are capable of producing images that at full size occupy in excess of 60” in a single dimension. As a result of what has become “overkill” for the typical consumer researchers have begun to consider other uses of pixels in imaging sensors. One such utilization of the pixels on a sensor is the realization of novel color filter arrays (CFA).		Moore’s law has resulted in unprecedented scaling of both the technology and economy of the semi-conductor industry. Among other advancements, semi-conductor scaling has enabled CMOS imaging sensors to reach resolutions of 10s of millions of pixels (megapixels). Traditional research in imaging technology has often focused on improving key metrics of these sensors such as SNR, power consumption, and cost. The abundance of pixels in today’s commercially available imaging sensors has become something of a marketing campaign and competition for leading camera manufacturers. At present both Canon and Nikon offer consumer cameras with more than 20 megapixels. Given a monitor with 72 dots per inch (DPI) such cameras are capable of producing images that at full size occupy in excess of 60” in a single dimension. As a result of what has become “overkill” for the typical consumer researchers have begun to consider other uses of pixels in imaging sensors. One such utilization of the pixels on a sensor is the realization of novel color filter arrays (CFA).

	The ubiquitous Bayer pattern seen in the figure below has served both the consumer and professional photographic community well. It’s design and consideration for the human visual system – namely high spatial sampling of green wavelengths – has rendered it a universally accepted standard in still camera design. Numerous papers have been published on the Bayer pattern’s merits as well as algorithms for demosaicing, denoising, and color transforming its resultant sensor data.		The ubiquitous Bayer pattern seen in the figure below has served both the consumer and professional photographic community well. It’s design and consideration for the human visual system – namely high spatial sampling of green wavelengths – has rendered it a universally accepted standard in still camera design. Numerous papers have been published on the Bayer pattern’s merits as well as algorithms for demosaicing, denoising, and color transforming its resultant sensor data.

imported>Dgal1: /* Results */

2011-03-21T01:50:50Z

Results

← Older revision		Revision as of 01:50, 21 March 2011
Line 92:		Line 92:

	= Results =		= Results =
	The C++/OpenCV implementation of L3 is a promising first step in realizing a truly data and thread parallel implementation on a GPU. As compared to the Matlab implementation the C++/OpenCV implementation realized roughly a 6x speed up on average. On a macbook pro with a 2.5 GHz Core 2 Duo with 4 GB 667 MHz DDR2 SDRAM the average Matlab execution of a single L3 image was found to be 11.59 seconds compared to an average 2.06 seconds using the C++/OpenCV implementation. Much of ~~this~~ time was utilized for memory allocation and temporary storage of a patch for the purposes of flipping and transposing. The serialization of such a memory bottleneck would be greatly reduced with the facilitation of a GPU architecture.		The C++/OpenCV implementation of L3 is a promising first step in realizing a truly data and thread parallel implementation on a GPU. As compared to the Matlab implementation the C++/OpenCV implementation realized roughly a 6x speed up on average. All images processed in this project were 401 x 481 pixels in size. On a macbook pro with a 2.5 GHz Core 2 Duo with 4 GB 667 MHz DDR2 SDRAM the average Matlab execution of a single L3 image was found to be 11.59 seconds compared to an average 2.06 seconds using the C++/OpenCV implementation. Much of the execution time spent for the C++/OpenCV implementation was utilized for memory allocation and temporary storage of a patch for the purposes of flipping and transposing. The serialization of such a memory bottleneck would be greatly reduced with the facilitation of a GPU architecture.

	Figure 5 shows an output image from the C++/OpenCV implementation.		Figure 5 shows an output image from the C++/OpenCV implementation.

imported>Dgal1: /* L3 Motivation and Background */

2011-03-21T01:44:11Z

L3 Motivation and Background

← Older revision		Revision as of 01:44, 21 March 2011
Line 19:		Line 19:
	L3 stands for local, linear, and learning. Indeed the L3 algorithm is all of these. There are two fundamental components to the L3 algorithm – a regression stage that builds libraries of filters and an application stage that utilizes these filters to perform image processing.		L3 stands for local, linear, and learning. Indeed the L3 algorithm is all of these. There are two fundamental components to the L3 algorithm – a regression stage that builds libraries of filters and an application stage that utilizes these filters to perform image processing.

	The regression stage involves taking several images with any CFA pattern at different illuminant levels. L3 categorizes 9 x 9 sections of an image known as patches into two fundamental types – flat patches and texture patches for smooth and edged surfaces respectively. Knowing what the ideal image should look like a novel regression algorithm creates an ideal filter set for various structural elements within an image. For every color element in the CFA pattern a filter library is created. In the case of a Bayer pattern there are four filter libraries – one for red, two for green, and one for blue. Figure 2 shows the contents of a filter library.		The regression stage involves taking several images with any CFA pattern at different illuminant levels. L3 categorizes 9 x 9 sections of an image known as patches into two fundamental types – flat patches and texture patches for smooth and edged surfaces respectively. Knowing what the ideal image should look like a novel regression algorithm creates an ideal filter set for various structural elements within an image. While the filter tap coefficients indeed change for novel CFA designs the mechanics of applying the filters do not. The burden of creating the optimal filter set is left to the regression algorithm, but is largely agnostic to the CFA design.

			For every color element in the CFA pattern a filter library is created. In the case of a Bayer pattern there are four filter libraries – one for red, two for green, and one for blue. Figure 2 shows the contents of a filter library.

	<br>		<br>
	[[File:FilterLib.jpg \| Figure 2 - Filter Library for a Red Pixel]]		[[File:FilterLib.jpg \| Figure 2 - Filter Library for a Red Pixel]]

	As seen there are several different filters in a single library. All filters are applied using the standard correlation method (convolution without kernel flipping) as seen in ~~equation N~~		As seen there are several different filters in a single library. All filters are applied using the standard correlation method (convolution without kernel flipping) as seen in Equation 1:

			:<math> dst(x,y) = \sum_{-4<=x'<=4, -4<=y'<=4} kernel(x',y')\times patch(x+x',y+y')</math> Equation 1

	~~:<math> dst(x,y) = \sum_{-4<=x'<=4, -4<=y'<=4} kernel(x',y')\times patch(x+x',y+y')</math>~~		As illustrated by equation 1 for each filter application there are a total of 81 multiply accumulate operations. In this project all data is assumed to be IEEE 754 floating point.
	~~PUT IN EQUATION HERE~~

	The final filters – the flat and texture filters – must be applied to a patch that is rotated and transformed to be in a canonical layout with the darkest area of the patch in the upper left corner. Patches may be flipped horizontally, vertically, translated across the major diagonal, or some combination of these transformations.		The final filters – the flat and texture filters – must be applied to a patch that is rotated and transformed to be in a canonical layout with the darkest area of the patch in the upper left corner. Patches may be flipped horizontally, vertically, translated across the major diagonal, or some combination of these transformations.

	The three Weiner filters known as Global CFA filters are utilized to give a rough estimate of the patch type – either texture or flat. If a patch is determined flat then the three flat filters give estimates for X, Y, and Z. If a patch is determined to be textured then N PCA filters are utilized to determine the texture type. In the case of this project there are 16 texture types. The PCA filters are organized in a binary tree fashion such that only 4 PCA filters need to be applied to determine the ultimate texture type. Finally the appropriate texture filters can be applied to give estimates in X, Y, Z color space. In its current iteration the L3 pipeline is implemented entirely in Matlab.		The three Weiner filters known as Global CFA filters are utilized to give a rough estimate of the patch type – either texture or flat. A patch is determined to be flat if the result of the average of the difference between a patch and its global CFA results is greater than a threshold. If a patch is determined flat then the three flat filters give estimates for X, Y, and Z. If a patch is determined to be textured then N PCA filters are utilized to determine the texture type. In the case of this project there are 16 texture types which means that there are 15 PCA filters. The PCA filters are organized in a binary tree fashion such that only 4 PCA filters need to be applied to determine the ultimate texture type (one for each level in the PCA tree). For each application of a PCA filter there is a corresponding threshold which indicates which next node PCA filter should be applied. In this way the binary tree of PCA filters is walked until the final texture type is known. Finally the appropriate texture filters can be applied to give estimates in X, Y, Z color space. In its current iteration the L3 pipeline is implemented entirely in Matlab.

	= Algorithm Analysis =		= Algorithm Analysis =

imported>Dgal1: /* Background and Motivation */

2011-03-21T01:33:02Z

Background and Motivation

← Older revision		Revision as of 01:33, 21 March 2011
Line 3:		Line 3:

	= Background and Motivation =		= Background and Motivation =
	Moore’s law has resulted in unprecedented scaling of both the technology and economy of the semi-conductor industry. Among other advancements, semi-conductor scaling has enabled CMOS imaging sensors to reach resolutions of 10s of millions of pixels (megapixels). Traditional research in imaging technology has often focused on improving key metrics of these sensors such as SNR, ~~sensitivity~~, and cost. The abundance of pixels in today’s commercially available imaging sensors has become something of a marketing campaign and competition for leading camera manufactures. At present both Canon and Nikon offer consumer cameras with more than 20 ~~million pixels~~. Given a monitor with 72 dots per inch (DPI) such cameras are capable of producing images that at full size occupy in excess of 60” in a single dimension.		Moore’s law has resulted in unprecedented scaling of both the technology and economy of the semi-conductor industry. Among other advancements, semi-conductor scaling has enabled CMOS imaging sensors to reach resolutions of 10s of millions of pixels (megapixels). Traditional research in imaging technology has often focused on improving key metrics of these sensors such as SNR, power consumption, and cost. The abundance of pixels in today’s commercially available imaging sensors has become something of a marketing campaign and competition for leading camera manufactures. At present both Canon and Nikon offer consumer cameras with more than 20 megapixels. Given a monitor with 72 dots per inch (DPI) such cameras are capable of producing images that at full size occupy in excess of 60” in a single dimension. As a result of what has become “overkill” for the typical consumer researchers have begun to consider other uses of pixels in imaging sensors. One such utilization of the pixels on a sensor is the realization of novel color filter arrays (CFA).
	As a result of what has become “overkill” for the typical consumer researchers have begun to consider other uses of pixels in imaging sensors. One such utilization of the pixels on a sensor is the realization of novel color filter arrays (CFA).

	The ubiquitous Bayer pattern seen in the figure below has served both the consumer and professional photographic community well. It’s design and consideration for the human visual system – namely high sampling of green wavelengths – has rendered it a universally accepted standard in still camera design. Numerous papers have been published on the Bayer pattern’s merits as well as algorithms for demosaicing, ~~denoiseing~~, and color transforming its resultant sensor data.		The ubiquitous Bayer pattern seen in the figure below has served both the consumer and professional photographic community well. It’s design and consideration for the human visual system – namely high spatial sampling of green wavelengths – has rendered it a universally accepted standard in still camera design. Numerous papers have been published on the Bayer pattern’s merits as well as algorithms for demosaicing, denoising, and color transforming its resultant sensor data.

	<br>		<br>
	[[File:Bayer Pattern.jpg \| Figure 1 - Bayer Pattern]]		[[File:Bayer Pattern.jpg \| Figure 1 - Bayer Pattern]]

	The Bayer pattern is, however, not without its limitations and despite it’s ~~universal~~ adoption should by no means be considered the end all and be all of CFA design. Researchers have considered countless designs of CFA patterns in the past~~, but~~ with the advent of such pixel rich sensors ~~they~~ may well find their way into the consumer market. One such considered improvement to CFA design is the addition of a “white” pixel in the array. Such a pixel would filter at near infrared frequencies and provide a true luminance channel. With this additional channel low-light photography could occur with faster shutter speeds thus reducing blur. Additionally it’s possible that higher dynamic range images could easily be captured.		The Bayer pattern is, however, not without its limitations and despite it’s wide adoption should by no means be considered the end all and be all of CFA design. Researchers have considered countless designs of CFA patterns in the past. Coupled with the advent of such pixel rich sensors new CFA patterns may well find their way into the consumer market soon. One such considered improvement to CFA design is the addition of a “white” pixel in the array. Such a pixel would filter at near infrared frequencies and provide a true luminance channel. With this additional channel low-light photography could occur with faster shutter speeds thus reducing blur. Additionally it’s possible that higher dynamic range images could easily be captured.

	The trouble with novel CFA design is not, however, in the design of the array itself. It is in fact in the design – or rather lack of design – of the image-processing pipeline that transforms sensor data to an image on a screen. As aforementioned countless methods for demosaicing, denoising, and color transformation exist for the Bayer pattern, but for a novel CFA pattern there is no reason to assume such algorithms exist. The necessity of such algorithms can be a major bottleneck in novel CFA research.		The trouble with novel CFA design is not, however, in the design of the array itself. It is in fact in the design – or rather lack of design – of the image-processing pipeline that transforms sensor data to an image on a screen. As aforementioned countless methods for demosaicing, denoising, and color transformation exist for the Bayer pattern, but for a novel CFA pattern there is no reason to assume such algorithms exist. The necessity of such algorithms can be a major bottleneck in novel CFA research. Without the requisite transformation pipeline there is no way to properly evaluate novel CFA design.

	= L3 Motivation and Background =		= L3 Motivation and Background =

imported>Dgal1 at 09:52, 18 March 2011

2011-03-18T09:52:12Z

← Older revision		Revision as of 09:52, 18 March 2011
Line 1:		Line 1:

	~~<br>~~


	= Introduction =		= Introduction =
	This project aims to realize a fast implementation of the novel L3 image-processing pipeline developed by Steven Lansel at Stanford University.		This project aims to realize a fast implementation of the novel L3 image-processing pipeline developed by Steven Lansel at Stanford University.

imported>Dgal1 at 09:50, 18 March 2011

2011-03-18T09:50:43Z

← Older revision		Revision as of 09:50, 18 March 2011
Line 1:		Line 1:
	~~Back to [[Psych204-Projects-2009 \|Psych 204 Projects 2009]]~~



	<br>		<br>

imported>Dgal1: /* Background and Motivation */

2011-03-18T09:46:44Z

Background and Motivation

← Older revision		Revision as of 09:46, 18 March 2011
Line 10:		Line 10:

	= Background and Motivation =		= Background and Motivation =
	Moore’s law has resulted in unprecedented scaling of both the technology and economy of the semi conductor industry. Among other advancements, semi-conductor scaling has enabled CMOS imaging sensors to reach resolutions of 10s of millions of pixels (~~MegaPixels~~). Traditional research in imaging technology has often focused on improving key metrics of these sensors such as SNR, sensitivity, and cost. The abundance of pixels in today’s commercially available imaging sensors has become something of a marketing campaign and competition for leading camera manufactures. At present both Canon and Nikon offer consumer cameras with more than 20 million pixels. Given a monitor with 72 dots per inch (DPI) such cameras are capable of producing images that at full size occupy in excess of 60” in a single dimension.		Moore’s law has resulted in unprecedented scaling of both the technology and economy of the semi-conductor industry. Among other advancements, semi-conductor scaling has enabled CMOS imaging sensors to reach resolutions of 10s of millions of pixels (megapixels). Traditional research in imaging technology has often focused on improving key metrics of these sensors such as SNR, sensitivity, and cost. The abundance of pixels in today’s commercially available imaging sensors has become something of a marketing campaign and competition for leading camera manufactures. At present both Canon and Nikon offer consumer cameras with more than 20 million pixels. Given a monitor with 72 dots per inch (DPI) such cameras are capable of producing images that at full size occupy in excess of 60” in a single dimension.
	As a result of what has become “overkill” for the typical consumer researchers have begun to consider other uses of pixels in imaging sensors. One such utilization of the pixels on a sensor is the realization of novel color filter arrays (CFA).		As a result of what has become “overkill” for the typical consumer researchers have begun to consider other uses of pixels in imaging sensors. One such utilization of the pixels on a sensor is the realization of novel color filter arrays (CFA).