Mazzochette

Introduction:

In the Microsystems Lab, we are studying the sense of touch, specifically how our bodies undergo mechanotransduction, the process in which mechanical signals such as forces and displacements are converted into electrochemical signals that our nervous system can understand. The human body is incredibly complex, so we are first tackling the touch sensation problem in a popular neuroscience model organism, the Caenorhabditis elegans (C. elegans).

C. elegans is a small nematode, about 1 mm in length and about 50 um wide. It consists of about 1000 somatic cells and exactly 302 neurons. It was the first animal to have its genome fully mapped, as well as its complete neural network. Finally, it is a convenient sample in a biology lab because it grows to become a full adult animal in only four days. Because of all these traits, the animal lends itself easily to genetic mutation, allowing us to modulate different aspects of the worm’s composition for study.

We know that the worm has exactly six neurons that respond to gentle touch sensation. The morphology of these neurons is consistent across the species, with three of the cell bodies located near the animal’s tail and the other three cell bodies located about 50% of the length of the body from the head. Each of the neurons has a long process that runs from the cell body towards the head. <citation – wormbase> There exists strong evidence that points to specific ion channels in the processes of the neurons that open and close as strain is applied to the neuron. <citation – o hagan> As those channels open and close, ions flow in and out of the cell, creating a change in electric potential.

One method we are using to characterize the sense of touch is to quantify the behavioral response of the C. elegans to gentle touch. To do this, we probe the body of the worm with nano- and micro-scale forces and score whether or not the worm responds. We’ve built micro-scale cantlievers with an embedded strain gauge to apply calibrated forces and integrated them into a clamping system. <citation – sj park force clamp system paper> We apply forces to a freely moving worm and observe the response through a video taken through a stereoscope. A behavioral response is scored positively when the worm’s trajectory changes immediately after a force application.

Currently, this system can apply forces with high precision, but the low spatial and temporal precision of targeting and analyzing behavioral response limits the data throughput and quality. Targeting of the worm is done semi-manually and the behavioral response is scored visually frame by frame. The figure below shows the manual intervention done on each frame so that the software can differentiate the worm from the cantilever. The user must actually drawn the white lines that are the boundary between the worm and the cantilever. The user must also specify the threshold used to find the skeleton indicated by the dashed line along the worm.

I am currently building a second generation of the system to enable automated tracking of the worm in real time. In this project I seek to design a more robust version of the hardware system as well as investigate methods for thresholding the images, the first step in the image processing. Acquiring images and thresholding are the first two steps in the process of tracking the worm. It is imperative that they must be robust in order to accurately find the worm skeleton and target later in the processing pipeline.

Prior to starting this project, I already had a working prototype for acquiring a data set of video and images, as well as a basis code to skeletonize the worm. The hardware prototype consists primarily of a chemistry stand with a ring attached to it to hold the sample. The camera is placed below the sample. A light is held to the chemistry stand using a clamp. After capturing the images of the worm, I have written code based on the algorithm developed in <C. FY paper> to segment the body of the worm, use the segments to skeletonize the worm and find specified target along the skeleton. The basis for my code development is a library of C++ functions called OpenCV, an open source computer vision platform that employs many novel algorithms in computer vision as well as a set of functions, classes and structures for computation with images.

Methods:

The success of high spatial tracking precision relies heavily on the front end image processing steps, namely taking a good image and thresholding smartly. In this project I hope to improve upon the original methods that I set out to use for these steps. The key metrics of success are: \begin{itemize} \item Smooth outline of the worm's body \item The worm's body must be outlined by the longest contour of any feature deemed to be in the foreground by the thresholding method \item The quality of the images must be repeatable \item To enable real time processing, the time to process the image must be well below 66 ms, as governed by the 15 fps capture rate of the camera. \end{itemize}

Part 1: Designing stable hardware:

In this investigation, I hope to design a more robust hardware system so that images can be taken with repeatable quality and control. The first step in image processing, if controllable, is to take high quality images. I have found through taking several videos with my current system that stability in the construction and repeatable light placement is imperative for good analysis of images. My imaging system has two key innovations. The first is an inverted configuration, where the light is collected from below the sample. The second is an obliquely oriented lighting source. This configuration leverages the relative optical properties of the C. elegans and the silicon, the material from which the cantilevers are made. The nematode will scatted the light as it enters the body of the worm, whereas the cantilever reflects light. With an oblique source, the light incident on the cantilever will reflect at such an oblique angle that it will not be captured by the optics below. The light incident on the worm however will be scattered in many directions, including downwards where the camera can capture it. The figure below shows a sketch of the system, as well as a set of sample images taken with a prototype of the system with two orientations of the light, above the sample and to the side of the sample.

In this investigation, I hope to design a more robust hardware system so that images can be taken with repeatable quality and control. The first step in image processing, if controllable, is to take high quality images. I have found through taking several videos with my current system that stability in the construction and repeatable light placement is imperative for good analysis of images.

Part 2: Smart Thresholding

The first step in processing the images is to threshold to find the outline of the worm. In thresholding, all the pixels in the image are classified as either foreground (maxValue = 255) or background (minValue = 0) based on specified threshold. All thresholding methods follow the same basic structure as shown in the following equation. All techniques are structured around smartly picking $T(x,y)$.

$dst(x,y)=\{\begin{array}{cc}1& src(x,y)>T(x,y)\\ 0& else\end{array}$

As shown in the images above, some of the images include faint outline of the cantilever, which must be classified as background. In my original pass at the code, I arbitrarily guessed at thresholds until the images correctly divided into classes such that when contours were found, the largest contour was the worm. In this particular video, I had to crop out the cantilever to be able to find and process the worm.

In order to build a more robust processing algorithm, active thresholding techniques must be employed. To achieve this, I investigated three advanced methods for choosing T(x,y) such that body of the worm is designated as the picture foreground, while everything else in the image is specified as background. The first method is a popular, stochastically based algorithm called Otsu’s Method. The second and third methods are adaptive threshold techniques where a mean kernel and Gaussian kernel are used to determine a local threshold. All three of these methods are built into the OpenCV image processing library, simplifying implementation.

Otsu’s Method

The goal of Otsu’s method is to find the threshold that minimizes the variances of the histogram in each class. This can be shown to be equivalent of maximizing the between class variance, after making some assumptions including the process is stationary. We choose to maximize the between class variance because it can be calculated recursively. Total variance is the sum of the within class variance and the between class variance. Since the image is stationary, the variance is constant and minimizing one term is equivalent to maximizing the opposing term.

${\sigma }^2={\sigma }_w^2(T)+{\sigma }_b^2(T)$

${\sigma }_w^2(T)=P_f(T){\sigma }_f^2(T)+P_b(T){\sigma }_b^2(T)$

${\sigma }_b^2(T)=P_f(T)[1-P_f(T)][{\mu }_f(T)-{\mu }_b(T){]}^2$

Need to find $T_{opt}$ that maximizes ${\sigma }_b^2(T)$

In class probabilities:

$P_f(T)=\sum _{g=0}^{T}p(g),P_b(T)=\sum _{g=T+1}^{G}p(g)$

In class means:

${\mu }_f(T)=\sum _{g=0}^T\frac{gp(g)}{P_f^2},{\mu }_b(T)=\sum _{g=T+1}^G\frac{gp(g)}{P_b^2}$

In class variances:

${\sigma }_f^2=\sum _{g=0}^T[g-{\mu }_f(T){]}^2\frac{p(g)}{P_f(T)},{\sigma }_b^2=\sum _{g=T+1}^G[g-{\mu }_b(T){]}^2\frac{p(g)}{P_b(T)}$

Probability mass function:

$p(g)=\frac{n_g}{N},g=0,\mathrm{...},G$

where, $G$ is the maximum luminance value in the image. In my case this value if 255. $N$ is number of pixels, and $n_g$ is the number of pixels at value $g$ Note: $n_1+n_2+\mathrm{...}+n_G=N$

I implemented this algorithm in two ways. In the first strategy, from a set of 22 images from a single video, I found the Otsu threshold in the first image and then used that threshold for the rest of the images in the set. In the second strategy, I calculated the Otsu threshold for each image and used that threshold.

Adaptive Threshold Methods

In adaptive threshold methods, a unique threshold is found for each pixel in the image using a kernel of a specified block size centered on the pixel of interest. This method effectively smooths the local area before thresholding. OpenCV has two adaptive methods that I ran on my image samples, the mean kernel and the Gaussian kernel.

Mean Kernel

This method computes the local mean in the specified block size. The value of the pixel is above the local mean, the pixel is classified as foreground. If not, it is classified as background. To analyze this method, I ran the algorithm for a variety of block sizes and parameter C.

$T(x,y)=\frac{1}{BlockSiz{e}^2}\sum _{i=-\lfloor BlockSize/2\rfloor }^{\lfloor BlockSize/2\rfloor }\sum _{j=-\lfloor BlockSize/2\rfloor }^{\lfloor BlockSize/2\rfloor }src(x-1,y-j)-C$

Gaussian Kernel

In the Gaussian kernel method, the threshold T(x,y) is found by pointwise multiplying a Gaussian window of the specified block size with the neighborhood around the current pixel. The product is them summed over all the points in the neighborhood. If the value of the pixel is above the calculated threshold, it is classified as foreground, otherwise it is classified as background. To analyze this method, I ran the algorithm for a variety of block sizes and parameter C.

$T(x,y)=\sum _{i=-\lfloor BlockSize/2\rfloor }^{\lfloor BlockSize/2\rfloor }\sum _{j=-\lfloor BlockSize/2\rfloor }^{\lfloor BlockSize/2\rfloor }src(x-1,y-j)*G(x-i,y-j)-C$

Where $G(x,y)$ is a Gaussian window specified by the block size.

$G_i=\alpha *e^{-\frac{(i-\frac{BlockSize-1}{2}{)}^2}{(2\sigma {)}^2}},\sigma =0.3(\frac{BlockSize}{2}-1)+0.8$

$\alpha$ chosen such that $\sum _i{G}_i=1$

Summary of Threshold Methods

The following table gives a summary of the different methods analyzed:

Results:

The Otsu method proved to be the most effective of all threshold methods. The result of the contour finding processing step from three arbitrarily chosen frames are compared in the following figure for methods 1, 2, and 3. In the contour finding step, contours are drawn around all the foreground objects found in the thresholding step and the largest continuous contour is specified at the worm and is indicated by the blue contour. The first method is the empirically chosen threshold of 75 that I started with. In this method, the largest contour was always found to be the noise and cantilever outline in the upper left corner of the image. In the second method, the Otsu threshold was found on the first image (Frame 400) and was used for the rest of the frames in the video. This works under the assumption that the relative histograms of the foreground and background will not change much from frame to frame and time could be saved by not running the calculation on each frame. The Otsu threshold found for the first frame was 115. In the third method, the Otsu threshold was found uniquely for each frame. The average Otsu threshold among the 22 frames was 122.13 with a standard deviation of 12.84.

Methods 2 and 3, which use the Otsu threshold were 100% successful in identifying the largest contour as the worm, the contour is not the desired outline of the body of the worm. Instead the side walls of the worm are illuminated creating portions of the worm to be left out and the contour not smooth. In order to use this method, the worm must be more evenly illuminated. This is a key finding from this investigation as it affects the hardware construction of the system. Instead of a single light source from one direction, I will employ a ring light that surrounds the sample, applying light from all directions so that the worm is more evenly illuminated.

The adaptive threshold methods each had two parameters to modulate: the block size and the parameter C. I ran each method with different combinations of both parameters. The results are shown in the figures below. For C = 0, C = 10, the threshold results yielded exactly zero found contours in the thresholded images. Increasing the block size yielded a better outline of the worm, as we will investigate in the next section.

Mean Kernel Results:

Gaussian Kernel Results:

To demonstrate the effectiveness of the adaptive thresholding methods, the results of the contour finding as well as the threshold time and the percentage of frames that correctly identified the worm as the largest contour. The Gaussian kernel was much more effective at creating the worm as the largest contour. It makes sense that this method is more accurate because of the way it weights the pixels in the neighborhood. The pixels closest to the center are weighted higher than those farther from the center. It is also remarkable that the score decreases as the block size gets really big. I suspect this is because the filtering becomes too high, effectively smearing the background making other objects such as the cantilever, making that contour bigger that the one surrounding the worm. The smearing also brings out the entire worm. Therefore, the contour around the worm is a nice outline of the body, as we desire.

While the effectiveness of finding a nice contour of the worm's body is imperative, I must also consider the computation time. In order to process these images in real time, the entire skeleton finding process must be less than 66.6 ms, as governed by the 15 fps rate of the camera. Being well under this processing time is preferred as a buffer. Therefore, the time required to threshold was found for each method. The following figure shows the average time it took to find the thresholded image. The Otsu method requires no significant extra processing time, which makes it a great candidate for this application. Calculating the Otsu threshold hardly increases the threshold finding time! The adaptive threshold techniques are inherently slow because of their two dimensional nature and thus make them poor candidates for this application.

To demonstrate the correlation between computation time and the two dimensional nature of the algorithm, I have plotted the block size versus the computation time for each method in the figure below. The Gaussian kernel method computation time increases linearly with block size, but, surprisingly, the mean kernel remains constant. I was surprised by this result, given that the amount of computation to find the threshold is effectively the same in both methods: weight each pixel by some value and then sum the weighted pixels. In the Gaussian kernel method though, we need to calculate what those weights are based on the size of the block, which is increases in complexity as the block size increases. This is the result of the linear increase in threshold finding time with block size that we see in these results.

Finally, given the findings in the threshold investigation, the next version of the hardware is shown below. The key improvement is the use of a ring light surrounding the sample, as opposed to a single light source from one side. This even lighting will better illuminate the entire worm so that the threshold finding algorithm will binarize the entire worm as opposed to just the side walls, creating a nice contour around the body of the worm. Other features of the hardware construction include a structurally sound platform, a movable x, y stage controllable by a computer so that the sample can be moved relative to the cantilever mounted above it. The camera and optics will also be mounted on an automatically controlled jack so that the focus of the camera can be well controlled and maintained. This is sketched in the figure below.