Mazzochette

Introduction:

In the Microsystems Lab, we are studying the sense of touch, specifically how our bodies undergo mechanotransduction, the process in which mechanical signals such as forces and displacements are converted into electrochemical signals that our nervous system can understand. The human body is incredibly complex, so we are first tackling the touch sensation problem in a popular neuroscience model organism, the Caenorhabditis elegans (C. elegans).

C. elegans is a small nematode, about 1 mm in length and about 50 um wide. It consists of about 1000 somatic cells and exactly 302 neurons. It was the first animal to have its genome fully mapped, as well as its complete neural network. Finally, it is a convenient sample in a biology lab because it grows to become a full adult animal in only four days. Because of all these traits, the animal lends itself easily to genetic mutation, allowing us to modulate different aspects of the worm’s composition for study.

We know that the worm has exactly six neurons that respond to gentle touch sensation. The morphology of these neurons is consistent across the species, with three of the cell bodies located near the animal’s tail and the other three cell bodies located about 50% of the length of the body from the head. Each of the neurons has a long process that runs from the cell body towards the head. <citation – wormbase> There exists strong evidence that points to specific ion channels in the processes of the neurons that open and close as strain is applied to the neuron. <citation – o hagan> As those channels open and close, ions flow in and out of the cell, creating a change in electric potential.

One method we are using to characterize the sense of touch is to quantify the behavioral response of the C. elegans to gentle touch. To do this, we probe the body of the worm with nano- and micro-scale forces and score whether or not the worm responds. We’ve built micro-scale cantlievers with an embedded strain gauge to apply calibrated forces and integrated them into a clamping system. <citation – sj park force clamp system paper> We apply forces to a freely moving worm and observe the response through a video taken through a stereoscope. A behavioral response is scored positively when the worm’s trajectory changes immediately after a force application.

Currently, this system can apply forces with high precision, but the low spatial and temporal precision of targeting and analyzing behavioral response limits the data throughput and quality. Targeting of the worm is done semi-manually and the behavioral response is scored visually frame by frame, as indicated in the Figure below.

I am currently building a second generation of the system to enable automated tracking of the worm in real time. In this project I seek to design a more robust version of the hardware system as well as investigate methods for thresholding the images, the first step in the image processing. Acquiring images and thresholding are the first two steps in the process of tracking the worm. It is imperative that they must be robust in order to accurately find the worm skeleton and target later in the processing pipeline.

Prior to starting this project, I already had a working prototype for acquiring a data set of video and images, as well as a basis code to skeletonize the worm. After capturing the images of the worm, I have written code based on the algorithm developed in <C. FY paper> to skeletonize the worm and find specified target. The basis for my code development is a library of C++ functions called OpenCV, an open source computer vision platform that employs many novel algorithms in computer vision as well as a set of functions, classes and structures for computation with images.

Methods:

The imaging system has two key innovations. The first is an inverted configuration, where the light is collected from below the sample. The second is an obliquely oriented lighting source. This configuration leverages the relative optical properties of the C. elegans and the silicon, the material from which the cantilevers are made. The nematode will scatted the light as it enters the body of the worm, whereas the cantilever reflects light. With an oblique source, the light incident on the cantilever will reflect at such an oblique angle that it will not be captured by the optics below. The light incident on the worm however will be scattered in many directions, including downwards where the camera can capture it. The figure below shows a sketch of the system, as well as a set of sample images taken with a prototype of the system with two orientations of the light, above the sample and to the side of the sample.

In this investigation, I hope to design a more robust hardware system so that images can be taken with repeatable quality and control. The first step in image processing, if controllable, is to take high quality images. I have found through taking several videos with my current system that stability in the construction and repeatable light placement is imperative for good analysis of images.

The first step in processing the images is to threshold to find the outline of the worm. In thresholding, all the pixels in the image are classified as either foreground (maxValue = 255) or background (minValue = 0) based on specified threshold.

$dst(x,y)=$

As shown in the images above, some of the images include faint outline of the cantilever, which must be classified as background. In my original pass at the code, I arbitrarily guessed at thresholds until the images correctly divided into classes such that when contours were found, the largest contour was the worm. In this particular video, I had to crop out the cantilever to be able to find and process the worm.

In order to build a more robust processing algorithm, active thresholding techniques must be employed. To achieve this, I investigated three advanced methods for choosing T(x,y) such that body of the worm is designated as the picture foreground, while everything else in the image is specified as background. The first method is a popular, stochastically based algorithm called Otsu’s Method. The second and third methods are adaptive threshold techniques where a mean kernel and Gaussian kernel are used to determine a local threshold. All three of these methods are built into the OpenCV image processing library, simplifying implementation.

Otsu’s Method

The goal of Otsu’s method is to find the threshold that minimizes the variances of the histogram in each class. This can be shown to be equivalent of maximizing the between class variance, after making some assumptions including the process is stationary. We choose to maximize the between class variance because it can be calculated recursively. Total variance is the sum of the within class variance and the between class variance. Since the image is stationary, the variance is constant and minimizing one term is equivalent to maximizing the opposing term.

${\sigma }^2={\sigma }_w^2(T)+{\sigma }_b^2(T)$

${\sigma }_w^2(T)=P_f(T){\sigma }_f^2(T)+P_b(T){\sigma }_b^2(T)$

${\sigma }_b^2(T)=P_f(T)[1-P_f(T)][{\mu }_f(T)-{\mu }_b(T){]}^2$

Need to find $T_{opt}$ that maximizes ${\sigma }_b^2(T)$

In class probabilities:

$P_f(T)=\sum _{g=0}^{T}p(g),P_b(T)=\sum _{g=T+1}^{G}p(g)$

In class means:

${\mu }_f(T)=\sum _{g=0}^T\frac{gp(g)}{P_f^2},{\mu }_b(T)=\sum _{g=T+1}^G\frac{gp(g)}{P_b^2}$

In class variances:

${\sigma }_f^2=\sum _{g=0}^T[g-{\mu }_f(T){]}^2\frac{p(g)}{P_f(T)},{\sigma }_b^2=\sum _{g=T+1}^G[g-{\mu }_b(T){]}^2\frac{p(g)}{P_b(T)}$

Probability mass function:

$p(g)=\frac{n_g}{N},g=0,\mathrm{...},G$

where, $G$ is the maximum luminance value in the image. In my case this value if 255. $N$ is number of pixels, and $n_g$ is the number of pixels at value $g$ Note: $n_1+n_2+\mathrm{...}+n_G=N$

I implemented this algorithm in two ways. In the first strategy, from a set of 22 images from a single video, I found the Otsu threshold in the first image and then used that threshold for the rest of the images in the set. In the second strategy, I calculated the Otsu threshold for each image and used that threshold.

Adaptive Threshold Methods

In adaptive threshold methods, a unique threshold is found for each pixel in the image using a kernel of a specified block size centered on the pixel of interest. This method effectively smooths the local area before thresholding. OpenCV has two adaptive methods that I ran on my image samples, the mean kernel and the Gaussian kernel.

Mean Kernel

This method computes the local mean in the specified block size. The value of the pixel is above the local mean, the pixel is classified as foreground. If not, it is classified as background. To analyze this method, I ran the algorithm for a variety of block sizes and parameter C.

$T(x,y)=\frac{1}{BlockSiz{e}^2}\sum _{i=-\lfloor BlockSize/2\rfloor }^{\lfloor BlockSize/2\rfloor }\sum _{j=-\lfloor BlockSize/2\rfloor }^{\lfloor BlockSize/2\rfloor }src(x-1,y-j)-C$

Gaussian Kernel

In the Gaussian kernel method, the threshold T(x,y) is found by pointwise multiplying a Gaussian window of the specified block size with the neighborhood around the current pixel. The product is them summed over all the points in the neighborhood. If the value of the pixel is above the calculated threshold, it is classified as foreground, otherwise it is classified as background. To analyze this method, I ran the algorithm for a variety of block sizes and parameter C.

$T(x,y)=\sum _{i=-\lfloor BlockSize/2\rfloor }^{\lfloor BlockSize/2\rfloor }\sum _{j=-\lfloor BlockSize/2\rfloor }^{\lfloor BlockSize/2\rfloor }src(x-1,y-j)*G(x-i,y-j)-C$

Where $G(x,y)$ is a Gaussian window specified by the block size.

$G_i=\alpha *e^{-\frac{(i-\frac{BlockSize-1}{2}{)}^2}{(2\sigma {)}^2}},\sigma =0.3(\frac{BlockSize}{2}-1)+0.8$

$\alpha$ chosen such that $\sum _i{G}_i=1$

Results:

The Otsu method proved to be the most effective of all threshold methods. The result of the contour finding processing step from three arbitrarily chosen frames are compared in the following figure for methods 1, 2, and 3. In the contour finding step, contours are drawn around all the foreground objects found in the thresholding step and the largest continuous contour is specified at the worm and is indicated by the blue contour. The first method is the empirically chosen threshold of 75 that I started with. In this method, the largest contour was always found to be the noise and cantilever outline in the upper left corner of the image. In the second method, the Otsu threshold was found on the first image (Frame 400) and was used for the rest of the frames in the video. This works under the assumption that the relative histograms of the foreground and background will not change much from frame to frame and time could be saved by not running the calculation on each frame. The Otsu threshold found for the first frame was 115. In the third method, the Otsu threshold was found uniquely for each frame. The average Otsu threshold among the 22 frames was 122.13 with a standard deviation of 12.84.

Methods 2 and 3, which use the Otsu threshold were 100% successful in identifying the largest contour as the worm, the contour is not the desired outline of the body of the worm. Instead the side walls of the worm are illuminated creating portions of the worm to be left out and the contour not smooth. In order to use this method, the worm must be more evenly illuminated. This is a key finding from this investigation as it affects the hardware construction of the system. Instead of a single light source from one direction, I will employ a ring light that surrounds the sample, applying light from all directions so that the worm is more evenly illuminated.

The adaptive threshold methods each had two parameters to modulate: the block size and the parameter C. I ran each method with different combinations of both parameters. The results are shown in the figures below. For C = 0, C = 10, the threshold results yielded exactly zero found contours in the thresholded images. Increasing the block size yielded a better outline of the worm, as we will investigate in the next section.

Mean Kernel Results:

Gaussian Kernel Results: