MichaelAlissa

Introduction

alt text

alt text

Systems neuroscience, the study of how the brain works at the scale of systems (neural circuits, cortical regions, etc.) is the foundation of Brain Machine Interfaces used in robotic prosthetics that are controlled by the brain. With the emergence of multichannel recordings over recent decades, the field has been revolutionized by access to hundreds of simultaneously recorded neurons, and awake behaving animal experiments have become even more critical to advancing our understanding (2). Classical nonhuman primate (NHP) in-rig experiments constrain most bodily movements except the movement of interest to reduce confounding variables to draw tight correlations between the desired behavior and neural population dynamics (Figure 1a) (3 citation Georgeopelous 1983). However, it is unclear if these results generalize to ambulatory behavior. Further, some evidence suggests that the complexity/variability of the neural recordings is constrained by the complexity of the task being performed, artificially and unintentionally limiting the observed neural data (4 Gao & Ganguly CONB 2015). To address this, experiments with higher task complexity need to be conducted. Our lab looks to conduct freely moving experiments to directly ask whether increasing task complexity yields greater neural variance and how the extra neural variance correlates to various limb kinematics. To do this, we aim to simultaneously record neural data from motor regions of cortex of a freely moving rhesus macaque using a commercial wireless electrophysiology system and capture video of kinematic movements using multiple stereo depth cameras surrounding a large, transparent, observational rig (Figure). This 3D data will yield a point cloud which will be fit to a skeleton to extract the kinematics of the monkey. In this project, we attempt to determine whether the Intel Realsense d435 depth camera would be a suitable device for the capture of this point cloud data.

Background

The Intel RealSense D435 depth camera is a low-cost camera capable of recording both depth and RGB data. For our application, however, we are only concerned with the depth information. This depth stream employs a global shutter, has a diagonal field of view of $95^{\circ }\pm 3^{\circ }$, and at a resolution of 848x480 pixels can achieve frame rates of up to 90 frames per second. Able to detect depths from 0.1 to 10 meters, this camera calculates depth using two monochrome sensors, as shown in Figure#. These monochrome sensors—which detect both visible and infrared light—perform stereoscopic matching to calculate the frame depth values. Depicted in Figure#, stereoscopic matching consists in first calculating the disparity (i.e. shift in the horizontal axis) between images created by the two cameras. As the focal length and baseline distance (i.e. the distance between the imaging sensors), depth values can be calculated with the following equation:

$depth=\frac{baseline\cdot focallength}{x_1-x_2}$ where $x_1,x_2$ are the horizontal positions of the object captured by cameras 1 and 2, respectively.

Along with these depth sensors, the RealSense D435 is equipped with an infrared projector which illuminates the scene with infrared dots, providing additional markers for image alignment and helping to reduce measurement error.

Methods

Accuracy and Temporal Noise Characterization

To establish a baseline of the error present in the point cloud construction from multi-camera setup, we first characterized the accuracy and temporal noise for a single camera. To accomplish this, we designed an experimental rig—depicted in Figures # and #—which allowed the imaging plane of the camera and the surfaced of the wall to oriented parallel to each other. The camera could then be adjusted to different distances from the wall while maintaining this orientation. We performed recordings ranging from 0.2 – 2 meters at increments of 0.2 meters. Five, 1-minute recording with a sampling rate of 90 frames per second were taken at each measurement distance, analysis for all recordings was confined to the same section of wall, which possessed a matte, textured finish.

alt text

alt text

Camera Calibration

With the characterization of a single camera complete, we moved to a multi-camera setup, depicted in Figure#. This setup was composed of two cameras, which were slightly offset from one another. This arrangement not only provided a larger field of view, but the collection of multiple perspectives also allowed us to capture a more complete representation of the object. In order to combine these cameras together, however, we needed to relate the camera coordinate systems to each other. While methods—such as checkerboard calibration—exist to calibrate cameras using RGB data, we did not want to use this method of calibration for two reasons: (1) aligning between the depth and color camera sensors could introduce additional error into our measurements; and (2) our project does not seek to capture RBG data. We therefore developed a calibration technique which relies solely on depth data.

The first step in this process was to fabricate a calibration tool, which is shown in Figure#. This tool, an asymmetrical body consisting of 4 spheres, provided four locations where similar points between the two cameras could easily be extracted using only depth data. These spheres were also optimal for the template matching algorithm we employed, allowing us automatically locate the positions of the spheres in our image. Template matching was performed by taking a cropped image of the sphere, with all values outside the sphere set to zero. This cropped image, or template, was then swept across the image, and the location which most closely matches the template is returned. This process is illustrated in Figure#, with numbered templates lining the left-hand side of the image and numbered squares in the depth image detailing the region that best aligned with the template. With the location of the sphere determined, our algorithm returns the midpoint of the sphere, which provides a common point between the two cameras that be used for alignment.

With this algorithm, we can successfully locate the four ball positions, providing four common points which we can use to align our camera system. Templates 3 and 4 in Figure# provide examples of successful template matches. In its current state, however, our algorithm can provide false positive matches. For example, templates 1 and 2 in Figure# correspond to templates that do not match the spheres on our calibration tool, yet find high confidence matching locations outside of the tool. Currently, these false positives must be manually excluded when running the algorithm, but future iterations of this calibration technique will look to remedy this issue.

alt text

3D Point Cloud Construction

With a two sets equivalent points described in each camera’s xyz coordinate system, we used an approach outlined in ref[] to calculate the rotation matrix and translation vector, allowing us to transform points in one camera’s coordinate system to the coordinate system of a second camera. For 3 x n matrices A and B, which represent similar sets of 3D points for camera 1 and camera 2, respectively, this approached can outlined as follows:

1. Calculate the centroid for each set of points, given by the equation $A_{centroid}=\frac{1}{N}\sum _{i=1}^N{A}_i$. Shown in Figure #, this is simply the average of the x, y, and z values for all points in a set.
2. Center both sets of points at the origin (Figure #). This is accomplished subtracting the centroid of each set from all the points in the corresponding set.
3. To find the rotation from A to B in Figure #, we take the singular value decomposition (SVD) of the cross-covariance matrix of A and B. This cross covariance matrix can be calculated as $H=(A_{centered}){(B_{centered})}^T$. We can then take the SVD of this cross-covariance matrix $Q_1\Sigma Q_2=SVD(H)$. Since we are assuming a rigid transformation (no scaling or shearing), the $\Sigma$ term can be neglected. The rotation matrix $R$I s then given as $(Q_1){(Q_2)}^T$. This rotation can be applied to A to so that it possesses the same orientation as B, as shown in Figure #.
4. To determine the translation vector $t$, we find the vector joining the B and the rotated set A, seen in Figure#: $t=(B_{centroid})-(R)(A_{centroid})$.

alt text

Results

Accuracy and Temporal Noise Characterization

With the recordings described in the Methods section, two metrics were calculated: (1) depth error and (2) temporal noise. Depth error, which relates the camera’s ability to accurately determine an object’s distance, can be described as $formula$ which is simply average of differences between the wall depths reported by the camera and true distance to the wall. Temporal noise, which characterizes the spread of depth values on an object over the duration of a recording, is calculated as $formula$ where is the standard deviation of the $i^{(}th)$ pixel over a recording. Plots of these calculations at different distances can be viewed in Figure #. As seen in these figures, both depth error and temporal noise increase as the square of the distance between the wall and the camera, which—as described below—impacts the ability of these cameras to accurately reconstruct 3D point clouds from this data.

alt text

While performing these calculations, we also analyzed the effect that different surfaces had on camera error. Examining a painted section of wall, shown in Figure #, the temporal noise at 2 meters was evaluated. A heat map of the pixel standard deviations over the minute-long recording can be viewed in Figure #. From this heat map, glossier areas (such as the brown trees in Figure #) create significantly more temporal noise than areas with matte finish. This highlights the d435’s susceptibility to increased error in highly reflective environments, a fact which must be considered whenever recordings are performed.

alt text

3D Point Cloud Construction

Using the method above to calculate the rotation matrix and translation vector, aligned point cloud reconstruction was performed using the two-camera setup. This setup was used to image the calibration tool 0.2 meter increments from 0.2 – 1 meter. As shown in Figure#, by applying the appropriate rotation and translation, data from the first camera can be aligned with the coordinate space of camera 2, allowing this data to be combined. While each separate camera image (Figure#a,b) contains only a partial depiction of the calibration tool, these images can be combined to provide a more complete representation of the object. When this process is repeated at greater distances from the two cameras, as depicted in Figure#, the point cloud reconstruction quality dramatically decreases. The data becomes noisier, the alignment between the cameras worsens, and it becomes difficult to distinguish finer details in the calibration tool. The rapid increase in measurement noise that occurs as distance increases—a phenomenon characterized above for a single camera—is a significant drawback of the RealSense d435, limiting its ability to perform 3d reconstruction outside of close range measurements.

Appendix

You can write math equations as follows: $y=x+5$

You can include images as follows (you will need to upload the image first using the toolbox on the left bar, using the "Upload file" link).

[[File:Screen_Shot_2016-11-29_at_7.05.37_PM.png|200px]