MichaelAlissa

Introduction

Systems neuroscience, the study of how the brain works at the scale of systems (neural circuits, cortical regions, etc.) is the foundation of Brain Machine Interfaces used in robotic prosthetics that are controlled by the brain. With the emergence of multichannel recordings over recent decades, the field has been revolutionized by access to hundreds of simultaneously recorded neurons, and awake behaving animal experiments have become even more critical to advancing our understanding (2). Classical nonhuman primate (NHP) in-rig experiments constrain most bodily movements except the movement of interest to reduce confounding variables to draw tight correlations between the desired behavior and neural population dynamics (Figure 1a) (3 citation Georgeopelous 1983). However, it is unclear if these results generalize to ambulatory behavior. Further, some evidence suggests that the complexity/variability of the neural recordings is constrained by the complexity of the task being performed, artificially and unintentionally limiting the observed neural data (4 Gao & Ganguly CONB 2015). To address this, experiments with higher task complexity need to be conducted. Our lab looks to conduct freely moving experiments to directly ask whether increasing task complexity yields greater neural variance and how the extra neural variance correlates to various limb kinematics. To do this, we will simultaneously record neural data from motor regions of cortex of a freely moving rhesus macaque using a commercial wireless electrophysiology system and capture video of kinematic movements using multiple stereo depth cameras surrounding a large, transparent, observational rig (Figure). This 3D data will yield a pointcloud which will be fit to a skeleton to extract the kinematics of the monkey. In this project, we attempt to determine whether the Intel Realsense d435 depth camera would be a suitable device for the capture of this point cloud data.

Methods

Accuracy and Temporal Noise Characterization

To establish a baseline of the error present in the point cloud construction from multi-camera setup, we first characterized the accuracy and temporal noise for a single camera. To accomplish this, we designed an experimental rig—depicted in Figures # and #—which allowed the imaging plane of the camera and the surfaced of the wall to oriented parallel to each other. The camera could then be adjusted to different distances from the wall while maintaining this orientation. We performed recordings ranging from 0.2 – 2 meters at increments of 0.2 meters. Five, 1-minute recording with a sampling rate of 90 frames per second were taken at each measurement distance, analysis for all recordings was confined to the same section of wall, which possessed a matte, textured finish.

alt text

3D Point Cloud Construction

With a two sets equivalent points described in each camera’s xyz coordinate system, we used an approach outlined in ref[] to calculate the rotation matrix and translation vector, allowing us to transform points in one camera’s coordinate system to the coordinate system of a second camera. For 3 x n matrices A and B, which represent similar sets of 3D points for camera 1 and camera 2, respectively, this approached can outlined as follows:

1. Calculate the centroid for each set of points, given by the equation $A_{centroid}=\frac{1}{N}\sum _{i=1}^N{A}_i$. Shown in Figure #, this is simply the average of the x, y, and z values for all points in a set.
2. Center both sets of points at the origin (Figure #). This is accomplished subtracting the centroid of each set from all the points in the corresponding set.
3. To find the rotation from A to B in Figure #, we take the singular value decomposition (SVD) of the cross-covariance matrix of A and B. This cross covariance matrix can be calculated as $H=(A_{centered}){(B_{centered})}^T$. We can then take the SVD of this cross-covariance matrix $Q_1\Sigma Q_2=SVD(H)$. Since we are assuming a rigid transformation (no scaling or shearing), the $\Sigma$ term can be neglected. The rotation matrix $R$I s then given as $(Q_1){(Q_2)}^T$. This rotation can be applied to A to so that it possesses the same orientation as B, as shown in Figure #.
4. To determine the translation vector $t$, we find the vector joining the B and the rotated set A, seen in Figure#: Failed to parse (syntax error): {\displaystyle t = (B_{centroid}) – (R)(A_{centroid})} .

alt text

Results

Accuracy and Temporal Noise Characterization

With the recordings described in the Methods section, two metrics were calculated: (1) depth error and (2) temporal noise. Depth error, which relates the camera’s ability to accurately determine an object’s distance, can be described as $formula$ which is simply average of differences between the wall depths reported by the camera and true distance to the wall. Temporal noise, which characterizes the spread of depth values on an object over the duration of a recording, is calculated as $formula$ where is the standard deviation of the $i^{(}th)$ pixel over a recording. Plots of these calculations at different distances can be viewed in Figure #. As seen in these figures, both depth error and temporal noise increase as the square of the distance between the wall and the camera, which—as described below—impacts the ability of these cameras to accurately reconstruct 3D point clouds from this data.

alt text

While performing these calculations, we also analyzed the effect that different surfaces had on camera error. Examining a painted section of wall, shown in Figure #, the temporal noise at 2 meters was evaluated. A heat map of the pixel standard deviations over the minute-long recording can be viewed in Figure #. From this heat map, glossier areas (such as the brown trees in Figure #) create significantly more temporal noise than areas with matte finish. This highlights the d435’s susceptibility to increased error in highly reflective environments, a fact which must be considered whenever recordings are performed.

alt text

Appendix

You can write math equations as follows: $y=x+5$

You can include images as follows (you will need to upload the image first using the toolbox on the left bar, using the "Upload file" link).

[[File:Screen_Shot_2016-11-29_at_7.05.37_PM.png|200px]