DeepLabCut ^1,2,3, SLEAP ⁴, and OpenPose ⁵ have revolutionized the way behavioral scientists analyze data. These algorithm utilizes recent advances in computer vision and deep learning to automatically estimate 3D-poses. Interpreting the positions of an animal can be useful in studying behavior; however, it does not encompass the whole dynamic range of naturalistic behaviors.

B-SOiD identifies behaviors using a unique pipeline where unsupervised learning meets supervised classification. The unsupervised behavioral segmentation relies on non-linear dimensionality reduction ^6,7,9,10, whereas the supervised classification is standard scikit-learn ⁸.

Behavioral segmentation of open field in DeepLabCut, or B-SOiD ("B-side"), as the name suggested, was first designed as a pipeline using pose estimation file from DeepLabCut as input. Now, it has extended to handle DeepLabCut (.h5, .csv), SLEAP (.h5), and OpenPose (.json) files.

Git clone the web URL (example below) or download ZIP.

Change your current working directory to the location where you want the cloned directory to be made.

git clone https://github.com/YttriLab/B-SOID.git

cd /path/to/B-SOID/

For MacOS users

conda env create -n bsoid_v2 -f requirements.yaml (macOS)

or for Windows users

conda env create -n bsoid_v2 -f requirements_win.yaml (windows) 

conda activate bsoid_v2

You should now see (bsoid_v2) $yourusername@yourmachine ~ %

streamlit run bsoid_app.py

We have provided our 6 body part DeepLabCut model. We also included two example 5 minute clips (labeled_clip1, labeled_clip2) as proxy for how well we trained our model. The raw video (raw_clip1, raw_clip2) and the corresponding h5/pickle/csv files are included as well.

• matlab
• python-tsne
• python-umap
• bsoid app version 1

Pull requests are welcome. For recommended changes that you would like to see, open an issue. Join our slack group for more instantaneous feedback.

There are many exciting avenues to explore based on this work. Please do not hesitate to contact us for collaborations.

This software package provided without warranty of any kind and is licensed under the GNU General Public License v3.0. If you use our algorithm and/or model/data, please cite us! Preprint/peer-review will be announced in the following section.

September 2019: First B-SOiD preprint on bioRxiv

March 2020: Updated version of our preprint on bioRxiv

1. Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, Bethge M. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci. 2018 Sep;21(9):1281-1289. doi: 10.1038/s41593-018-0209-y. Epub 2018 Aug 20. PubMed PMID: 30127430.

2. Nath T, Mathis A, Chen AC, Patel A, Bethge M, Mathis MW. Using DeepLabCut for 3D markerless pose estimation across species and behaviors. Nat Protoc. 2019 Jul;14(7):2152-2176. doi: 10.1038/s41596-019-0176-0. Epub 2019 Jun 21. PubMed PMID: 31227823.

3. Insafutdinov E., Pishchulin L., Andres B., Andriluka M., Schiele B. (2016) DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model. In: Leibe B., Matas J., Sebe N., Welling M. (eds) Computer Vision – ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol 9910. Springer, Cham

4. Pereira, Talmo D., Nathaniel Tabris, Junyu Li, Shruthi Ravindranath, Eleni S. Papadoyannis, Z. Yan Wang, David M. Turner, et al. 2020. “SLEAP: Multi-Animal Pose Tracking.” bioRxiv.

5. Cao Z, Hidalgo Martinez G, Simon T, Wei SE, Sheikh YA. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. IEEE Trans Pattern Anal Mach Intell. 2019 Jul 17. Epub ahead of print. PMID: 31331883..

6. McInnes, L., Healy, J., & Melville, J. (2018). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.

7. McInnes, L., Healy, J., & Astels, S. (2017). hdbscan: Hierarchical density based clustering. The Journal of Open Source Software, 2(11), 205.

8. Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.

9. L.J.P. van der Maaten. Accelerating t-SNE using Tree-Based Algorithms. Journal of Machine Learning Research 15(Oct):3221-3245, 2014.

10. Chen M. EM Algorithm for Gaussian Mixture Model (EM GMM). MATLAB Central File Exchange. Retrieved July 15, 2019.