Scientists who study animal behavior spend endless hours observing and taking notes about a creature’s actions and reactions. Computer vision could automate much of that work.
What’s new: Researchers at Facebook, the Max Planck Institute for Evolutionary Anthropology, and the Pan-African Programme (part of a partnership between African and European governments) built a neural network that tracks the body position of chimpanzees. The system captures the animals’ behavior in three dimensions for study and analysis.
How it works: The researchers started with DensePose, a pose estimator pre-trained on videos of humans. They fine-tuned it for chimps in two phases, first using a segmentation model and then using a teacher-student scheme.
- A single feature extractor fed the segmentation model and DensePose. The segmentation model updated the feature extractor by learning to detect animals in labeled examples from the Coco image dataset.
- The researchers purchased a 3D model of a chimpanzee and mapped its points to DensePose’s 3D human model. This enabled DensePose predictions to be interpreted as a location on a chimp.
- A teacher model (DensePose using updated features) received around 18,500 unlabeled video clips of chimps in the wild. It predicted how their pixels mapped to the 3D chimp model and provided its confidence in each prediction.
- The student model learned to recreate the teacher’s most confident predictions. Then the student became the teacher for a new round of learning, and so on.
Behind the news: This work complements earlier efforts to use deep learning to help scientists study animal behavior.
- DeepLabCut generates wireframe pose estimations of creatures such as fruit flies, rats, and even horses.
- AniPose offers 3D pose estimations of animal imagery captured across multiple cameras.
Why it matters: Annotating videos of animal behavior is labor-intensive, and building annotated datasets for thousands of species would be prohibitively expensive. The authors adapted a neural network’s knowledge of human anatomy to work with another species, albeit a similar one. They believe their method could work with less human-like species as well.
We’re thinking: What a brilliant ape-lication!