Accurate quantification of multi-animal behavior plays a pivotal role in unraveling the intricacies of animal social interactions, with far-reaching applications in neuroscience and ecology. 

Researchers from the Brain Cognition and Brain Disease Institute (BCBDI) of the Shenzhen Institute of Advanced Technology at the Chinese Academy of Sciences have proposed a few-shot learning AI framework, the Social Behavior Atlas (SBeA), for multi-animal 3D social pose estimation, identification, and behavior embedding.

The research was published on January 8th in Nature Machine Intelligence. 

While recent advances in deep learning methods have improved the accessibility of quantifying high-dimensional social behaviors in animals, including pose estimation, identity recognition, and behavior classification, their applications are restricted by the availability of insufficiently annotated datasets. 

In this study, the researchers developed a continuously occluded copy-paste algorithm (COCA) as a universal data augmentor to reduce data annotation to about 400 frames in the multi-animal pose estimation step. This is equivalent to single-animal annotations and achieves higher performance than state-of-the-art methods. Combined with the camera array, SBeA achieves 3D reconstruction of social animals. 

The proposed bidirectional transfer learning allows SBeA to recognize each animal's identity during social interaction without the need for manual annotations. This resolves the problem of perplexing identity recognition for animals with similar appearances, even for professional human annotators. 

The 3D social poses with identities are further decomposed and clustered by the unsupervised social behavior classification of SBeA. It classifies social behavior without predefined categories, which is useful for revealing undefined behaviors. 

SBeA helps researchers identify undefined subtle social behavior modules in Shank3B mutant mice, an animal model used to simulate autism spectrum disorders. It indicates the existence of unknown neural modulation mechanisms behind subtle social behaviors. In addition to mice, SBeA effectively identifies subtle social behavior across species, such as birds and dogs. Neuroscience and ecology would benefit from the accurate animal social behavior quantification provided by SBeA.