Lee, D. W., Kim, Y., Picard, R., Breazeal, C., & Park, H. W. (2023). Multipar-T: Multiparty-Transformer for Capturing Contingent Behaviors in Group Conversations. arXiv preprint arXiv:2304.12204.

Abstract

As we move closer to real-world social AI systems, AI agents must be able to deal with multi- party (group) conversations. Recognizing and interpreting multiparty behaviors is challenging, as the system must recognize individual behavioral cues, deal with the complexity of multiple streams of data from multiple people, and recognize the subtle contingent social exchanges that take place amongst group members. To tackle this challenge, we propose the Multiparty-Transformer (MultiPar- T), a transformer model for multiparty behavior modeling. The core component of our proposed approach is Crossperson Attention, which is specifically designed to detect contingent behavior be- tween pairs of people. We verify the effectiveness of MultiPar-T on a publicly available video-based group engagement detection benchmark, where it outperforms state-of-the-art approaches in average F-1 scores by 5.2% and individual class F-1 scores by up to 10.0%. Through qualitative analysis, we show that our Crossperson Attention module is able to discover contingent behaviors.