LLM-based Teamwork Role Inferencing for Fostering Social Online Learning

Abstract

Higher education institutions are progressively implementing online learning modules due to their promotion of equity through scalability, accessibility, inclusivity, and affordability. Despite the advantages, online learning environments face persistent challenges in facilitating meaningful networked learning (NL) opportunities. One particularly pressing challenge is supporting effective NL through group formation and collaborative teamwork. With rising enrolments, manual grouping has become impractical. Random allocation overlooks complementary skills and often produces unbalanced teams, while self-assessment relies on students’ often inaccurate self-perceptions, leading to mismatched roles and group tensions. A substantial body of research has examined strategies for group formation, and many studies have emphasised the value of team role allocation in promoting effective NL. However, little attention has been paid to approaches that infer learners’ collaborative role tendencies from their actual behavioural interactions and subsequently use this information to inform group composition and achieve a better NL experience. Accordingly, this study examines whether large language models (LLMs) can infer higher-education learners’ teamwork role tendencies comparably to human judgment, using data from an international sample of undergraduate and postgraduate learners recruited via Prolific who interacted with a custom-built chatbot in collaborative learning scenarios. The resulting interactions were analysed through an LLM to infer teamwork role tendencies based on previously established Belbin Team Roles. The roles inferred by the LLM were then compared against those coded by human coders, with inter-rater alignment evaluated using Cohen’s Kappa and percentage agreement. Learner responses were coded by trained human researchers using a consensus-based Belin Team Role framework. The findings reveal that LLM achieved a moderate degree of alignment with human coders, suggesting its viability as a tool for inferring learners’ teamwork role tendencies. Moreover, exploratory analyses revealed that the length of learners’ responses to the chatbot is associated with the extent to which the LLM’s inferences aligned with human coders. However, future research would benefit from larger sample sizes and the use of more advanced statistical methods to better capture the effects of interaction quality. This study contributes to the growing body of work on LLM-supported NL by highlighting both the potential and the limitations of using LLMs for role inference. Future implementations should pay particular attention to fostering high-quality learner-chatbot interactions to maximise reliability and pedagogical value to help design for effective social participation in online NL experiences. The findings of this research are expected to advance the development of technology-enhanced, NL practices within online higher education.