An Initial Study on Enhancing AR Physical Coaching
Multimodal Posture Feedback Using Local ML and LLMs
DOI:
https://doi.org/10.54337/aau.icdvrat26.08Abstract
This preliminary study presents an Augmented Reality (AR) coaching application for physical rehabilitation. Using a Meta Quest 3 and an iPhone, the system offers a low-strain environment for home exercises. In a user study (N=11), participants performed shoulder presses guided by a 3D avatar. A local machine learning model analyzed their form, triggering personalized audio feedback via a Large Language Model (LLM). NASA-TLX evaluations showed minimal user exertion. Despite minor tracking and API limits, the system demonstrates strong potential for integrating immersive, multimodal AR into accessible physical therapy.
References
Argent, R., Daly, A. & Caulfield, B. (2018), ‘Patient involvement with home-based exercise programs: can connected health interventions influence adherence?’, JMIR mHealth and uHealth 6(3), e8518.
Gil, M. J. V., Gonzalez-Medina, G., Lucena-Anton, D., Perez-Cabezas, V., Ruiz-Molinero, M. D. C. & Mart´ın-Valero, R. (2021), ‘Augmented reality in physical therapy: systematic review and meta-analysis’, JMIR Serious Games 9(4), e30985.
Goh, E. S., Sunar, M. S. & Ismail, A.W. (2019), ‘3d object manipulation techniques in handheld mobile augmented reality interface: A review’, IEEE Access 7, 40581–40601.
NASA Human Systems Integration Division (n.d.), ‘NASA Task Load Index (TLX) Rating Scale PDF’. Accessed: 2026-01-05. URL: https://humansystems.arc.nasa.gov/groups/tlx/downloads/TLXScale.pdf
Nizam, S. S. M., Abidin, R. Z., Hashim, N. C., Lam, M. C., Arshad, H. & Majid, N. A. A. (2018), ‘A review of multimodal interaction technique in augmented reality environment’, International Journal on Advanced Science, Engineering and Information Technology 8(4-2), 1459–1466.
