Real-Time Animation, AI and Virtual Production: Opportunities and Challenges : And how to build a low-budget ICVFX stage

Henrik Schoenau-Fog; Aishah Hussain; Mathias Ramsø Thomsen

Authors

Henrik Schoenau-Fog Aalborg University
Aishah Hussain Aalborg University
Mathias Ramsø Thomsen Aalborg University

Keywords:

Virtual Production, Real-time animation, Final Pixels in Camera, In-Camera Visual Effects, ICVFX, Rear-projection, Projector-based, DIY, Low-budget, Unreal Engine, Unity 3D, Generative AI, DMX

Abstract

This paper explores the opportunities of utilizing real-time animation and AI in Virtual Production (VP) workflows. The investigation is based on insights from the ViZARTS project, where filmmakers, TV producers and animations are educated during hands-on workshops to utilize the potential of VP. The study will describe some of the challenges we have encountered and introduce a few innovative workflows, we have developed during the workshops, including how to build a low-budget sustainable VP training facility studio. Finally, we will look ahead and discuss the future of real-time animation and AI-driven workflows in Virtual Production.

References

ViZARTS Webpage, 2025. https://vizarts.aau.dk Visited 10.May, 2025.

Zwerman, S. and Jeffrey A. Okun J.A. 2023. The VES Handbook of Virtual Production. CRC Press.

Kadner, N., 2019. The virtual production field guide. Vol 1. Epic Games. (p.3) .https://cdn2.unrealengine.com/vp-field-guide-v1-3-01-f0bce45b6319.pdf

Unity. 2024. Time Ghost. https://unity.com/demos/time-ghost

Weta Digital & Unreal Engine. 2020. Meerkat. https://www.unrealengine.com/en-US/spotlights/behind-the-scenes-on-weta-digital-s-real-time-hair-and-fur-short-meerkat

ViZARTS video 1. 2020. https://youtu.be/Oi8ZtJIUsjQ?t=277

Akoulinine, I. 2021. DMX in Virtual Production for Film and Stage Performance. Aalborg University, Medialogy Masterthesis. https://kbdk-aub.primo.exlibrisgroup.com/permalink/45KBDK_AUB/a7me0f/alma9921568824605762

Raursø, N.E., Rasmussen, M.E., Persson, M.K., Petersen, T.A., Garðarsson, K.B., Schoenau-Fog, H. 2021. Lean-Back Machina: Attention-Based Skippable Segments in Interactive Cinema. In: Mitchell, A., Vosmeer, M. (eds) Interactive Storytelling. ICIDS 2021. Lecture Notes in Computer Science(), vol 13138. Springer, Cham. https://doi.org/10.1007/978-3-030-92300-6_12

Hansard, W. 2007. Projection-Compositing. In Burum, S.H., 2007. American Cinematographer Manual, vol. 1. American Cinematographer. (pp.253)

Amunet. 2024. Real-time Puppeteering in Unreal Engine 5 - Development showcase. https://www.youtube.com/watch?v=le6KuNA7oSE

Persson, C.B., Jensen, M.K.N., Sørensen, J.U. 2021. Camera Tracking Solution for Low-Budget Virtual Production. Aalborg Universitet. https://kbdk-aub.primo.exlibrisgroup.com/permalink/45KBDK_AUB/a7me0f/alma9921562915005762

Maicat video, 2022. https://www.youtube.com/watch?v=UTHKFjHnwwI

Apostolou, Achillefs. Luminating scenes: A study of light automation using deep learning in virtual production environments. Aalborg Universitet, 2024. https://kbdk-aub.primo.exlibrisgroup.com/permalink/45KBDK_AUB/a7me0f/alma9921883085005762

ViZARTS videos. https://www.youtube.com/@thevizartscollaboratorium5421/videos