Real-Time Animation, AI and Virtual Production: Opportunities and Challenges : And how to build a low-budget ICVFX stage

Henrik Schoenau-Fog; Aishah Hussain; Mathias Ramsø Thomsen

doi:10.54337/caga24-11168

Authors

Henrik Schoenau-Fog Aalborg University
Aishah Hussain Aalborg University
Mathias Ramsø Thomsen Aalborg University

DOI:

https://doi.org/10.54337/caga24-11168

Keywords:

Virtual Production, Real-time animation, Final Pixels in Camera, In-Camera Visual Effects, ICVFX, Rear-projection, Projector-based, DIY, Low-budget, Unreal Engine, Unity 3D, Generative AI, DMX

Abstract

This paper explores the opportunities of utilizing real-time animation and AI in Virtual Production (VP) workflows. The investigation is based on insights from the ViZARTS project, where filmmakers, TV producers and animations are educated during hands-on workshops to utilize the potential of VP. The study will describe some of the challenges we have encountered and introduce a few innovative workflows, we have developed during the workshops, including how to build a low-budget sustainable VP training facility studio. Finally, we will look ahead and discuss the future of real-time animation and AI-driven workflows in Virtual Production.

References

ViZARTS Webpage, 2025. https://vizarts.aau.dk Visited 10.May, 2025.

Zwerman, S. and Jeffrey A. Okun J.A. 2023. The VES Handbook of Virtual Production. CRC Press. DOI: https://doi.org/10.4324/9781003366515

Kadner, N., 2019. The virtual production field guide. Vol 1. Epic Games. (p.3) .https://cdn2.unrealengine.com/vp-field-guide-v1-3-01-f0bce45b6319.pdf

Unity. 2024. Time Ghost. https://unity.com/demos/time-ghost

Weta Digital & Unreal Engine. 2020. Meerkat. https://www.unrealengine.com/en-US/spotlights/behind-the-scenes-on-weta-digital-s-real-time-hair-and-fur-short-meerkat

ViZARTS video 1. 2020. https://youtu.be/Oi8ZtJIUsjQ?t=277

Akoulinine, I. 2021. DMX in Virtual Production for Film and Stage Performance. Aalborg University, Medialogy Masterthesis. https://kbdk-aub.primo.exlibrisgroup.com/permalink/45KBDK_AUB/a7me0f/alma9921568824605762

Raursø, N.E., Rasmussen, M.E., Persson, M.K., Petersen, T.A., Garðarsson, K.B., Schoenau-Fog, H. 2021. Lean-Back Machina: Attention-Based Skippable Segments in Interactive Cinema. In: Mitchell, A., Vosmeer, M. (eds) Interactive Storytelling. ICIDS 2021. Lecture Notes in Computer Science(), vol 13138. Springer, Cham. https://doi.org/10.1007/978-3-030-92300-6_12 DOI: https://doi.org/10.1007/978-3-030-92300-6_12

Hansard, W. 2007. Projection-Compositing. In Burum, S.H., 2007. American Cinematographer Manual, vol. 1. American Cinematographer. (pp.253)

Amunet. 2024. Real-time Puppeteering in Unreal Engine 5 - Development showcase. https://www.youtube.com/watch?v=le6KuNA7oSE

Persson, C.B., Jensen, M.K.N., Sørensen, J.U. 2021. Camera Tracking Solution for Low-Budget Virtual Production. Aalborg Universitet. https://kbdk-aub.primo.exlibrisgroup.com/permalink/45KBDK_AUB/a7me0f/alma9921562915005762

Maicat video, 2022. https://www.youtube.com/watch?v=UTHKFjHnwwI

Apostolou, Achillefs. Luminating scenes: A study of light automation using deep learning in virtual production environments. Aalborg Universitet, 2024. https://kbdk-aub.primo.exlibrisgroup.com/permalink/45KBDK_AUB/a7me0f/alma9921883085005762

ViZARTS videos. https://www.youtube.com/@thevizartscollaboratorium5421/videos