Multimodal Data Integration and User Interaction for Avatar Simulation in Augmented Reality

Introduction

Simulating avatars in an Augmented Reality (AR) environment becomes an important step for the users to attend to the virtual world. A realistic, vivid, and friendly avatar helps users quickly integrate into the virtual world (Feng et al., 2014). Existing algorithms have already simulated the voice and movement of an avatar and made it appear as a real person in the virtual world (Yang & Bhanu, 2011). Combined with AR technology, avatar simulation has the potential to evolve traditional human-computer interaction in 2D space to 3D mixed reality interaction in realistic environments (Pouyanfar, Yang, et al., 2018). In addition, services such as intelligent voice assistants and intelligent butlers are integrated to further enhance avatar-based interaction in the AR environment. With the development of avatar simulation technology in the AR environment and deep learning, more and more industries begin to take notice of this technology and apply it in the fields of entertainment, education, and healthcare (Pouyanfar, Sadiq, et al., 2018).

Childcare has always been a crucial part of the field of healthcare. Taking care of the children in the early childhood is extremely challenging since they are not yet able to express their needs with words, and their emotions can be intense (Österman & Björkqvist, 2010). Teachers, caregivers, parents, and other groups who have long-term contact with children need the training to manage children's tantrums. Good child tantrum management skills can also help improve teacher satisfaction (Schaack et al., 2020). Teachers in special education schools need to develop their skills to manage children's emotions and attend to tantrums when dealing with children with disabilities (Konst et al., 2013). Due to the environment and time, it is difficult for people to receive targeted professional training in the real world. Therefore, the introduction of avatar simulation combined with AR technology into pediatrics is one of the possible ways to address such childcare challenges.

Confronting the tantrum behaviors that result from early childhood children in real-life situations is the best training for teachers, caregivers, parents, and others who come into direct contact with children (Eisbach et al., 2014). However, it is impossible to create such a training environment in the reality nor provide a tantrum child to train teachers, caregivers, and parents (Belden et al., 2008). In the past, they have gradually learned from their experience in practice and trained newcomers through textual abstraction. However, introducing virtual avatar simulation in the real-world environment to simulate the child's behavior and provide interactive training for teachers, caregivers, and parents through AR can fundamentally revolutionize the existing childcare training methods, significantly improving the training effect.

In recent years, although virtual avatars have become widely available in education (Davis, 2019), healthcare (Foronda et al., 2020), and entertainment (Rusdorf et al., 2007), existing applications and frameworks have apparent limitations. Many nursing and educational applications simply place a fixed-process avatar in a virtual environment that cannot interact with the user in real-time (Kidd et al., 2012). Even for applications that interact with users in real-time, they mostly use controllers instead of realistic human interaction such as eyes, body movements, and voices (Albright et al., 2012).