Human-Centered AI for Immersive XR Environments: A Multisensor Fusion Approach for Adaptive Interaction and Cognitive Modeling
DOI:
https://doi.org/10.35335/cit.Vol17.2025.1397.pp219-229Keywords:
Human-Centered Artificial Intelligence, Multisensor Fusion, Cognitive State Modeling, Adaptive XR Interaction, Extended Reality (XR) SystemsAbstract
Immersive Extended Reality (XR) systems are rapidly expanding across education, training, healthcare, and industrial applications, yet most existing frameworks lack real-time adaptivity and personalized support based on users’ cognitive and emotional states. This research proposes a human-centered AI framework that integrates multisensor fusion with cognitive state modeling to enable adaptive and intelligent interaction within XR environments. The system combines data from eye tracking, body and hand motion capture, environmental sensors, audio input, and physiological signals such as EEG, EMG, and HRV. A hierarchical fusion engine performs low-, mid-, and high-level integration of multimodal signals, while deep learning models including CNNs, LSTMs, and multimodal transformers estimate user states related to attention, workload, fatigue, and emotion. The framework dynamically adapts the XR environment through real-time modifications to UI complexity, lighting, haptic feedback, content pacing, and virtual assistant behavior. Experimental results demonstrate substantial improvements in cognitive load prediction accuracy, interaction robustness, and user immersion compared to single-sensor or static XR systems. Users experienced reduced cognitive overload, enhanced task performance, and greater engagement across various simulated tasks. Overall, this research advances human-centered AI by demonstrating how multisensor fusion and cognitive modeling can transform XR from passive simulation platforms into adaptive, perceptive, and user-responsive environments. The findings offer a foundation for next-generation XR systems that prioritize human well-being, performance, and comfort through continuous AI-driven personalization.
Downloads
References
G. Margetis, G. Papagiannakis, and C. Stephanidis, “Realistic natural interaction with virtual statues in x-reality environments,” Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci., vol. 42, pp. 801–808, 2019.
H. Nguyen and T. Bednarz, “User experience in collaborative extended reality: overview study,” in International Conference on Virtual Reality and Augmented Reality, Springer, 2020, pp. 41–70.
A. Çöltekin et al., “Extended reality in spatial sciences: A review of research challenges and future directions,” ISPRS Int. J. Geo-Information, vol. 9, no. 7, p. 439, 2020.
F. Chen et al., “Multimodal behavior and interaction as indicators of cognitive load,” ACM Trans. Interact. Intell. Syst., vol. 2, no. 4, pp. 1–36, 2013.
S. Agarwal et al., “Unleashing the power of disruptive and emerging technologies amid COVID-19: A detailed review,” arXiv Prepr. arXiv2005.11507, 2020.
N. Aliman, “Hybrid cognitive-affective strategies for AI safety,” 2020.
A. L. Gardony, R. W. Lindeman, and T. T. Brunyé, “Eye-tracking for human-centered mixed reality: promises and challenges,” in Optical architectures for displays and sensing in augmented, virtual, and mixed reality (AR, VR, MR), SPIE, 2020, pp. 230–247.
N. Sawhney and C. Schmandt, “Nomadic radio: speech and audio interaction for contextual messaging in nomadic environments,” ACM Trans. Comput. Interact., vol. 7, no. 3, pp. 353–383, 2000.
R. C. Luo, C.-C. Yih, and K. L. Su, “Multisensor fusion and integration: approaches, applications, and future research directions,” IEEE Sens. J., vol. 2, no. 2, pp. 107–119, 2002.
G. Matthews, L. Reinerman-Jones, J. Abich IV, and A. Kustubayeva, “Metrics for individual differences in EEG response to cognitive workload: Optimizing performance prediction,” Pers. Individ. Dif., vol. 118, pp. 22–28, 2017.
X. Fu, H. Wang, Z. Wang, Z. Shi, W. Yang, and P. Ma, “Research on micro-grid group intelligent decision mechanism under the mode of block-chain and multi-agent fusion,” Energies, vol. 12, no. 21, p. 4196, 2019.
M. Kristiansson, “Memory, aging and external memory aids: Two traditions of cognitive research and their implications for a successful development of memory augmentation.” 2011.
M. G. Mendonça and V. R. Basili, “Validation of an approach for improving existing measurement frameworks,” IEEE Trans. Softw. Eng., vol. 26, no. 6, pp. 484–499, 2002.
Z. Wang, Y. Wu, and Q. Niu, “Multi-sensor fusion in automated driving: A survey,” Ieee Access, vol. 8, pp. 2847–2868, 2019.
A. Korbach, R. Brünken, and B. Park, “Differentiating different types of cognitive load: A comparison of different measures,” Educ. Psychol. Rev., vol. 30, no. 2, pp. 503–529, 2018.
M. L. Rahman, Towards Improving Cybersecurity and Augmenting Human Training Performance Using Brain Imaging Techniques. University of California, Riverside, 2020.
E. P. Blasch and S. Plano, “JDL Level 5 fusion model: user refinement issues and applications in group tracking,” in Signal processing, sensor fusion, and target recognition XI, SPIE, 2002, pp. 270–279.
E. Volta, “Multisensory learning in adaptive interactive systems,” 2020.
L. S. Kao and E. J. Thomas, “Navigating towards improved surgical safety using aviation-based strategies,” J. Surg. Res., vol. 145, no. 2, pp. 327–335, 2008.
J. Ellis and A. Zaretsky, “Assessment and management of posttraumatic stress disorder,” Contin. Lifelong Learn. Neurol., vol. 24, no. 3, pp. 873–892, 2018.
D. Lahat, T. Adali, and C. Jutten, “Multimodal data fusion: an overview of methods, challenges, and prospects,” Proc. IEEE, vol. 103, no. 9, pp. 1449–1477, 2015.
A. K. Goel and M. E. Helms, “Theories, models, programs, and tools of design: views from artificial intelligence, cognitive science, and human-centered computing,” in An Anthology of Theories and Models of Design: Philosophy, Approaches and Empirical Explorations, Springer, 2014, pp. 417–432.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Arvando L. Meskir, Talira N. Juvens, Junelle Vorsteyn
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
