Due to the haptic experience in the virtual environment, MED1stMR decided for a mixed reality (MR) as training approach to create a better learning experience for medical first responders.
MED1stMR project goals
Our main development in the project is a mixed reality solution with large movement, two high-fidelity simulation manikins and a biosignals measurement suite including a trainer dashboard to support the trainer in the adaption of the training flow in accordance to the stress level of the trainees executing mass casualty incidents in training. This solution is built to be exploitable after the end of the project.
Research beyond project scope
But MED1stMR is also doing a lot of research to explore the wide universe of mixed reality for training options. One approach is the AIT-developed “green manikin”. We started with a simple torso simulation manikin that used chroma-keying enabling a mix of real tools and hands in the virtual environment. The patient is haptic but at the same time overlayed with different injuries and animations as well as the virtual background of additional (screaming) avatars, the damaged car and other environmental effects. Using real tools and the trainee’s own hand is a major advantage of this prototype.
First results
The first studies were conducted with the green manikin, which proposes a novel approach combining a physical manikin with a virtual avatar, allowing trainees to interact with both real and virtual elements by using the Varjo XR-3 mixed reality headset. In the designed scenario, trainees deal with a car accident involving multiple injured people. The research team measured the participants’ levels of presence, stress, and technology acceptance, as well as their feedback on the MR system and the training scenario. The results showed that the participants experienced high levels of physical- and self-presence, increased stress levels, and high technology acceptance during the simulation. The MR approach was well received, and they particularly appreciated the haptic and multi-sensory feedback and the realism and complexity of the training scenario. The study concluded that MR could be seen as a viable technological solution to create immersive skill training, perceived by end-users as a useful tool and addition to existing training solutions.
Future of simulation in MR
In the course of the project, the manikin was exchanged for a full-body simulation manikin and enhanced with full-body tracking and a totally new approach: communication via speech recognition and an LLM for suitable answers. This gives first insights on how the lack of communication with patients, especially the response from patients as a clear indication of the mental state on injuries can be solved in the future of simulation training with virtual environments.
At the end of the project even more results will be available as the green manikin is also touring Europe to some of the field trial locations in order to gather more data on future training options in the field of mixed reality training for medical first responders.
