The EUREKA partners BHS Technologies GmbH and HYVE Innovate GmbH have developed a state-of-the-art training simulator for the RoboticScope, providing surgeons with a comprehensive virtual training environment for the entire workflow in the operating room. This groundbreaking solution eliminates the need for physical devices during training, offering a realistic and interactive experience.
I led the project from writing the grant application to heading the development team.
The simulator leverages advanced Virtual Reality (VR) to create a precise digital twin of the RoboticScope. At its core is the simulation of the robotically controlled stereoscopic camera, supporting various operational modes such as Freeview for unrestricted camera movement, Orbit for rotation around a focus point, Zoom with optical and digital zoom levels, adjustable working distances with refocusing capabilities, and customizable light intensity via LEDs and display settings. A scalable software framework was also developed, enabling this architecture to be adapted for future simulators and other use cases.
The technical implementation focused on accuracy and efficiency. Optical modeling was based on a lookup table (LUT) to replicate parameters like focal length, aperture, and stereoscopic angles, later optimized with mathematical functions for real-time performance. A kinematic model of the RoboticScope was developed to represent the motion space accurately, incorporating mechanical constraints and precise camera positioning through iterative calculations.
To enhance user immersion, the simulator employs a head-mounted display (HMD) and a haptic interface. The HMD displays a detailed view of the simulated surgical environment while capturing head movements to intuitively control the camera. The haptic interface, featuring a stylus with six degrees of freedom, provides tactile feedback to simulate the real-world manipulation of surgical tools.
The training experience is enriched by an engaging and interactive learning design. Users progress through step-by-step modules to master fundamental functions such as camera control, zoom, and lighting adjustments. Animated guides and real-time feedback support each lesson. Gamification elements, such as tasks to collect virtual blue gemstones, enhance motivation and reinforce hand-eye coordination while building familiarity with system features. A final skill assessment evaluates the user’s proficiency based on metrics like efficiency, precision, and task completion time.
The simulator hardware is compactly integrated into a portable box, facilitating easy transport and setup. Key components, including the HMD, haptic interface, and foot pedal, communicate through a central software hub built on the Unity engine. A custom .NET interface ensures seamless data exchange and smooth operation across all connected devices.
📅 2021-2023
💼 HYVE – The Innovation Company
🧔 Felix Laarmann, Evgenija Pavlova, Simon Liao, Philipp Geuder, Gerd Schwarz, Andreas Beer
