Multimodal mixed reality visualisation for intraoperative surgical guidance

A Hamlyn-led project that introduced a novel MR visualisation platform for intra-operative surgical guidance has won the IPCAI 2020 Audience Award.

Mixed reality (MR) is emerging as a vital tool in surgery as it enables the surgeon to visualise subsurface anatomical structures in 3D. In the last decade, there has been a great effort to bring MR into the operating room to assist surgeons intra-operatively.

The virtual objects, showing a patient’s data, can be consulted by a surgeon for decision-making and can also be moved around the operating room using mid-air hand gestures. Therefore, MR allows a surgeon to consult data when and where needed, making it a valuable tool for surgery.

Regardless of numerous research efforts contributed to MR technology development for clinical applications, the progress of this type of development is still at an early stage.

Multimodal MR visualisation platform for the Microsoft HoloLens

Collaborated with Department of Surgery and Cancer, our researchers at the Hamlyn Centre proposed a MR visualisation platform for the Microsoft HoloLens, aiming to assist surgeons during the surgery in the operating room.

The platform contains three visualisation components, namely a 3D organ model, volumetric data, and tissue morphology captured with intra-operative imaging modalities. Furthermore, a set of novel interactive functionalities have been designed including scrolling through volumetric data and adjustment of the virtual objects’ transparency.

Our research team invited a group of surgeons to participate in this project. The participants were allowed to interact with the visualisation components and test the different functionalities. 

The majority of the surgeons found the proposed visualisation platform intuitive and would use it in their operating rooms for intra-operative surgical guidance.

This pilot study verified the potential of the proposed visualisation platform and its usability in the operating room. Our research team also received the IPCAI 2020 Audience Award on Wednesday 24^th June at the 11^th International Conference on Information Processing in Computer Assisted Interventions.

The future work our research team plans to carry out is to focus on enhancing the platform by incorporating the surgeons’ suggestions and conducting extensive evaluation on a large group of surgeons.

FIND OUT MORE >>

This research was supported by NIHR Imperial BRC (Biomedical Research Centre) and the Royal Society (UF140290). It was published in International Journal of Computer Assisted Radiology and Surgery in April 2020 (João Cartucho, David Shapira, Hutan Ashrafian and Stamatia Giannarou, "Multimodal mixed reality visualisation for intraoperative surgical guidance", International Journal of Computer Assisted Radiology and Surgery, 15: 819–826, 2020).