AR, VR, and 3D Printing Bridge the Gap between Radiology and Surgery at UCSF

In this blog, we sit down with Dr. Jesse Courtier, Pediatric Radiologist, who specializes in utilizing augmented reality for research and patient care at UCSF.

At UCSF, we are a referral center for very complex surgeries across a number of subspecialties. As such, the degree of surgical case planning required is especially high. Medical imaging plays a critical role in this planning. We have found that 3D imaging provides surgeons with important additional conceptual information for planning these complex cases that is beyond what is achievable with simple 3D reconstruction on 2D displays. Personally, as a Pediatric Radiologist, the ability to convey complex information from the CTs and MRIs in a way that is clear and understandable to my surgical colleagues is critical. 

3D technologies such as 3D printing and AR, which is my area of focus, provide a powerful tool to help bridge the gap between the 2D world of radiology and the 3D real world of surgery. Further, from the surgical standpoint, finding methods to reduce the overall cognitive load encountered during these complex cases through sophisticated planning is an additional welcome benefit.

While I see applications for AR across a number of fields, we have found certain subspecialties that are especially amenable to planning with AR. Specialties such as orthopaedics, cardiothoracic surgery, interventional cardiology, liver transplant, and pediatric surgery are among the fields which deal with challenges of high visual-spatial complexity and extensive anatomic variation in their cases. I have made more than 80 AR models at UCSF across a number of subspecialties. Some specific examples include complex pediatric elbow fractures, deformity repairs, large liver tumors, and a very complex congenital cardiac case. 

We are also researching mobile AR technology applications in medical student anatomy small group sessions. In addition, we are also beginning to investigate AR in patient education settings in congenital heart disease. We are excited about exploring the potential for improved patient compliance and overall lower anxiety with an improved understanding of their conditions.

I believe AR complements 3D printing by allowing for rapid prototyping and iterative model improvement in an economical and environmentally friendly way. Models can be repeatedly tested and viewed at full anatomical scale without regard to some of the constraints of physical models (gravity, thickness, printer size, cost). It also allows for remote collaboration through the sharing of 3D models with colleagues without the need for physical modeling.

I initially began my journey into the 3D world using freeware and off-the-shelf applications. However, I became increasingly aware of the need for more sophisticated software applications as I went further along in creating multiple model types at UCSF. I needed software that was robust, included a variety of features, and had attention to user experience and user interface design. 

At UCSF, through our Center for Advanced 3D Imaging + of which I am a Co-Director, we purchased the Materialise Mimics Innovation Suite, and I have been very impressed with the features and capabilities. Currently, as a final step to optimize for my specific AR app for the Microsoft HoloLens, which we named ‘Radha’ (or Radiology with Holographic Augmentation), I use Blender to add AR-optimized colors, shading effects, and polygon count.

A current project we’re working on with my colleagues in the ZSFGH Orthopedic Trauma Institute is an assessment of 3D AR models to improve preoperative classification of complex acetabular fractures. We hypothesize these models will allow for a lower degree of inter-observer variability in fracture classification, which in turn guides appropriate clinical management in these cases. We aim to investigate the ability of 3D AR models created with the Mimics software’s thickness analysis tool to determine optimal placement of fixation devices based on the bone thickness.

One particular case was a patient with a very complex congenital cardiac and abdominal malformation that required extensive chest and abdominal wall reconstructive repair. This case was planned between surgeons in pediatric surgery, pediatric cardiothoracic surgery, and plastic surgery. I segmented the case and created a full-scale holographic model including skin surface, bone, heart, lungs, airway, solid organs, and bowel. This model would have been extremely costly and challenging to display with any other method. I reviewed the model with my surgical colleagues in our surgical planning conference prior to the procedure, and it was a success!

I believe continued evidence building on the impact of augmented reality on clinical outcomes such as time under anesthesia, overall operating room time, and time under fluoroscopy will be critical for its widespread adoption. Continued innovation both in AR hardware and software will also be necessary to enable larger, more complex models that are not simply static representations, but animated models that reflect physiologic movements (breathing, heart rate, etc.). This will truly allow for realistic simulation and conceptualization for pre-surgical planning and more.