Deep-Learning Enhanced Augmented Reality for Robotic Radical Prostatectomy on Consumer Grade Hardware.

Authors

Vincent Benard, Olivier Windisch, Mina Bjelogrlic, Hugues Turbé, Christian Lovis, Michele Diana, Massimo Valerio

Published in

Surgical innovation. Pages 15533506261504154. Nov 06, 2026. Epub Nov 06, 2026.

Abstract

BackgroundAugmented Reality (AR) during Robotic-Assisted Radical Prostatectomy (RARP) provides surgical guidance by overlaying patient-specific 3D Virtual Models (VM) onto the intraoperative view. A main drawback is instrument occlusion, where VM overlaps the surgical tools, disturbing the augmented view. Deep learning (DL)-based desocclusion addresses this issue, but current implementation rely on expensive, specialized hardware.MethodsWe developed an AR desocclusion pipeline running on consumer-grade hardware (∼1150 US Dollars). For model selection, we trained and benchmarked 21 encoder architectures paired with a U-Net decoder. The selected model was integrated into a macOS application using CoreML inference on the Apple Neural Engine. Memory usage, thermal state and framerate were monitored on a continuous 81-minute console session. Photon-to-photon latency (from surgical field to the surgeon eye) was measured on the DaVinci® Xi using a microcontroller-phototransistor setup.ResultsFastViT-S12/Unet offered the highest accuracy among models meeting our inference-time budget (∼2 ms inference), achieving a Dice score of 95.1% (95% CI, 94.3-95.7) internally, 91.6% (95% CI, 90.2-92.9) and 90.2% (95% CI, 84.7-94.4) on externals tests, with a mean inference time of 1.89 ± 0.36 ms. The pipeline maintained 57.98 frames per second using 1.47 GB of unified memory with a nominal thermal state. On the surgical console, photon-to-photon median latency increased from 60 ms (Q1-Q3: 57.5-62.5) to 120.5 ms (Q1-Q3: 111-131) with the AR pipeline.ConclusionsReal-time AR-RARP with DL-based desocclusion is achievable on low-cost hardware with an added latency remaining within ranges reported as technically acceptable for intraoperative use.

PMID:
42990074
Bibliographic data and abstract were imported from PubMed on 06 Nov 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.