This project aims to develop a robotic scrub nurse that can assist surgical teams by responding to spoken requests and handling instruments. We have developed an application in the Juliet programming language that lets users request instruments by voice. The robot detects instruments using 6D pose estimation and learns how to hand them over using visuomotor policies, vision-language-action models, and planning methods.

The project also investigates how surgical scenes can be understood from video. We track hands and instruments, detect and track segmented tissue in heart surgery videos, and have demonstrated detection of knot tying. These capabilities support the analysis of surgical activity and could help robots recognize relevant objects, anatomy, and actions over time.

By combining voice interaction, visual perception, and learned robot behavior, we are exploring practical approaches to human-robot collaboration in the operating room. The focus is on developing and evaluating these capabilities as steps toward useful robotic assistance for surgical teams.