From Instrument-Mounted Demonstrations to In-Vivo Execution: Learning Bimanual Laparoscopic Appendectomy Without Robot-Collected Demonstrations

Most minimally invasive surgery is still performed with hand-held laparoscopic instruments, and the surgeon's instrument kinematics are lost when the operation ends; only the endoscope video is kept. This paper presents an end-to-end pipeline that captures this motion in the operating room and uses it to train a surgical robot policy, validated on live animals. We introduce a surgical instrument-state logger that mounts on the shaft of a standard laparoscopic instrument and recovers its pose and jaw state from an inertial sensor, a time-of-flight sensor and a Hall sensor, with no external camera or tracker. A data pipeline measures the latency of every sensor channel against a robot ground truth and aligns the channels before forming observation–action pairs. On these demonstrations we train a diffusion policy with a fine-tuned DINOv3 backbone, selecting its design by closed-loop rollouts in a physics simulator reconstructed from depth maps of an ex-vivo rabbit appendix. The policy is then retrained on 849 in-vivo demonstrations from four live rabbits and deployed on four additional live rabbits with electrosurgery armed. With the surgeon selecting the surgical phase, the policy completed the appendectomy in three of the four animals. The results show that demonstrations recorded from a surgeon's own instruments are sufficient to train, select and deploy a bimanual surgical policy in vivo. The robot serves only as the timing reference for sensor calibration and as the executor, and collects no demonstrations. Both demonstration corpora are released to support future surgical robot learning research.

The surgical instrument-state logger records the pose and jaw opening of a hand-held laparoscopic instrument during real operations, turning the surgeon's own motion into demonstration data. It mounts on the shaft outside the patient, needs no external camera or tracker, and leaves the part that enters the patient untouched. A rigid instrument through a trocar has four degrees of freedom, not six, because the remote-center-of-motion (RCM) constraint removes two. Orientation about the trocar comes from an IMU. The remaining degree of freedom, insertion depth $d=\ell -r$, is the shaft length $\ell$ from the board to the tip minus the board-to-trocar distance $r$, which a time-of-flight (ToF) sensor pointed down the shaft measures directly. Each instrument therefore carries one board (Seeed XIAO nRF52840) with a BNO085 IMU and a ToF sensor, plus a linear Hall sensor on the handle (WSH137-XPAN1, sampled at 200 Hz) that reads a magnet on the moving handle ring and so measures jaw opening.

Two generations share this principle. The ex-vivo logger (VL53L0X ToF) was powered and read over a USB-C cable and smoothed the depth with an exponential moving average. The in-vivo logger was redesigned for the operating room: a faster ToF (VL53L1X, five-sample moving average, 40 ms), a 1000 mAh lithium-polymer cell and BLE streaming at 50 to 200 Hz, so that no cable leaves the instrument, all housed in a compact enclosure. At the measured draw of about 50 mA the cell lasts roughly 20 h, beyond one operating-room day.

Against the FR3 tip pose during free motion, the latency-matched channels track shaft roll and depression to 0.42° and 0.17° RMS and insertion depth to 1.36 mm RMS once the firmware's fixed gain and offset are calibrated out, and the tip displacement over a 1.6 s action chunk to 1.38 mm RMS (2.89 mm at the 95th percentile). Video-based instrument tracking, for comparison, reaches 2.6 to 3.0 mm at best on public benchmarks and 5.4 to 12.4 mm under occlusion; on our own rig, a markerless fit of the shaft's silhouette edges located the axis only to a median 10.7 mm.

Each channel reaches the log later than the event it describes, by a different amount. A Franka FR3 drove an instrument carrying the logger through the phantom trocar along swept-sine trajectories while its tip pose, published at 200 Hz from the 1 kHz joint state, served as ground truth, and for each channel we fitted the response

where $L$ is a pure transport delay and $T$ the time constant of a first-order lag. The distinction matters: a channel with $T\approx 0$ can be corrected by a time shift, one with a large $T$ cannot, because part of the motion is gone.

On the ex-vivo logger the attitude channels showed pure transport delays of 24 to 26 ms, while insertion depth showed $L+T=349$ ms, 93% of it first-order lag from the firmware's exponential moving average, with gain falling to 0.42 at 1 Hz. The resulting 325 ms depth–attitude skew is 3.5 policy steps at 10 Hz, and no common time shift can remove it. Rewinding each channel by its own latency (−17, +307 and +37 ms for attitude, depth and jaw) cut dynamic depth error from 2.69 to 1.36 mm (−49.5%) and shaft-roll error by 28%. The depression channel, whose 26 ms latency is below what the motion resolves, changed by +2% and serves as the control.

Every recorder, the boards, the endoscope capture and a foot-pedal "energy" channel that mirrors the electrosurgical pedal, stamps each sample with the host's absolute time at the earliest receive point; streams are merged offline by that time, rewound by their own measured latency and resampled on the endoscope clock. The magnetometer heading of a low-cost IMU drifts in an operating room, so the observation uses only what gravity and the trocar define: the endoscope frame at 320 × 192 and, per arm, $[\cos \rho ,\sin \rho ,\theta ,d/d_{\max }]$. The jaw aperture is deliberately not an input; with it, the policy learns "it is closed now, so keep closing".

Actions are 16-step chunks at 10 Hz of per-arm tip-frame deltas relative to the chunk's anchor pose, plus absolute jaw aperture and, in vivo, an absolute electrosurgery channel owned by the cutting arm. Because every label is relative to its anchor, it is invariant to the heading zero and to where the trocar sits, so no correspondence between the demonstration trocar and the robot's trocar is needed.

The policy is a 1-D U-Net diffusion policy with DDIM sampling over 16-step chunks, with three changes. A single shared DINOv3 ViT-B/16 replaces the two ResNet-18 encoders, with attention pooling and only the last transformer block trainable (7.1 M of 85.7 M parameters). Because the observation window is 0.2 s, the policy cannot infer where in the procedure it is, so the U-Net is conditioned by FiLM on an ordinal encoding of the surgical phase plus two timing channels. And a chunk executor commits three actions per chunk and replans, with the index it starts from absorbing the observation anchor, the arm's measured execution latency and inference.

Rather than select the configuration by offline validation error, 31 candidates were compared by closed-loop rollouts in a simulator whose organs are reconstructed from RealSense D405 depth recordings of the ex-vivo scene: 16 episodes of 15 s each, scored on stage counts (lift, arrival at the site, jaw closed on tissue) and on one hard failure, a trocar violation above 10 mm. Because sim-to-real transfer for deformable tissue remains unreliable, the simulator is used to compare configurations under one protocol, and only the chosen one runs on the real rig. The simulator itself is described in a companion paper (see below).

Both corpora were captured with the surgical instrument-state logger alone: one unit on the shaft of each instrument streamed its channels to the recording host (over USB ex vivo, over BLE in vivo), which also captured the endoscope and the pedal channel. No marker, external tracker or calibration object entered the field, and the surgeon worked with the instruments as usual. Each episode is one attempt at the procedure. The ex-vivo corpus is 533 episodes on two rabbit cadaver specimens in a laparoscopic training phantom, recorded by two surgeons. The in-vivo corpus comes from four live rabbits in one operating-room session, trocars through the abdominal wall, with a dissector and an electrosurgical unit on the right arm; of 1,003 raw episodes, 849 survive the quality filters.

Selection criterion. A policy that does nothing never violates the trocar, so violations alone cannot select a configuration; GR00T-H and the frozen backbone score well on it for that reason. The criterion was therefore the full approach without a violation (lift, arrive at the site and close the jaw on tissue), the furthest stage the simulator can score, reached with few violations. On this criterion the DINOv3 configuration with its last transformer block unfrozen was the best of the 31 (81%), ahead of two unfrozen blocks (50%) and of the fully frozen backbone (44%), whose low violation rate comes at the price of never closing the jaw on tissue (grasp depth 0.0 mm). GR00T-H (0%, none of 24 checkpoints closed a jaw on tissue) and ResNet-18 (≤12%) are far behind.

Backbone. Replacing the two ResNet-18 encoders with a frozen DINOv3 reduced trocar violations from 88% to 6% (Fisher p = 7 × 10⁻⁶) and raised lift from 12% to 69% (p = 0.003).

Latency matching. Training the deployed configuration on the same episodes processed without per-channel latency matching leaves trocar violations unchanged (19% vs. 19%) but eliminates successful lifts (19% vs. 81%, p = 0.001), while its validation mean absolute action error (MAE) is better (0.138 vs. 0.148): the 325 ms depth–attitude skew breaks the coordinated insertion that lifting requires, which offline error cannot see.

Unfreezing. Safety degrades as more transformer blocks are unfrozen (violations 6%, 19%, 19%, 56%, 69% for zero to four blocks), while only unfreezing a single block raises the full-approach rate notably above the frozen backbone (81% vs. 44%).

Offline error vs. closed-loop safety. That offline validation error does not predict closed-loop success is known from robomimic; in our surgical setting it goes further and is anti-correlated with safety. Across the unfreezing sweep, validation MAE improves monotonically (0.152 to 0.142) while violations rise from 6% to 69%. The least safe encoder, ResNet-18, even has a better MAE (0.145) than the frozen DINOv3 (0.152).

Run-time phase input and replanning. A phase predictor trained on demonstration video cut the deployed configuration's lift from 81% to 31% and its full approach from 81% to 25%; one trained in the simulator's own domain recovered part of this loss but still halved lift. Committing one action per chunk instead of three raised violations from 19% to 69% and cut the full approach from 81% to 31%.

Real ex-vivo rig. The deployed configuration ran on the two-FR3 rig with a rabbit cadaver (15 sessions, about 18 min of policy control): zero trocar violations, peak RCM error 0.23 mm. The phase predictor, however, saturated at the final phase within a median of 10.6 s (rig frames lay outside its training manifold), so in vivo the surgeon set the phase.

The policy was retrained from scratch on the 849 in-vivo demonstrations, with the ex-vivo configuration and optimizer-step budget unchanged. Because offline error had already failed as a selector, the checkpoint was accepted through pre-registered event-level gates on unseen episodes: every jaw-closing event detected, energy false-fire 0.019 and missed-fire 0.032 against gates of 0.05 and 0.30, and closed-loop gain 0.86/0.82 against a 0.8 gate.

Ex-vivo pre-training turned out not to matter. Trained from random initialization and from the ex-vivo weights on 25%, 50% and 100% of the corpus, both initializations passed the same gates in all six runs; pre-training helped in offline error only while data were scarce (MAE 1.221 vs. 1.413 at 25% of the corpus, 1.157 vs. 1.173 at 100%). A policy for live tissue has to be trained on live-tissue demonstrations, which is the argument for an instrument that can collect them in the operating room.

On a single day the policy ran on four live rabbits at 5 Hz with electrosurgery armed. Inference on one RTX 4090 took a median 23.5 ms per step (95th percentile 59.7 ms), well inside the 200 ms control period. In each animal the surgeon placed the instruments, checked the hand-over state against the demonstration band, handed control to the policy, supplied the phase by keypress, and could take control back at any moment.

In three of the four animals a single rollout carried the instruments from grasp approach to the energized cut without the surgeon touching them. In the fourth the left arm was handed over in a poorly conditioned inverse-kinematics branch and the joint-limit governor aborted before the policy could act; the procedure was completed by hand. All four animals survived, with no gross abnormality during a week of observation.

Safety. Across all policy-controlled time that day the RCM error never exceeded 0.84 mm against a 5 mm abort threshold, and every sample that did exceed the threshold in the day's logs occurred under human teleoperation. In every sample of cautery under policy control (108 samples at 10 Hz) the cutting jaw was closed, with aperture at most 0.7%.

A shaft-mounted surgical instrument-state logger with per-channel latency matching turned hand-held laparoscopic demonstrations into a bimanual policy that was selected in simulation, validated on a real rig and executed electrosurgery in live animals. Per-channel latency matching halved the dynamic depth error against the FR3 ground truth (2.69 to 1.36 mm) and raised lift in closed loop from 19% to 81% of episodes. Closed-loop rollouts of 31 candidate configurations identified the policy design: a frozen DINOv3 backbone reduced trocar violations from 88% to 6% relative to ResNet-18, unfreezing its last block gave the highest full-approach rate (81%), validation error improved as safety degraded, and the selected configuration ran on the real rig without a trocar violation. Retrained on 849 in-vivo demonstrations, the configuration completed the appendectomy in three of four live rabbits under shared autonomy, with the cutting jaw closed at every cautery sample and a peak RCM error of 0.84 mm.

This study has two limitations. First, the surgeon supplied the phase in vivo, because the vision-based phase predictor agreed with the surgeon on only 24.6% of policy control steps under the deployment endoscope, a domain shift known from phase recognition across hospitals. Second, due to the complexity of experiments on live animals, the in-vivo study was limited to demonstrations from four rabbits and one deployment in each of four others on a single day.

Eight live rabbits were used in total, four for demonstrations and four for deployment. All animal procedures involving rabbits were approved by the BIOSTEP Institutional Animal Care and Use Committee (IACUC S26-BZ-0562).

Both training corpora are released in their recorded form, not as derived training arrays. Every stream carries absolute host timestamps (Unix epoch, UTC) so that the per-channel latency matching of the paper can be reproduced or replaced. File layouts, channel definitions, label vocabularies and known issues are documented in the data repository README.