Curie's Jarvis: Autonomous Robotic Chemistry Experiments from Printed Protocols

Rumi Loghmani, Bozhang Song, Victor Alvarado, Aliyah Redding, Amir Barati Farimani

November 2026

TL;DR A Franka arm runs a chemistry experiment from a photo of the printed instruction sheet, on an unmarked bench where identical containers are told apart by reading their paper labels in four camera views. It completed 9 of 10 runs of two kids' chemistry-kit protocols with no human intervention and identified the right container in 160 of 160 locates.

What they do

Lab automation usually adapts the experiment to the robot: scripted procedures, fixed container positions, fiducial markers. Curie’s Jarvis goes the other way. A Franka Panda with four RealSense D415 cameras gets a photo of a printed instruction sheet and a list of the objects on the bench, then scoops, pours, stirs and pipettes its way through the protocol, finding each container again before every step.

How it works

Sheet to skill calls. GPT-4.1 parses the photo into goal, steps and reagents; a planner orders sub-tasks and replans up to three times after a failure; a skill-call agent emits one JSON-constrained call per sub-task, including a speed chosen from what is held.
Locating. SAM 3 segments each view from a text prompt. Opaque objects are placed from depth; transparent cups from intersecting camera rays, since their depth centroids disagreed by up to 6 cm.
Labels. Identical containers stand on paper slips. The VLM reads the label in every view and the cameras vote; a container whose label is unread can be found by elimination.
Skills. Pick, scoop, dump, pour, stir and pipette plan from measured geometry and check gripper width and reached poses to detect failures.

Results

End to end: Magic Beaker succeeded in 5/5 runs (about 30 sub-tasks, about 30 minutes each), jellyfish in 4/5. The failure was the scoop being grasped instead of the unlabelled pipette.
Label reading: 160/160 locates found the right container across printed labels and three people’s handwriting, with at most 1.1 mm offset. One camera misread in 21 locates and was outvoted every time.
Containers moved off their labels: 48/48 found with the other labels on the bench, 32/48 with them removed; no locate returned the wrong container.

Limitations

Reagents come pre-measured and the robot does not measure amounts. The evaluation covers one bench, one arm and two protocols from a children’s kit. Re-locating after a person moves objects and the adapted speeds were not tested on their own. A four-camera locate takes 14.1 s on average, and the system depends on a hosted VLM.

Figures

From Figure 1: Curie’s Jarvis workflow. (A) A vision-language model reads the protocol photo and the LLM orders subta... — From Figure 1: Curie’s Jarvis workflow. (A) A vision-language model reads the protocol photo and the LLM orders subtasks. (B) Votes across four camera views identify vessels from paper labels. (C) Multi-view geometry locates the target vessel. (D) The LLM selects predefined skills and their parameters for the Franka Panda.

From Figure 2: System architecture. Once per run, the instruction parser reads the steps from a photo of the sheet, t... — From Figure 2: System architecture. Once per run, the instruction parser reads the steps from a photo of the sheet, the scene agent names the objects on the bench, and the planner orders the sub-tasks. For each sub-task, the skill-call agent produces one skill call, and the skill locates its target again before it acts.

From Figure 3: One locate on the real bench. (a) SAM 3 masks in each of the four camera views. (b) The label read on ... — From Figure 3: One locate on the real bench. (a) SAM 3 masks in each of the four camera views. (b) The label read on the container in each view and the vote across cameras. (c) The container’s position from multi-view geometry.

Abstract

A person running a bench experiment works from a printed instruction sheet, tells identical containers apart by the labels written on them, and looks at the bench again before each step. Lab automation usually works the other way around: the procedure is written as a script, and containers stand at fixed positions or carry fiducial markers. We present Curie's Jarvis, a system that lets a robot arm carry out a chemistry experiment on a bench set up for people. It reads the steps from a photo of the printed sheet, finds containers on an ordinary bench with no fixed positions or markers, tells identical-looking containers apart by reading their labels in four camera views, locates its target again before every step, and sets how each step is carried out, including its speed, from what is being handled. On a Franka Panda arm, Curie's Jarvis completed 5 of 5 runs of one protocol from a children's chemistry kit and 4 of 5 runs of another, with no human intervention, and in 160 locates on printed and handwritten labels it identified every container correctly. This brings robots a step closer to doing bench chemistry the way people do.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong