VLM-guided Grounding Expert for Generalised Referring Expression Comprehension

TL;DR VGE couples a frozen Qwen3.5-9B vision-language model with a trainable Grounding DINO detector, so a robot can find zero, one or many objects named in a reworded or chatty request. On the robotic-scene RoboRefIt benchmark it keeps 95.2% F1 when context is added to the instruction (Grounding DINO falls to 62.6%) and runs 5.3x faster than a LoRA-tuned Qwen3.5-9B.
What they do
A robot told to “find all red cups” must return every matching object, or none if there is none, and give the same answer when the request is reworded or buried in conversation. The authors find that grounding detectors (Grounding DINO, LLMDet) lose accuracy when context is added, while VLMs (Qwen3.5) are stable across wording but often fail to reject absent targets. VGE combines the two: a frozen Qwen3.5-9B guides a trainable Grounding DINO-base, which still predicts boxes directly.
How it works
- Layerwise joint attention. The detector’s six decoder layers are paired with the VLM’s last six full-attention layers. Projected VLM keys and values join the detector’s own in a single softmax, behind a zero-initialised gate per layer.
- Text-conditioned instance scoring. Each final object query attends to the VLM’s text features, and a zero-initialised linear layer corrects its matching score. Thresholding and NMS on these scores give zero, one or several boxes.
- Training. The VLM stays frozen; the detector and both interfaces train on original instructions only. Testing adds LLM-generated paraphrases and context-augmented versions of every request.
Results
- Highest F1 under added context on all five test splits, varying by at most 1.2 points across instruction types within a split. RoboRefIt Test A with context: 95.2% vs. 62.6% for Grounding DINO and 92.5% for LoRA-tuned Qwen3.5-9B.
- On RoboRefIt multi-target requests: 96.0% and 81.3% F1 (Test A/B) vs. 90.7% and 76.3% for LoRA-tuned Qwen3.5-9B.
- Mean latency 140-145 ms, 5.3-13.0x faster than LoRA-tuned Qwen3.5-9B but about 2.1x Grounding DINO’s.
- Ablation on RoboRefIt: both interfaces together beat scoring alone by about 11.4 points; joint attention alone lowers F1 there.
Limitations
Evaluation is on image datasets only, with no robot experiments. Keeping the VLM frozen costs accuracy on general scenes: adapting it adds 4.62 points on gRefCOCO Test B. There, and on Ref-ZOM, the LoRA-tuned VLM is better on original instructions and at rejecting absent targets. Each configuration was trained once, and code is not yet released.
Figures



Abstract
Generalised referring expression comprehension (GREC) aims to identify and localise the zero, one or multiple objects referred to by a language expression. In robotic manipulation, grounding also needs to remain accurate when requests for the same objects are paraphrased or embedded in additional context in robotics. However, grounding expert models are sensitive to instruction variation, while generative vision--language models (VLMs) struggle with complete target recovery and rejection of absent targets. These complementary limitations motivate combining the contextual representations of VLMs with the direct spatial prediction capabilities of grounding experts. We therefore introduce VLM-guided Grounding Expert (VGE), which combines a frozen VLM with a trainable grounding expert through layerwise joint attention and text-conditioned instance scoring. These interfaces leverage multimodal understanding of the VLM to guide object-query refinement and candidate selection while retaining direct bounding-box prediction. Across three datasets, VGE achieves the highest averaged F1 score among the evaluated methods on all five test splits, with differences of at most 1.2 percentage points across instruction conditions within each split. On the dataset with robotic scenes, VGE also achieves the highest F1 across instruction variants and no-target, single-target and multiple-target cases. VGE provides a $5.3$--$13.0\times$ inference speedup over the baseline with the same VLM base model. These results support combining VLM context with specialised spatial prediction to address both grounding challenges, with particularly strong performance in robotic scenes. Code will be released upon publication.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
