Three Ways to Miss a Grasp: A Same-Metric Comparison of Rule-Based, Learned, and Zero-Shot Vision-Language Prediction

Introduction

Grasp prediction sits early in a manipulation pipeline: a wrong contact point, angle, or jaw width can defeat an otherwise capable robot arm. Three families of approach embody different bets about where useful intelligence should live—explicit rules, task-specific learned weights, or a large model’s general knowledge. Cross-family comparisons under identical outputs and scoring are uncommon: published work usually benchmarks within one family; rule-based baselines are often described rather than scored; vision-language models usually sit inside larger systems where their contribution cannot be isolated.

This project places all three on the same Cornell test images, producing grasp rectangles scored by one shared path. Grasps follow the rectangle representation of Jiang et al.; matching follows Lenz et al. (angle within 30°, IoU above 25%). An external answer key avoids circular grading against self-written ground truth. The paper does not claim state-of-the-art detection accuracy; it measures how three technique families fail under conditions held as close to identical as the design allows.

Methods

Object identity for train/validation/test was reconstructed because Cornell provides no object-identity labels and no object may appear in more than one split. The resulting split covered 234 objects across 883 images: 164/620 train, 35/140 validation, and 35/123 test.

System A (rule-based) uses a COCO-pretrained detector plus a fixed category-to-grasp table committed before evaluation code existed, with a geometric fallback for objects the detector cannot name. Nothing is learned from grasp labels. Orientation is limited to 0° or 90°. After an initial 40.7% score, a post-hoc geometric guard against background clutter raised accuracy to 57.7%; both figures are retained because the decision to add the guard was prompted by inspecting test behavior.

System B regresses a grasp rectangle from RGB images. Orientation is encoded as sine and cosine of twice the angle (as in Redmon and Angelova and Morrison et al.). Three architectures trained on the same split: a ~1M-parameter network from scratch, fine-tuned ResNet18, and fine-tuned ResNet34. Loss used the best-matching labeled grasp per image. ResNet18 was selected on validation and is the headline learned result.

System C prompts GPT-4o zero-shot via OpenRouter with the raw 640×480 image and a prompt frozen after development on 30 training images. The model returned two fingertip points and jaw width as JSON; rectangle corners and angle were derived by the same frozen function used for ground truth. Each image was prompted five independent times; parseable replies arrived on 614 of 615 calls.

Results

On the 123-image test split, System A achieved 57.7% (71/123; pre-guard 40.7%), System B (ResNet18) 79.7% (98/123), and System C 12.4% (76/615 calls). System C’s best-of-five ceiling was 35.0% (43/123). Object-clustered 95% intervals were System A [46.2, 68.2], System B [70.8, 87.1], System C [8.0, 17.4], and best-of-five [24.6, 45.5]. McNemar’s exact test found System B significantly ahead of System A; System C trailed both. Bootstrapped object-clustered differences: B over A by 22.0 points [9.7, 35.1]; A over C best-of-five by 22.8 points [9.2, 35.6]. The from-scratch network reached only 23.6% but still beat GPT-4o by about 11 points.

On an exploratory orientation split (103 axis-aligned vs. 20 diagonal images), System A dropped 21.2 points on diagonal images, System B gained 12.3 points, and System C moved about 2.0 points—matching representation expectations for A and B, though diagonal intervals are wide and the finding remains exploratory.

Failure Taxonomy and Coordinate Binding

Shared scoring separated failures into angle-only, overlap-only, both, or no prediction. System C failed on both angle and overlap on 59.3% of calls (365/615), far more than either criterion alone. A looser diagnostic asked whether the predicted center fell inside the convex hull of labeled grasps: System C failed on 53.4% of scored calls (328/614), versus 7.0% for System A and 6.5% for System B.

Informal reading of model text alongside coordinates often found a correctly named object part with coordinates elsewhere in the same reply. The study does not isolate that mechanism experimentally; it is offered as a hypothesis. Related work (e.g., Set-of-Mark, VLAD-Grasp) often avoids having the vision-language model emit raw pose coordinates, and Wang et al. document text–coordinate disagreement in another multimodal setting. Rerunning System C with region selection instead of coordinate emission would test whether the gap is binding versus perception; that experiment was not run here.

Limitations

Evaluation used a single split rather than five-fold cross-validation common in published Cornell results; seven of 35 test object groups had only one image. Jacquard was unavailable. Engineering effort was unequal: System B received architecture search and learning-rate sweeps, while System C used one frozen prompt, so 12.4% is not a claimed ceiling for zero-shot GPT-4o. Image-level intervals treat non-independent samples; object-weighted accuracies are lower (A 51.3%, B 75.4%, C 10.5%) but preserve ordering. All inputs were RGB only, without depth.

Conclusion

Under one metric and one held-out Cornell split, learned ResNet18 (79.7%) outperformed the rule-based baseline (57.7%), which in turn outperformed zero-shot GPT-4o (12.4%). System A often places a rectangle on the object but cannot express diagonal grasps. System B orients well when it succeeds; remaining errors are mostly placement. System C’s distinctive pattern is coordinates frequently unbound from the region its own text describes—failing before angle or overlap matching becomes decisive. The ordering survives object-level resampling; the shape of each failure, especially coordinate binding for the vision-language model, is as informative as the accuracy gap.

Works Cited