Reading the Table: Sim-to-Real Game State Detection for the Chef's Hat Card Game

TL;DR Render 25,413 auto-labeled, domain-randomized images of the Chef's Hat card game in simulation and finetune YOLO detection and segmentation models on them. On real images from a NICO humanoid's camera, the pipeline assigns cards to game zones with an F1-score of 0.906, against 0.627 without randomization.
What they do
To play the Chef’s Hat card game with people, a robot has to read the game state: which of 18 card designs lie in which of 11 zones of the playing field. The 13 ingredient cards share an almost identical layout, and labeling real images by hand is slow. The authors generate all training data in simulation and test zero-shot on images from the fisheye eye camera of a real NICO humanoid.
How it works
- Simulated replica. Table, playing field and 3D-printed card stands are rebuilt in CoppeliaSim. Simulated games with simplified placement rules, 100 full games in all, give 25,413 samples. Boxes and zone masks come from projecting known 3D corner points into the camera.
- Domain randomization. Each game state is saved twice: once canonical, once with randomized lights, random-shape wall and floor textures, camera pose (up to 10 cm and 0.2 rad), field of view (90-135°, in place of modeling the fisheye distortion) and board and card-stand placement.
- Game-state pipeline. YOLO26n detects and classifies cards, YOLO26n-seg segments the zones, and each card goes to the zone that contains its box center.
Results
- Test set: 100 real images under 5 lighting and card-placement conditions, with no real training data.
- Card detection: F1 0.845 and mAP50 0.850 (3.5 ms per image), against 0.556 and 0.514 for a model trained on canonical images only.
- Card-to-zone assignment: F1 0.906 with 1.52 cards missed per image, against 0.627 and 7.02 missed without randomization. The larger YOLO26s does slightly worse (0.902).
Limitations
About 1.5 cards are missed per image, so a single observation does not give the full game state; using game rules to catch errors is proposed but not tested. The robot does not yet pick up or play cards, and card backs, distractor objects and realistic camera simulation are left to future work.
Figures




Abstract
Tabletop games offer a compelling scenario to study the interaction between artificial and human agents, combining social interaction with game state tracking and physical manipulation. To assess the game state, a robot needs to precisely detect closely related game objects, such as different card designs, requiring dedicated training data that is costly to annotate in the real world. Instead, we develop a fully automated synthetic data collection pipeline, using re-projection of known 3D coordinates to generate labels automatically and domain randomization to bridge the reality gap. This approach is used to collect a dataset of 25413 samples to finetune an object detection and a segmentation model, which are combined to assign detected cards to game-relevant zones and infer the game state. When evaluated on a real NICO humanoid robot, our pipeline achieves an F1-score of 0.906 for card-to-zone assignment, thus providing the perceptual foundation for a robot that can autonomously play Chef's Hat against human players.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
