Multimodal Person Concept Model for Person Identification under Varying Observation Conditions
TL;DR A nonparametric Bayesian model groups a few 10 s clips of visual, voice and transcript features into one concept per person, without labels or the number of people, and later identifies people from those concepts. Over eight conditions with six participants it averages 0.89 accuracy, matching k-means given the true head count (0.88), and a Toyota HSR uses it to recall what a person said before.
What they do
A service robot should recognize people it has talked to, though it sees each briefly and may meet them again in other clothes or a noisy room. This paper forms “person concepts” from a few minutes of multimodal recordings, without identity labels or a known number of users, and identifies people from them under changed conditions.
How it works
- Observations. Each person’s recording is cut into 10 s clips. Each clip gives one frame (CLIP image features), the waveform (CLAP audio features) and a Whisper transcript (CLIP text features), each reduced to 4 dimensions with PCA.
- Concept formation. A Chinese-restaurant-process mixture with Gaussian per-modality likelihoods is fit by Gibbs sampling. Every clip starts as its own concept, so the number of people is inferred, and a final sample is chosen without labels.
- Identification. A new observation is assigned to the concept with the highest likelihood over whichever modalities are available.
Results
- Six participants, 18 training clips each, tested under eight combinations of appearance change, kitchen noise (10 dB SNR) and corrupted transcripts: the model averages 0.89 accuracy, against 0.88 for k-means given the true count, 0.65 for the same model with the count fixed at 6 or 7, and at most 0.29 for Qwen2.5-VL.
- Accuracy is 0.98 in the stable condition, 0.85 under visual change and 0.76 when all three modalities change.
- With noise down to -5 dB, audio alone drops to 0.34 but all three modalities stay at 0.97-0.98. Text features are near chance on their own; visual plus audio carry concept formation (ARI 1.00).
- On a Toyota HSR, a concept was reached from a single image or about 5 s of speech and its stored utterance recalled.
Limitations
Only six participants, with synthetic audio and text variation. Japanese transcripts are encoded with an English-oriented CLIP text encoder and contribute little. The HSR demo skips PCA, over-segments concepts and is qualitative only, with no on-robot accuracy measured.
Videos
Figures



Abstract
Service robots in domestic and office environments must identify users to provide personalized services based on their preferences and previous interactions. This is challenging because only a few observations may be available for learning, while observation conditions change owing to appearance, environmental noise, and utterance content. To address this challenge, this study proposes a multimodal person concept model that learns person concepts from a small number of observations, without person labels, and enables robust identification under changing conditions. The model employs a nonparametric Bayesian generative model based on the Chinese restaurant process and integrates visual, acoustic, and linguistic features extracted using Contrastive Language-Image Pre-training (CLIP) and Contrastive Language-Audio Pretraining (CLAP), followed by principal component analysis (PCA). We evaluated six participants under eight conditions in which appearance changes, additive noise, and transcription errors were introduced individually and in combination. Averaged over the eight conditions, the proposed model achieved an identification accuracy of 0.89 while inferring the number of person concepts from the observations, which is comparable to k-means clustering that was supplied with the true number of participants (0.88). We also examined PCA dimensionality, modality availability, and variation strength, and demonstrated the model on a Toyota Human Support Robot. These results show that multimodal person concept learning supports robust person identification under changing conditions, and that identification can succeed even when the number of concepts is not correctly recovered. The project page and demonstration video are available at https://okitamisan01.github.io/person_concept/.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
