Rethinking Radar 3D Perception: Representation Learning from Raw Spectra

Miao Zhang, Mohamed Abdelsamad, Bin Yang, Sherif Abdulatif, Benedikt Loesch, Marco Altmann, Abhinav Valada, Bin Yang

November 2026

TL;DR Masked-autoencoder pretraining directly on raw 4D radar spectra, with a corruption that mimics spectral leakage and Doppler treated as a spatial axis. A generic detector pretrained this way reaches 55.9/50.1 mAP (BEV/3D) on K-Radar, about 5 points above point-cloud-based self-supervised baselines.

What they do

Radar perception usually reduces the raw spectrum to a sparse point cloud (for example with CFAR detection) before any learning, which fixes in advance what counts as noise. The authors learn from the unreduced post-FFT spectrum instead. Their premise: signal is the part of the spectrum that can be reconstructed from its corrupted surroundings, and noise is the part that cannot. That makes masked autoencoding the pretraining objective.

How it works

Spectral MAE (SPMAE). Each range-azimuth-elevation bin becomes a point with mean Doppler power as its feature. A share of the points is masked, and an encoder-decoder reconstructs their features with an L1 loss.
Leakage-aware variant (L-SPMAE). The visible points are also corrupted by convolving them with kernels built from random binary windows, which imitates the cross-bin energy spreading of spectral leakage.
Doppler as an axis. Points are indexed by (x, y, Doppler) and elevation bins become the features (RAD), so attention can model Doppler structure. Stacking Doppler bins as channels brings no benefit.
Backbone. A generic 3D detector, PTv3 with a CenterHead, is pretrained and then fine-tuned; no radar-specific architecture.

Results

K-Radar, supervised only: dense spectra do worse than point clouds (47.4/40.8 against 49.0/42.3 mAP, BEV/3D).
With L-SPMAE pretraining on RAD: 55.9/50.1 mAP, 5.1/4.7 points above the best point-cloud self-supervised baselines.
Fine-tuning on 25% of the labels (50.4/44.4) beats the supervised model trained on all of them (48.9/43.1).
On RADIal the gain is smaller, +0.7 mAP over FFTRadNet. Occupancy prediction and multi-object tracking on K-Radar also improve over training from scratch.

Limitations

Dense spectra are expensive: against point clouds, the 4D spectrum is 9.5x slower in training and 11x slower at inference, and training needs at least 50 GB of GPU memory at batch size 8. The signal-as-reconstructable argument is a motivation, not a proof. Bicycle detection drops after fine-tuning, which the authors attribute to very few training samples. Code is not yet released.

Figures

From Figure 1: Reducing dense spectra into sparse point representations suppresses both structured signals and noise,... — From Figure 1: Reducing dense spectra into sparse point representations suppresses both structured signals and noise, which may remove informative spectral structures. Dense spectra preserve the full spectral information. Red: ground truth, blue: prediction by spectrum-based SSL, light blue: prediction by point-cloud-based SSL. RD: Range-Doppler view. BEV: Bird’s-eye view.

From Figure 3: Overall pipeline. For L-SPMAE, random spatial masking splits the radar spectrum into a visible and a m... — From Figure 3: Overall pipeline. For L-SPMAE, random spatial masking splits the radar spectrum into a visible and a masked part. Spectral leakage modeling then corrupts the visible part, which is fed into the encoder, followed by the decoder with mask tokens for reconstruction. For downstream tasks, only the encoder is retained as the feature extractor.

From Figure 5: Prediction results in bird’s-eye view, with ground-truth and predicted boxes, for supervised learning ... — From Figure 5: Prediction results in bird’s-eye view, with ground-truth and predicted boxes, for supervised learning (left) and the fine-tuned L-SPMAE model (right).

Abstract

Raw radar spectra are widely viewed as too cluttered and noisy for direct representation learning, motivating handcrafted reductions, sparse point clouds, or compact Doppler descriptors in radar perception. We challenge this assumption and argue that the main limitation lies not in the raw representation itself, but in the absence of a learning objective that separates structured signal from noise. Handcrafted reductions partially address this problem but impose rigid assumptions about what constitutes signal, discarding weak yet informative structure while retaining artifacts that fit the reduction model. Once removed, such information cannot be recovered by supervised learning. We propose an alternative: structure is the component of a radar spectrum that remains reconstructable under localized corruption, while noise is the component that does not. This perspective motivates self-supervised masked modeling for dense radar spectra without predefined signal-noise separation. We instantiate it with Leakage-aware Spectral MAE, a masked autoencoder for raw radar cubes using a corruption strategy inspired by cross-bin energy spreading caused by spectral leakage. We also demonstrate the importance of Doppler organization: treating Doppler as a primary spatial dimension in an x-y-D representation, with height encoded as features, substantially outperforms naive channel stacking and enables effective learning from dense 4D spectra. Without handcrafted reductions, our approach consistently outperforms state-of-the-art radar-based detectors and point-cloud-based self-supervised learning (SSL) counterparts. These results establish raw radar spectra as an effective representation for 3D perception when paired with an appropriate pretraining objective.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong