Semantic Scene Understanding for Autonomous Vehicles: A Comprehensive Review of Vision Transformers

Abstract

Semantic scene understanding—the ability to parse visual scenes into meaningful object categories, spatial relationships, and contextual information—represents a foundational capability for autonomous vehicle perception, with vision transformers emerging as a paradigm shift from convolutional neural networks in achieving human-level scene comprehension. This comprehensive review examines vision transformer architectures and their applications to autonomous vehicle perception tasks including semantic segmentation, panoptic segmentation, object detection, instance segmentation, and scene graph generation. We systematically analyze the fundamental mechanisms distinguishing transformers from CNNs: self-attention operations capturing long-range dependencies across entire images, positional encodings preserving spatial information, and multi-head attention enabling diverse feature relationships. The review traces the evolution from the original Vision Transformer (ViT) requiring massive pre-training datasets, through data-efficient variants (DeiT, Swin Transformer) with hierarchical architectures and shifted windowing schemes, to specialized designs for dense prediction tasks (SegFormer, Mask2Former, OneFormer). Particular emphasis is placed on transformer adaptations addressing autonomous driving requirements: real-time inference through efficient attention mechanisms and model compression, multi-scale feature representation for detecting objects ranging from distant vehicles to nearby pedestrians, and temporal modeling for video understanding incorporating motion cues. We examine pre-training strategies that enable transformers to learn robust visual representations including supervised pre-training on ImageNet, self-supervised methods (MAE, DINO, BEiT) learning from unlabeled data, and domain-specific pre-training on driving datasets. The review comprehensively analyzes benchmark performance on autonomous driving datasets (Cityscapes, BDD100K, nuScenes, Waymo) across semantic segmentation, object detection, and multi-task learning scenarios, comparing transformers against state-of-the-art CNN baselines. Application-specific considerations are examined for different perception challenges: urban driving requiring fine-grained segmentation of road infrastructure, highway scenarios emphasizing long-range detection, and adverse weather conditions where transformers’ global context modeling may improve robustness. We critically evaluate computational requirements, analyzing inference latency, memory consumption, and energy efficiency on automotive-grade hardware (NVIDIA Drive, Qualcomm Snapdragon Ride), identifying that while transformers achieve superior accuracy, their quadratic complexity with image resolution poses deployment challenges. The paper examines hybrid architectures combining convolutional feature extraction with transformer reasoning, offering improved efficiency-accuracy trade-offs. Advanced capabilities are reviewed including cross-modal transformers fusing camera, LiDAR, and radar data through unified attention mechanisms, temporal transformers aggregating information across video frames for robust tracking, and query-based detection transformers (DETR, Deformable DETR) eliminating hand-crafted components like anchor boxes and non-maximum suppression. Emerging research directions are analyzed including efficient transformer designs with linear attention complexity, neural architecture search for task-specific optimization, and foundation models pre-trained on massive multi-modal datasets enabling zero-shot transfer to novel driving scenarios. The review identifies critical gaps including limited interpretability of attention mechanisms for safety validation, vulnerability to adversarial perturbations, and insufficient evaluation on rare but safety-critical edge cases.