Learning to Sense for Driving: Joint Optics-Sensor-Model Co-Design for Semantic Segmentation

This paper presents a novel approach to autonomous driving perception by co-designing optics, sensor modeling, and semantic segmentation networks. The traditional approach of decoupling camera design from perception is challenged, and a unified end-to-end pipeline is proposed. The key innovation lies in optimizing the entire system, from RAW image acquisition to semantic segmentation, for task-specific objectives. The results on KITTI-360 demonstrate significant improvements in mIoU, particularly for challenging classes. The compact model size and high FPS suggest practical deployability. This research highlights the potential of full-stack co-optimization for creating more efficient and robust perception systems for autonomous vehicles, moving beyond traditional, human-centric image processing pipelines.