Depth reconstruction

Depth perception is a critical component in every autonomous driving system. Accurate depth information can be exploited at various levels of reasoning such as: vehicle localization, occlusion detection and handling, object dynamics modeling, tracking, path planning, etc. Currently, the market (2017) for depth sensing technologies is limited in a way that system cost rises exponentially with scanning fidelity. Examples techniques such as depth from stereo disparity, automotive LiDAR or time of flight cameras, all have various downsides ranging from sensitivity to miss-calibration, data sparsity or limited range.
The proposed depth reconstruction system is able to produce high fidelity, dense depth images while preserving object edges and suppressing measurement noise. It does so by up-sampling the sparse, but very accurate automotive LiDAR data. Input data in the form of a point cloud is segmented into objects which are then projected on a virtual camera image. This image is inherently sparse with more than 90% empty data. The missing pixels are subject to a multi-lateral filtering and up-sampling which is guided by the object masks.
The system is designed to run on a GPU enabled computer and was fully developed using the Quasar programming language. Point cloud segmentation and up-sampling is performed using fast image processing algorithms on 2D matrices and execution time on a mid-range GPU is around 50ms for generating a 0.5 mega pixel depth image.

Operation of the system on real world data

Quantitative analysis on real-world data show state-of-the-art reconstruction fidelity with less than 2.8% outlier pixels which falling below a predefined error threshold. Qualitative experiments were also conducted to confirm the usability of the dense depth images. A multi-modal pedestrian detector was trained to exploit both RGB and depth data. When this detector was compared to the RGB-only baseline it presented improved detection rates of 11%.

For more details please refer to the publication: "Semantically aware multilateral filter for depth upsampling in automotive LiDAR point clouds "DOI: 10.1109/IVS.2017.7995854