This is a demonstration for a 3D object recognition for autonomous driving. This experiment is implemented on a real car and tested over real-life data gathered under mid and heavy traffic situations.

For the task of 3D object recognition, a multi-modal perception pipeline is implemented via fusing:

• 2D camera

• 3D LIDAR

• Detection is limited to camera's field of view.

• Vision detection and classification is done via TensorRT-YOLOV3.

• 3D clusters are extracted by Euclidean Clustering.

• Creating 3D map of the environment vehicle is supposed to operate in.

• Activating and calibrating sensors.

• Specifying transformations between different frames.

• Pre-processing pointcloud.

• Localization.

• Detection (vision, range).

• Fusion.

• LIDAR sensor is used to create a 3D pointcloud of the road in which the vehicle is supposed to operate. And a vectormap layer can be also added, over pointcloud data to specify some semantic information about road, topology (e.g. location of driving area, lanes, lines, traffic signs, etc.) and, traffic rules (e.g. speed limits).

• Sensors (LIDAR, Camera) must be activated with proper parameters and they should be calibrated individually and together in order to be sure about their measurement accuracy. This is necessary to find a precise correspondence between LiDAR points and camera pixels.

• Following link depicts a precise calibration performance:

• world to map: This is a static transformation from world origin to the origin of local map.

• map to base-link: This is a static transformation from local map to the control center of the ego-vehicle.

• base-link to LİDAR: This is a dynamic transformation interpreting LIDAR points to vehicle's control center for better control and planning performances.

• LIDAR to camera: This is a static transformation to obtain a correspondence between LiDAR and camera observations.

• This is a step for optimizing the performance of further processes by reducing amount of data and eliminating the negative effect of unnecessary data in execution time.

1. Down sampling the pointcloud while retaining the necessary information.

2. Cropping the pointcloud data to remove unnecessary parts that is not going to be used in further processes.

3. Removing ground from pointcloud since it is not a collidable obstacle for autonomous vehicle.

4. Comparing online LIDAR data with offline pointcloud map to remove similar static objects. This will retain only dynamic objects in interest.

Localization:

• Localizing the ego-vehicle relative to the local pointcloud map. This is done via ndt-matching algorithm which compares live LIDAR data in execution time with previously gathered pointcloud data of the same environment to localize the vehicle inside map.

Detection:

• LIDAR Detection: LIDAR detection is done via Euclidean Clustering algorithm that extracts the pointcloud cluster of dynamic objects. This only gives the shape and location of detected objects and is not providing any classification information about it.

• Camera Detection: Camera detection is done via TensorRT-YOLOv3 which provides objects' location over image-plane and also classification label.

Fusion:

• Here I use a late fusion approach that associates vision detected labels with range detected clusters after each domain specific network processes its own data and makes its own decision.

• Image data is not involved.

• There is no fusion (no multi-modality in data).

Sensor used: One LiDAR

• car

• pedestrian

• cyclist

Two different network structures designed to be exploited:

• One for detecting {car} class and one for {pedestrian, cyclist} classes.

Paper Link:

https://arxiv.org/pdf/1812.05784.pdf