In one embodiment, a system receives a number of point clouds captured by one or more LIDAR sensors of an ADV and corresponding poses. The system receives a number of RGB images from one or more image capturing sensors of the ADV. The system synchronizes the RGB images with the point clouds to obtain RGB point clouds. The system extracts features from the RGB point clouds, the features including contextual and spatial information of the RGB point clouds. The system registers the RGB point clouds based on the extracted features and generates a point cloud map based on the registration of the RGB point clouds.