The present disclosure relates to detection of obstacles by an autonomous vehicle in real-time. An obstacle detection system of an autonomous vehicle obtains point cloud data from single frame Light Detection and Ranging (LIDAR) data and camera data, of the surroundings of the vehicle. Further, the system processes the camera data to identify and extract regions comprising the obstacles. Further, the system extracts point cloud data corresponding to the obstacles and enhances the point cloud data, with the detailed information of the obstacles provided by the camera data. Further, the system processes the enhanced point cloud data to determine the obstacles along with the structure and orientation of obstacles. Upon determining the details of the obstacles, the system provides instructions to the vehicle to manoeuvre the obstacle. The disclosed obstacle detection system provides accurate data about a structure, an orientation and a location of the obstacles.