A method and device for controlling a vehicle based on neighboring vehicles is disclosed. The method includes identifying a fiduciary object associated with each of at least one neighboring vehicle based on at least one image captured for the at least one neighboring vehicle. The method further includes tracking a position of the fiduciary object associated with each of the at least one neighboring vehicle based on a bounding box surrounding the fiduciary object. The method includes determining a direction-speed vector for the vehicle and determining trajectory data associated with each of the one or more neighboring vehicles. The method further includes correlating the position of the fiduciary object and the trajectory data associated with each of the at least one neighboring vehicle. The method includes re-computing the direction-speed vector of the vehicle based on a result of correlating and controlling the vehicle thereafter.