Disclosed subject matter relates to a field of vehicle navigation system that performs a method for navigating an autonomous vehicle at an intersection of roads. An intersection management system may receive sensor data including at least one of depth of an object, images and a video of environment surrounding the autonomous vehicle. Further, traffic police and auxiliary objects associated with each traffic police are detected from plurality of objects of interest present in the images, when the autonomous vehicle is within a predefined distance from an intersection of roads. Thereafter, a correlation matrix comprising inferred data related to each traffic police and the auxiliary objects may be generated. Based on the correlation matrix, the video and the images, a gesture of the traffic police may be determined accurately. Finally, navigation information may be determined based on the correlation matrix and determined gesture, for navigating the autonomous vehicle.