A system and method for navigating an autonomous driving vehicle (ADV) by capturing and analyzing information of a global scene and local objects around the ADV, is disclosed. The system comprises a sensor assembly incorporated on the ADV and a computing device in communication with the sensor assembly. The sensor assembly is configured to collect environmental data around the ADV. The computing device comprises a processor, and a memory unit for storing a predefined scene template and environmental data. The computing device is configured to process the environmental data to identify a moving and static object. The computing device is further configured to observe an environmental scene around the ADV. The observed environmental scene is aligned with a predefined scene template. Further, the predefined scene template is adjusted using the processed environmental data. The computing device provides instruction to control the vehicle based on the adjusted scene template.