The present teaching relates to method, system, medium, and implementation of estimating a pose of an object in autonomous driving. Stereo images are obtained from a scene via at least two cameras. Objects present in the scene are then detected from the stereo images. For each object detected, a plurality of feature points are identified and depth information associated with the feature points are obtained. An orientation of each object is then estimated based on the feature points and depth information of the object and the pose of the object is then estimated based on the orientation and additional features of the object.