This invention provides a system and method for determining the pose of shapes that are known to a vision system that undergo both affine transformation and deformation. The object image with fiducial is acquired. The fiducial has affine parameters, including degrees of freedom (DOFs), search ranges and search step sizes, and control points with associated DOFs and step sizes. Each 2D affine parameter's search range and the distortion control points' DOFs are sampled and all combinations are obtained. The coarsely specified fiducial is transformed for each combination and a match metric is computed for the transformed fiducial, generating a score surface. Peaks are computed on this surface, as potential candidates, which are refined until a match metric is maximized. The refined representation exceeding a predetermined score is returned as potential shapes in the scene. Alternately the candidate with the best score can be used as a training fiducial.