Disclosed herein is a framework for localizing anatomical structures. In accordance with one aspect, the framework receives a learned regressor and image data of a subject. The learned regressor may be invoked to predict a first spatial metric from a seed voxel to a target anatomical structure in the image data. The learned regressor may further be invoked to predict second spatial metrics from candidate voxels to the target anatomical structure. The candidate voxels may be located around a search region defined by the first spatial metric. The candidate voxel associated with the smallest second spatial metric may then be output as a localized voxel.