A method and system for segmenting structures such as lesions in an image is provided. The method comprises selecting one seed inside a lesion in an image either by a user or automatically. The method further includes deriving a directionally statistical model based on a background region or a foreground region of the lesion and determining candidate voxels along a radial direction. The candidate voxels represent the lesion. The method further includes segmenting the lesion using the candidate voxels.