The present invention is a system and a method of segmenting and detecting objects which can be approximated by planar or nearly planar surfaces in order to detect one or more objects with threats or potential threats. The method includes capturing imagery of the scene proximate a platform, producing a depth map from the imagery and tessellating the depth map into a number of patches. The method also includes classifying the plurality of patches as threat patches and projecting the threat patches into a pre-generated vertical support histogram to facilitate selection of the projected threat patches having a score value within a sufficiency criterion. The method further includes grouping the selected patches having the score value using a plane fit to obtain a region of interest and processing the region of interest to detect said object.