This disclosure relates to a system and method for attention-based surface crack segmentation. Existing methods do not efficiently handle the sub-problem of data imbalance and inaccurate predicted pixels are ignored. The present disclosure obtains a binary edge map by passing a m-channel image through an edge detection algorithm and concatenate the obtained binary edge map with a channel dimension to obtain a (m+1)-channel image. Feature maps are extracted from an encoder and a decoder by feeding the obtained (m+1)-channel image into a network, wherein the feature maps are convolved with an attention mask and merged in a fused network. The merged feature maps are up sampled and concatenated to obtain a final fused feature map. The final fused feature map is passed through a sigmoid activation function to obtain a probability map which is iteratively thresholded to obtain a binary predicted image. The binary image is indicative of crack pixels.