Methods and systems for detection and removal of cast shadows from an image. In particular, one or more embodiments compute correspondences between image patches in the image using a grid-based patch-matching algorithm. One or more embodiments then train a regression model to detect shadows from the computed patch correspondences. One or more embodiments then segment the detected shadows into shadow regions and identify cast shadows from the shadow regions. Once the cast shadows are identified, one or more embodiments use patch-based synthesis of pixels guided by a direct inversion of the image. Optionally, one or more methods can use pixels from the synthesized image and the naïve inversion of the image, based on a synthesis confidence of each pixel, to produce a combined result.