Shape measurement of a specular object even in the presence of multiple intra-object reflections such as those at concave regions of the object. Silhouettes of the object are extracted, by positioning the object between a camera and a background. A visual hull of the object is reconstructed based on the extracted silhouettes, such as by image capture of shadows of the object projected onto a screen, and image capture of reflections by the surface of the object of coded patterns onto the screen. The visual hull is used to distinguish between direct (single) reflections of the coded patterns at the surface of the object and multiple reflections. Only the direct (single) reflections are used to triangulate camera rays and light rays onto the surface of the object, with multiple reflections being excluded. The 3D surface shape may be derived by voxel carving of the visual hull, in which voxels along the light path of direct reflections are eliminated. For surface reconstruction of heterogeneous objects, which exhibit both diffuse and specular reflectivity, variations in the polarization state of polarized light may be used to separate between a diffuse component of reflection and a specular component.