Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. An initial texture can be segmented into materials and an initial coarse color estimate is determined for each material. Scene geometry is estimated from the captured views of the scene and is used to scale the initial coarse color estimates relative to each other such that the different materials appear to be lit with a similar irradiance. In this way, a global irradiance function is estimated describing the scene illumination. This provides a starting point for a color estimate and shading estimate extraction. The shading estimate can be used to fit surface normals to the global irradiance function. The set of surface normals and the color estimate are stored for subsequent use to allow relighting of the scene.