An “Oriented Disk Interpolator” provides various techniques for interpolating between points in a point cloud using RGB images (or images in other color spaces) to produce a smooth implicit surface representation that can then be digitally sampled for ray-tracing or meshing to create a high fidelity geometric proxy from the point cloud. More specifically, the Oriented Disk Interpolator uses image color-based consistency to build an implicit surface from oriented points and images of the scene by interpolating disks in 3D space relative to a point cloud of a scene or objects within the scene. The resulting implicit surface is then available for a number of uses, including, but not limited to, constructing a high fidelity geometric proxy.