A method and system for fusing sensed measurements includes a depth sensor to acquire depth measurements of a scene as a sequence of frames, and a camera configured to acquire intensity measurements of the scene as a sequence of images, wherein a resolution of the depth sensor is less than a resolution of the camera. A processor searches for similar patches in multiple temporally adjacent frames of the depth measurements, groups the similar patches into blocks using the intensity measurements, increases a resolution of the blocks using prior constraints to obtain increased resolution blocks, and then constructs a depth image with a resolution greater than the resolution of sensor from the increased resolution blocks.