Systems and methods which implement an image reconstruction framework to incorporate complex motion of structure, such as complex deformation of an object or objects being imaged, in image reconstruction techniques are shown. For example, techniques are provided for tracking organ motion which uses raw time-stamped data provided by any of a number of imaging modalities and simultaneously reconstructs images and estimates deformations in anatomy. Operation according to embodiments reduces artifacts, increase the signal-to-noise ratio (SNR), and facilitates reduced image modality energy dosing to patients. The technology also facilitates the incorporation of physical properties of organ motion, such as the conservation of local tissue volume. This approach is accurate and robust against noise and irregular breathing for tracking organ motion.