A method for clustering images includes acquiring initial image data including a scene of interest. A 3D model is constructed of the scene of interest based on the acquired initial image data. Additional image data including the scene of interest is acquired. The additional image data is fitted to the 3D model. A line-of-sight of the additional image data is estimated based on the fitting to the 3D model. The additional image data is clustered according to the estimated line-of-sight.