Optimizations are provided for generating passthrough visualizations for Head Mounted Displays. The interpupil distance of a user wearing a head-mounted device is determined and a stereo camera pair with a left and right camera is used to capture raw images. The center-line perspectives of the images captured by the left camera have non-parallel alignments with respect to center-line perspectives of any images captured by the right camera. After the raw images are captured, various camera distortion corrections are applied to the images to create corrected images. Epipolar transforms are then applied to the corrected images to create transformed images having parallel center-line perspectives. Thereafter, a depth map is generated of the transformed images. Finally, left and right passthrough visualizations are generated and rendered by reprojecting the transformed left and right images.