A plenoptic otoscope enables three-dimensional and/or spectral imaging of the inside of the ear to assist in improved diagnosis of inflammations and infections. The plenoptic otoscope includes a primary imaging system and a plenoptic sensor. The primary imaging system includes an otoscope objective and relay optics, which cooperate to form an image of an inside of an ear at an intermediate image plane. The plenoptic sensor includes a microimaging array positioned at the intermediate image plane and a sensor array positioned at a conjugate of the pupil plane. An optional filter module may be positioned at the pupil plane or one of its conjugates to facilitate three-dimensional and/or spectral imaging.