A spectral sensing system includes an array of sampling optical elements (e.g. filters) and an array of optical sensors, and a deep learning model used to process output of the optical sensor array. The deep learning model is trained using a training dataset which includes, as training inputs, optical sensor output generated by shining each one of multiple different light waves with known spectra on the sampling optical element array for multiple different times, each time from a known angle of incidence, and as training labels, the known spectra of the multiple light waves. The trained deep learning model can be used to process output of the optical sensor array when light having an unknown spectrum is input on the sampling optical element array from unknown angles, to measure the spectrum of the light. The spectral sensing system can also be used to form a hyperspectral imaging system.