A system for inferring features in a digital image includes a vision sensor for capturing and optically encrypting an image of a visual scene so as to shroud any human-perceivable features contained therein. An image interpreter in communication with the vision sensor is programmed to recognize and classify features in the encrypted image without prior decryption, thereby ensuring the privacy of any subjects depicted therein. The vision sensor is encoded with a calibration key that applies a unique transformation function to optically encrypt the captured image. The image interpreter is permitted access to the calibration key and applies a convolutional network constructed using the calibration key, a source convolutional network, and a set of private layer keys in order to infer features in the encrypted image without prior decryption. For applications requiring enhanced privacy protection, the inference output produced by the image interpreter remains encrypted but available for further feature detection.