In one embodiment, a method includes receiving an input tensor corresponding to a media object at a binary convolutional neural network, wherein the binary convolutional neural network comprises at least one binary convolution layer comprising one or more weights, and wherein the media object is associated with a particular task, binarizing the input tensor by the at least one binary convolution layer, binarizing the one or more weights by the at least one binary convolution layer, and generating an output corresponding to the particular task by the binary convolutional neural network based on the binarized input tensor and the binarized one or more weights.