A convolutional neural network system includes a sensor and a controller, wherein the controller is configured to receive an image from the sensor, divide the image into patches, each patch of size p, extract, via a first convolutional layer, a feature map having a number of channels based on a feature detector of size p, wherein the feature detector has a stride equal to size p, refine the feature map by alternatingly applying depth-wise convolutional layers and point-wise convolutional layers to obtain a refined feature map, wherein the number of channels in the feature map and the size of the feature map remains constant throughout all operations in the refinement; and output the refined feature map.