A convolution processing apparatus and method are disclosed. The convolution processing apparatus may include a controller configured to load a pixel of an input image and skip a process associated with the pixel in response to a value of the loaded pixel being 0, a filter bank including at least one filter and configured to extract at least one kernel element corresponding to the pixel from the at least one filter based on an index of the pixel and an input channel of the pixel, and a multiplier-accumulator (MAC) configured to perform a convolution operation based on the value of the pixel and a value of the at least one kernel element and accumulatively store an operation result of the convolution operation, the operation result corresponding to an output image.