A video analytics system and associated methodology for performing low-level video analytics processing divides the processing into three phases in order to efficiently use SIMD instructions of many modern data processors. In the first phase, pixels of interest are gathered using a predetermined mask and placed into a pixel matrix. In the second phase, video analytics processing is performed on the pixel matrix, and in the third phase the pixels are scattered using the same predetermined mask. This allows many pixels to be processed simultaneously, increasing overall performance. A DMA unit may also be used to offload the processor during the gathering and scattering of pixels, further increasing performance. A network camera integrates the video analytics system to reduce network traffic.