An apparatus embodiment is operative to scale video and includes an input buffer coupled to a real time source of video data and configured to hold a number of lines of video, a horizontal resizer coupled to the input buffer to resize lines of image data, outputting horizontally scaled line(s) to an intermediate buffer configured to store a number of lines. The apparatus has a vertical resizer coupled to the intermediate buffer configured to output vertically and horizontally resized lines of image data. At any given time, some of the lines in the input buffer are scheduled using the DMA controller for replacement via DMA by lines generated by the source of video data, and some or all of the remaining lines in the input buffer are available for processing by the horizontal resizer. A sufficient number of lines are available in the intermediate buffer, such that in operation, the intermediate buffer need not introduce latency.