Disclosed herein is a video processing apparatus and method based on multiple texture images, which can process videos with optimal video quality at a low transfer rate. For this, an input video is divided into shot segments, and one is selected from a plurality of frames of each shot segment as a seed image. A plurality of texture points are detected from the seed image. The plurality of texture points are tracked from the plurality of frames of the shot segment and then spatio-temporal location transform variables for the respective texture points are calculated. A plurality of texture images are defined using texture points at which the spatio-temporal location transform variables correspond to one another.