A system and method are disclosed for fast sub-pixel optical flow estimation in a two-dimensional or a three-dimensional space. The system includes a pixel-level optical flow estimation module, a matching score map module, and a sub-pixel optical flow estimation module. The pixel-level optical flow estimation module is configured to estimate pixel-level optical flow for each pixel of an input image using a reference image. The matching score map module is configured to generate a matching score map for each pixel being estimated based on pixel-level optical flow estimation. The sub-pixel optical flow estimation module is configured to select the best pixel-level optical flow from multiple pixel-level optical flow candidates, and use the selected pixel-level optical flow, its four neighboring optical flow vectors and matching scores associated with the pixel-level optical flows to estimate the sub-pixel optical flow for the pixel of the input image.