A direction of regularity, which minimizes a directional energy computed from pixel values of consecutive first and second frames of an input video sequence, is respectively associated with each pixel of the first frame and with each pixel of the second frame. Another direction of regularity (vz), which minimizes a directional energy computed from pixel values of the first and second frames, is also associated with an output pixel (z) of a frame of an output video sequence, located in time between the first and second frames. For processing such output pixel, the respective minimized directional energies for the output pixel, at least one pixel (z′) of the first frame and at least one pixel (z″) of the second frame are compared to control an interpolation performed to determine a value of the output pixel. The interpolation uses pixel values from at least one of the first and second frames of the input video sequence depending on the comparison of the minimized directional energies.