A system for modifying video including a source of a filter stack specifying filter operations to be carried out on a video clip, a filter registry of filter functions including information on filter operations that can be carried out by the filter functions, and a filter planner that receives the filter stack and accesses the registry in order to generate a filter plan specifying filter functions to be used in carrying out the filter operations specified in the filter stack.