A method of estimating output motion vector includes generating a group of candidate motion vectors, selecting two or more reference vectors, variably establishing a search window, and estimating an output motion vector by performing a motion vector refinement on the search window. The method selects the two or more reference vectors from the group of the candidate motion vectors and the search window is established according to a relativity among the reference vectors. A transcoder using the method includes an encoding parameter estimator configured to select the two or more reference vectors and to variably establish the search window according to the relativity between the reference vectors. Therefore, the search window may be efficiently established without increasing calculation time, so that the output motion vector may be more correctly estimated.