A method of decoding video data includes constructing a motion vector candidate list of merge candidates for the current block of video data based on motion information from a number of neighboring blocks relative to the current block, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block, and wherein the number of neighboring blocks is greater than 5. In some examples, the method includes deriving a histogram of motion vector information for the neighboring blocks, and constructing the motion vector candidate list based on the derived histogram.