This invention is method for determining direction mode for an image block in intra prediction of video frames. The method calculates a plurality of energy terms from the spatial frequency domain image block, then determines a coarse classification from these energy terms. For at least some coarse classifications the method searches from among a subset of less than all possible direction modes. The method also searches a most likely direction mode determined from neighbor blocks when available.