An initial value of a current depth Depth is 1, and an inter-frame mode selection method includes: S701: if calculation of a coding overhead of coding a coding unit CUDepth is to be skipped, invoking S703 to S705; and if the calculation is not to be skipped, invoking S702 to S705; S702: determining a current optimum coding mode and coding overhead of the coding unit CUDepth; S703: dividing the coding unit CUDepth into multiple coding subunits having a depth Depth+1, and recursively performing S701 to S705 until the depth of the coding subunit reaches a preset maximum depth or satisfies a division stopping condition; S704: comparing a sum of coding overheads of the multiple coding subunits with the current coding overhead of the coding unit CUDepth; and S705: determining that a mode corresponding to the smaller one in S704 is an optimum coding mode.