In HEVC (High Efficiency Video Coding), a 2N×2N coding unit can be partitioned into various partition types hierarchically. The coding system uses a criterion to determine the best partition, where RD-rate is often used as the criterion. N×N partition at level k becomes redundant if 2N×2N at level k+1 will be evaluated. In order to eliminate the above redundancy, the allowable partition sizes are constrained according to a method previously disclosed. In the current invention, the complexity is further reduced. According to one embodiment, N×N partition is not allowed for any INTER mode regardless of the coding unit size. Furthermore, flexibility is provided so that either the method and apparatus with further complexity reduction can be selected or an alternative method and apparatus can be selected. Syntaxes to support embodiments according to the present invention are also disclosed.