An apparatus and method for video coding and decoding with constrained PU partition are disclosed. In the High Efficient Video Coding (HEVC) system, rate-distortion function or other performance criterion usually is evaluated for various CU partition and PU partition during the encoding process in order to select a configuration with best possible performance. The PU design in the current HEVC development results in some redundancy that causes rate-distortion function or other performance criterion repeatedly evaluated for same PU configuration. Accordingly, constrained PU partition is developed to eliminate or reduce the redundancy in processing. Furthermore, necessary syntax to convey the information related to constrained PU partition between an encoder and a decoder is developed. Systems embodying the present invention has been shown to result in sizeable reduction in encoding and decoding time while the performance in terms of RD-rate remains approximately the same or slightly higher than a conventional HEVC system.