A method of and an apparatus for controlling intra and/or inter prediction for decoding of a video sequence are provided. The method includes determining whether a width or a height of a coding unit is a power of two, and based on the width or the height of the coding unit being determined to not be a power of two, splitting the coding unit into sub-blocks, each of the sub-blocks having a width or a height that is a power of two and maximized, so that a number of the sub-blocks is minimized. The method further includes applying the intra and/or inter prediction on the sub-blocks into which the coding unit is split.