A method and apparatus of Intra mode coding for a scalable video coding system are disclosed. For a current Intra-coded block in the enhancement layer (EL), predictive coding is applied to the current Intra mode based on the base layer (BL) coding mode associated with the co-located block in the BL and neighboring coding modes associated with neighboring blocks of the current block in the EL. The neighboring blocks of the current block in the EL comprise a left block adjacent to a left side of the current block and a top block adjacent to a top side of the current block. One or more most probable modes (MPMs) can be derived from the neighboring coding modes and the BL coding mode, and the MPMs is then used for predictive coding of the current Intra mode.