A method and apparatus of Inter prediction for video coding including IBC (Intra Block Copy) are disclosed. In one method, an IBC Merge candidate list is generated from Merge candidates associated with one or more candidate types, wherein one or more target candidates associated with a target candidate type are excluded from the IBC Merge candidate list when one or more constraints are satisfied. Current motion information associated with the current block is encoded or decoded using the IBC Merge candidate list. According to another method, a pruning process for the IBC Merge candidate list is skipped or simplified according to block width, block height, or block area of the current block.