A method and apparatus for a three-dimensional or multi-view video encoding or decoding system utilizing unified disparity vector derivation is disclosed. When a three-dimensional coding tool using a derived disparity vector (DV) is selected, embodiments according to the present invention will first obtain the derived DV from one or more neighboring blocks. If the derived DV is available, the selected three-dimensional coding tool is applied to the current block using the derived DV. If the derived DV is not available, the selected three-dimensional coding tool is applied to the current block using a default DV, where the default DV is set to point to an inter-view reference picture in a reference picture list of the current block.