Method and apparatus of video coding using decoder derived motion information based on bilateral matching or template matching are disclosed. According to one method, merge index for merge candidate group comprising bilateral matching merge candidate and/or template matching merge candidate are signalled using different codewords. According to another method, the first-stage MV or the first-stage MV pair is used as an only initial MV or MV pair or used as a central MV of search window for second-stage search. According to yet another method, after the reference template for a first reference list is found, the current template is modified for template search in the other reference list. According to yet another method, the sub-PU search is disabled for the template search. According to yet another method, block difference calculation is based on reduced bit depth during MV search associated with the decoder-side MV derivation process.