A method and apparatus of three-dimensional/multi-view coding using a candidate list including a second inter-view candidate in the candidate list for Merge mode, Skip mode or AMVP based (Advanced Motion Vector Prediction based) Inter mode are disclosed. The second inter-view candidate can be derived based on already coded or decoded texture data for the candidate list to include. For example, the second inter-view candidate can be determined from the motion information associated with a corresponding block in a reference view, where the corresponding block is located according to the location of the right-bottom neighboring block and a selected disparity vector. The right-bottom neighboring block is located across from a right-bottom corner of the current texture block. The second inter-view candidate can be inserted into the candidate list only when the number of previous available candidates is smaller than a pre-specified number.