A method of handling at least a call in handover in a wireless communication system and a communication device using the same are proposed. The wireless communication system includes at least a wireless communication network and a wireless communication device. The method includes the following steps. First, a wireless communication device receives a message for switching at least a packet-switched (PS) call to at least a circuit-switched (CS) call from the wireless communication network. Further, the wireless communication device sets a stream identifier of each of the CS calls to an identifier value. The method enables smooth single radio voice call continuity (SRVCC) handover from the PS call in one wireless communication network to the CS call in another wireless communication network. In addition, the method also enables SRVCC multicall handover from the PS call in one network operation mode to the CS call in another wireless communication network.