In a network having an L2SW in which uplink VLAN is set, a transfer delay of a voice call packet is prevented. In a process of establishing a session between terminals (5), a packet transfer device (1) stores information capable of specifying a sending terminal (5-1) and a receiving terminal (5-2) and a correspondence relation (user connection information) to a connection port into a table. A SIP message of a session control packet communicated between the terminals (5) is snooped, and a destination IP address, a transmission source IP address, a destination port number, a transmission source port number, a connection port etc. are stored into the table. In a case where an RTP packet as voice data is received after the session is established, the packet transfer device (1) performs a loopback transfer or transfers it to an uplink port in accordance with the table.