A user transparent telephone fraud detection and prevention system employs speech analysis techniques to determine whether at least one participant in a telephone conversation is an authorized user of the telephone line. A multi-party conversation on a phone line (e.g., a PBX line) is analyzed to separate speech information from noise and then the speech information is separated according to each participant in the conversation. Speech data for each participant is then compared to a library of speech data of authorized users of the system. If a match is found the system determines that the call is authorized. If no match is found the call is determined to be unauthorized and appropriate line control functions are carried out. Examples of the line control functions are: terminating the call; recording the call; disabling the line; intruding into the call and playing noise or a warning message; and intruding into the call to demand identification data from the call participants. The system is also used to verify that a user of a calling card number is an authorized user.