Systems and methods of providing intelligent handset testing are disclosed. The system performs tests on handsets using customized scripts to analyze the handset on a variety of performance metrics while the handset is optionally running on an active telecommunications network. The system allows a user to “teach” a device to the system by interacting with virtual keys on an image of the hand held device and thus calculates the actual position on the hand held device by way of mathematical algorithms, taking into account any camera distortion at different heights and angles. Thus, the user can manipulate the subject test handset via the testing “fingers” using commands entered through the standalone unit to actuate, for example, any push buttons, slide buttons, recessed sensors, or screen touches. Additionally, the user can compare recorded images and sounds against expected images and sounds to validate tests.