A handheld user interface device for controlling a robotic system may include a member, a housing at least partially disposed around the member and configured to be held in the hand of a user, and a tracking sensor system disposed on the member and configured to detect at least one of position and orientation of at least a portion of the device. At least one of the detected position of the portion of the device and detected orientation of the portion of the device is correlatable to a control of the robotic system.