A system and method is provided that automatically predicts the link that a user would like to follow by collecting and modeling user actions, and navigates to the link. In one aspect, the system may store a history of the user's manipulation of a browser's user interface, and predict the link of greatest interest to the user by comparing the visual similarity of potential links with previously selected and non-selected links, and estimating the location of the screen that the user is currently viewing.