Techniques are described for determining locations of interest based on user visits. In some situations, the techniques include obtaining information about actual locations of users at various times, and automatically analyzing the information to determine particular locations in a geographic area that are of interest, such as for frequent destinations visited by users. After determining a particular location of interest, it may be represented by generating a corresponding location model to describe the geographic subarea or other location point(s) covered by the determined location of interest, and one or more points of interest (e.g., businesses, parks, schools, landmarks, etc.) may be identified that are located at or otherwise correspond to the determined location of interest. In addition, a determined location of interest may be further used in various ways, including to identify later user visits to that location (e.g., to a point of interest identified for the location).