A server system receives a visual query and information about the current location of the querying device. The server system sends the query and location information to a visual query search system and receives one or more search results in accordance with the visual query and the location information. From the one or more search results, the server system identifies an entity in the visual query. The server system further identifies a client-side action corresponding to the identified entity and creates an actionable search result element configured to launch the client-side action. For example, if the identified entity were a restaurant the actionable search result elements could include buttons to initiate a telephone call, to read reviews, to map the address, or to make a reservation. If the actionable search result is selected, the client system launches the client-side action in an application distinct from the visual query client application.