A server system receives a visual query and identifies an entity in the visual query. The server system further identifies a client-side action corresponding to the identified entity and creates an actionable search result element configured to launch the client-side action. Examples of actionable search result elements are buttons to initiate a telephone call, to initiate email message, to map an address, to make a restaurant reservation, and to provide an option to purchase a product. The entity identified in the visual query may be indirectly associated with a client-side action whose contact address or appropriate link is found in a search result associated with the identified entity. The client system receives and displays the actionable search result element, and upon a user selection of the actionable search result element, launches the client-side action in an application distinct from the visual query client application.