An online system selects for display content items having an option to allow a user to converse with a content provider over an electronic communication system in a way that optimizes for the occurrence of that communication. Content items provided by the online system include links to an online communication system from which the online system can directly observe whether the user engaged in communications with third parties. The online system can thus obtain labeled training data describing communications between users and users' previous interactions with content items and pages of the online system. The trained model is applied to candidate content items to predict a probability that a user will engage in a communication with the content provider if the content is provided to the user, and the model optimizes the selection of content items for which the call to action is to engage in the communication.