Embodiments are directed to managing communication over a network. A co-located assessment agent may monitor communication traffic and determine assets and interactions with communication clients associated with one or more remote services. Asset groups associated with the assets may be determined based on characteristics of the assets. Representative assets from each asset group may be determined. Risk scores associated with each asset in a same asset group as the representative assets may be determined based on the remote services associated with the representative assets and the interactions with the communication clients such that a first risk score is initially generated based on a catalog local to a communication client and a succeeding risk score may be generated remotely based on an assessment model. The assets may be modified based on the first risk score or the succeeding risk score and provided to communication clients for display to a user.