Devices including gaming consoles, personal computers, tablets, smartphones, and the like may be configured with a client for capturing signals which are representative of user interactions with applications such as games. The captured signals are sent to a cloud-based service for archival storage. The service can subsequently replay the archived captured signals and compare them against known rules to determine if an achievement obtained by a user during interaction with the application was legitimately obtained in compliance with the rules or obtained improperly, for example by cheating or exploiting a bug in the application to falsely trigger the achievement. If the achievement is invalid, then the service can retroactively revoke the achievement. Alternatively, the service can replay the captured signals and detect instances in which an achievement was validly achieved but not properly acknowledged. The achievement can then be awarded or unlocked for the user retroactively.