The subject disclosure is directed towards evaluating search engine implementation performance using a task-based assessment metric. Logged user activities associated with the search engine implementation are processed into sessions and atomic events corresponding to the user activities within the sessions. The atomic events corresponding to the user activities are classified into tasks based on similarity of the queries within the user activities. After applying the task-based assessment metric to task information associated with the search engine implementation, an evaluation mechanism determines indicia of search engine implementation performance in terms of user behavior.