Approaches provide for using voice information to influence the importance of search result categories for a search query. For example, various embodiments may provide search results for a search query based on a most relevant search result category to the search query. Voice information associated with a subsequent user interaction may be analyzed to identify whether the search result category is correct or if search results from a different category should be provided. Additionally, the voice information may be used to update the relevance score of the search result category to the search query to improve the category matching of future queries.