Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, modifying queries based on non-textual content. In one aspect, a method includes receiving, from a user device, a query including a plurality of terms; determining active non-textual data displayed in an application environment on the user device; determining, from the non-textual textual data, modification data for the query; generating a set of modified queries based on the query and the modification parameters; scoring the modified queries according to one or more scoring criteria; selecting one of the modified queries based on the scoring; and providing, to the user device, search results responsive to the selected modified query.