A computer system automatically organizes, retrieves, annotates and/or presents media data files as collections of media data files associated with one or more entities using context captured in real time from a viewing environment. The computer system presents media data from selected media data files on presentation devices in the viewing environment and receives and processes signals from sensors in that viewing environment. The processed signals provide context, which can be used to select and retrieve media data files, and which can be used to further annotate the media data files and/or other data structures representing collections of media data files and/or entities. The computer system can be configured to be continually processing signals from sensors in the viewing environment to continuously identify and use the context from the viewing environment.