Provided are a computer program product, system, and method for processing images from a gaze tracking device to provide location information for tracked entities. A gazed image is received from a gaze tracking device at which the user is gazing while operating the gaze tracking device. User speech is received from a microphone. The user speech is as having memorization intent. A description of a tracked entity that is an object of the memorization intent is extracted from the user speech. A tracking object in a tracking object database is created including the gazed image and the extracted description of the tracked entity for which the user has memorization intent. A determination is made of a location of the tracked entity and the determined location of the tracked entity is associated with the created tracking object in the tracking object database.