This disclosure describes a system for tracking removal or placement of items at inventory locations with a materials handling facility. In some instances, a user may remove an item from an inventory location and the inventory management system may detect that removal and update a user item list associated with the user to include an item identifier representative of the removed item. Likewise, if the user places an item at an inventory location, the inventory management system may detect that placement and update the user item list to remove an item identifier representative of the placed item.