Method of identifying actions of a shopper to account for taken items by the shopper in a cashierless checkout includes sampling a shopping environment using one or more video cameras to generate video features related a shopper in connection to an item and sampling using one or more supplemental sensors to generate supplemental sensor feature data, receiving output of the sampled video and supplemental sensor features as feature inputs to a deep learning model used for making inferences related to the state of a scenario involving shopper action of taking the item into their possession or held and other actions including moving outside a zone initially associated with the item.