Various implementations of the subject matter described herein relate to a neural network-based action detection. There is provided an action detection scheme using a neural network. The action detection scheme can design and optimize the neural network model based on respective importance of different frames such that frames that are more important or discriminative for action recognition tend to be assigned with higher weights and frames that are less important or discriminative for action recognition tend to be assigned with lower weights.