A video action detection system uses feature a data extractor to extract feature data from video data at detected spatiotemporal interest points. A feature data quantizer assigns the extracted feature values to bins of a feature vector. Bin values are computed from a sum of contributions of spatiotemporal points of interest that have been assigned to the bin, with a bin dependent adjustment of a size of the sum and/or the contributions. The video action detection system computes a sum of match scores between the feature vector and reference vectors for the predetermined type of action. The bin dependent adjustment of the size of the sum and/or the contributions is adapted in a training step, based on partial match scores for individual bins obtained using an initial action detector.