The invention relates to a vision-based attention system, comprising: at least one vision sensor, at least one image processing module processing an output signal of the vision sensor in order to generate at least one two-dimensional feature map, a dorsal attention subsystem generating a first saliency map on the basis of the at least one feature map, the saliency map indicating a first focus of attention for the driver assistance system, a ventral attention subsystem, independent to the dorsal attention subsystem, for generating a second saliency map on the basis of at least one feature map, which can be the same as the one used for the dorsal attention system or a different one, the second saliency map indicating unexpected visual stimuli.