A violation inspection system based on visual sensing of a self-driving vehicle and a method thereof are disclosed. The violation inspection system receives traffic images captured by the self-driving vehicle, identifies image objects by artificial intelligence, continuously detects a movement status of each of the image objects, to generate image object dynamics; when the image object matches one of a trigger object factor and an association object factor, the image object and the image object dynamics thereof are inputted to an AI-based situation recognition model, to output a current situation, and when the current situation matches a violation situation template, the image objects corresponding to the trigger object factor and the association object factor corresponding to the violation situation template, and a violation item are outputted, so as to achieve the technical effect of improving accuracy and diversity in violation behavior determination.