An image-based jig inspection system for inspecting a state of a sensorless jig managed in a vehicle body factory includes: a jig configured to fix a vehicle body component mounted by a plurality of clamping units; a camera module configured to photograph an inspection image of the jig; a vision system configured to compare the inspection image with a predetermined image analysis reference to determine a forward moving state or a backward moving state of a respective one of the clamping units; a welding robot configured to weld the component; and a main controller configured to control the welding robot according to a predetermined process based on the state information of the respective one of the clamping units.