A system and method for intelligently controlling headlights receive a multiplicity of images that represent frames of a video sequence of an external environment of a vehicle. At least one bright spot, or blob, is found that stands out from a dark background of the external environment within each frame of the multiplicity of images. A multiplicity of features is extracted from a found blob. A type is recognized of a found blob that is selected from a multiplicity of types of blobs. A determination is then made whether to turn on a high beam light or a low beam light based at least on the recognized type of the found blob and a set of rules. Finally, an action based on such decision is performed.