This disclosure describes techniques for creating and manipulating software notes representative of physical notes. For example, techniques are described for recognizing physical notes present within a physical environment, capturing information therefrom and creating corresponding digital representations of the physical notes, referred to herein as digital notes or software-based notes. At least some aspects of the present disclosure feature system and methods for note recognition using color classification. The system receives a visual representation of a scene having one or more notes, where each note has a color. The system generates indicators indicative of color classes of pixels in the visual representation. The system further determines a general boundary of one of the notes based on the indicators.