Techniques for creating and manipulating software notes representative of physical notes are described. A note management system comprises a note recognition module configured to receive image data capturing a note having a plurality of color segments, wherein the note recognition module is further configured to generate a plurality of indicators, each indicator indicative of a color class of a pixel or group of pixels within the image data and based on color values of the pixel or group of pixels; and a note extraction module configured to determine general boundaries of the color segments of the note based on the plurality of indicators and extract content using the general boundaries, the content comprising a plurality of content pieces, each of the content pieces corresponding to one of the color segments of the note.