An information processing system and method for gathering and interpreting information includes capturing information from at least one of a plurality of information streams/sensors wherein the information includes video, audio, seismic, radio frequency (RF), and/or text then applying a standardized tag to an event at a predetermined time or over a predetermined period of time and storing the standardized tag in a repository which can be interrogated rapidly for situation/scene understanding. The information processing system and method include providing a plurality of segmentation algorithms, determining the type of information to be processed and selecting one or more of the segmentation algorithms to process the information based upon the type of information to be processed.