To process media data from one or more sources, a hierarchy of media components are formed. Each media component receives as input one or more streams of media data and manipulates the media data to produce an output media stream. The media components conform to a uniform plug-in structure. The plug-in structure defines an abstraction that facilitates the hierarchical organization of the media components and that allows the media components to interact, in a decoupled manner, with each other and with processors that perform specific types of media data manipulations. The hierarchy can be reorganized dynamically.