Techniques for motion recognition using multiple data streams are disclosed. Multiple data streams from inertia sensors as well as non-inertial sensors are received to derive a motion recognition signal from motion recognizers. These motion recognizers are originally constructed from a training set of motion signals and may be updated with received multiple sensing signals. In one aspect, multiple data streams are converted to device-independent motion signals that are applied with the motion recognizers to provide a generalized motion recognition capability.