Systems, methods and computer program products related to aligning heterogeneous sequential data. A first sequential data stream and a second sequential data stream are received. An action related to aligning the first sequential data stream and the second sequential data stream is determined using an alignment neural network. The alignment neural network includes a fully connected layer that receives as input: data from the first sequential data stream, data from the second sequential data stream, and data relating to a previously determined action by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream.