Systems and methods for predicting head pose for a rendering engine of an augmented or virtual reality device can include a recurrent neural network (RNN) that accepts a time series of head pose data and outputs a predicted head pose. The recurrent neural network can include one or more long short term memory (LSTM) units or gated recurrent units (GRUs). A fully connected (FC) layer can accept input from the RNN and output a 3 degree-of-freedom (DOF) head pose (e.g., angular orientation or spatial position) or a 6 DOF head pose (e.g., both angular orientation and spatial position). The rendering engine can use the predicted head pose to generate and display virtual content to the user at the time the user looks toward the position of the virtual content, which reduces system latency and improves user experience.