Techniques related to capturing 3D faces using image and temporal tracking neural networks and modifying output video using the captured 3D faces are discussed. Such techniques include applying a first neural network to an input vector corresponding to a first video image having a representation of a human face to generate a morphable model parameter vector, applying a second neural network to an input vector corresponding to a first and second temporally subsequent to generate a morphable model parameter delta vector, generating a 3D face model of the human face using the morphable model parameter vector and the morphable model parameter delta vector, and generating output video using the 3D face model.