Systems and methods perform automatic 3D face modeling. In one implementation, a brief video clip of a user's head turning from front to side provides enough input for automatically achieving a model that includes 2D feature matches, 3D head pose, 3D face shape, and facial textures. The video clip of the user may be of poor quality. In a two layer iterative method, the video clip is divided into segments. Flow-based feature estimation and model-based feature refinement are applied recursively to each segment. Then the feature estimation and refinement are iteratively applied across all the segments. The entire modeling method is automatic and the two layer iterative method provides speed and efficiency, especially when sparse bundle adjustment is applied to boost efficiency.