A camera-based localization system is provided. The camera-based localization system may assist an unmanned vehicle to continue its operation in a GPS-denied environment with minimal increase in vehicular cost and payload. In one aspect, a method, a computer-readable medium, and an apparatus for localization via visual inertial odometry are provided. The apparatus may construct an optical flow based on feature points across a first video frame and a second video frame captured by a camera of the apparatus. The apparatus may refine the angular velocity and the linear velocity corresponding to the second video frame via solving a quadratic optimization problem constructed based on the optical flow, the initial values of the angular velocity and the linear velocity corresponding to the second video frame. The apparatus may estimate the pose of the apparatus based on the refined angular velocity and the refined linear velocity.