Embodiments disclosed herein facilitate virtual reality (VR) video playback using handheld controller gestures. More specifically, jog and shuttle gestures are associated with controller rotations that can be tracked once a triggering event is detected (e.g., pressing and holding a controller play button). A corresponding jog or shuttle command can be initialized when the VR controller rotates more than a defined angular threshold in an associated rotational direction (e.g., yaw, pitch, roll). For example, the jog gesture can be associated with changes in controller yaw, and the shuttle gesture can be associated with changes in controller pitch. Subsequent controller rotations can be mapped to playback adjustments for a VR video, such as a frame adjustment for a jog gesture and a playback speed adjustment for the shuttle gesture. Corresponding visualizations of available gestures and progress bars can be generated or otherwise triggered to facilitate efficient VR video playback control.