Virtual and augmented reality sports training environments are disclosed. A user interacts with virtual players in a simulated environment of a virtual reality sporting event. In some embodiments, the user's actions and decisions are monitored by the simulated environment. The environment evaluates the user's performance, and provides quantitative scoring based on the user's decisions and timing. Coaches and other users may design customized scenarios or plays to train and test users, and the resultant scores may be reviewed by the coach. In one application, real life pitchers and their pitching data are tracked and replicated in a simulated pitching environment. A team of users may practice against a simulation of a pitcher they are about to compete against, swinging at pitches that they would see in a real game. Such environments may maximize effective practice time for users, and help develop better players with improved decision-making skills.