Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for validating software functionality. In some implementations, data indicating a configuration of a first server environment running a third-party installation of a first version of an application is obtained. A second server environment is generated and is configured to run the first version of the application. The second server environment can be configured to use a set of configuration settings for the first server environment that are indicated by the data for the first server environment. An installer for a second version of the application to transition the second server environment to the second version of the application is run. Capabilities of the second version of the application are evaluated using a series of tests for the application on the second server environment.