The present invention provides methods and systems for dynamically generating configuration files used in installing software distributions and software installation testing. In particular, the present invention employs a framework comprising an API to discover and monitor an inventory of distributions, a set of templates for the configuration files, and plugins that may be selectively invoked to customize the configuration files. When performing installation testing, a server will access the distribution inventory and select appropriate templates that are to be part of the test. The server will then automatically generate the installation configuration files from the templates. The server may also execute one or more plugins that customize the configuration files as desired. Small client applications run on the target machines of the installation tests and are used to download the configuration files and the distributions. The present invention may also employ a scheduling service to coordinate the operations of the server and target machines.