A system is described for testing an application on one or more host devices in a host device farm using an application programming interface (“API”) to send a test package containing an application to a test server. The sending may be initiated by a single action such as a click on a control in a user interface, or may be automatic such as on completion of a build. The test server may then execute and test the application across one or more host devices using one or more testing frameworks. Test results based at least in part on the type of testing framework used in the application may then be provided to a client device.