In a method of software item testing, an automated test of a software item is initiated in a cloud-based computing environment in exchange for a fee. The test is part of a testing framework in the cloud-based computing environment. In response to occurrence of an error condition during the test, the test is paused within the cloud-based computing environment. An alteration to the test is accepted while the test is paused. The alteration is accepted via a user interface to the cloud-based computing environment. Testing is continued within the cloud-based computing environment, using altered test.