In various embodiments, methods and systems for testing failover and recovery are provided. Systematic testing of a distributed system is performed, where the systematic testing probabilistically determines a processing order of events to effectuate system states for the plurality of state machines. An iteration of the systematic testing tests one the system states and includes sending a termination message to a state machine; receiving a termination acknowledgment message, the termination message causing the state machine to halt at the state and event of the state machine for the system state; and instantiating a recovery state machine. The recovery state machine is instantiated with a same state and same role as the halted state machine. Results of the systematic testing are verified against an expected outcome, the results being generated by running the distributed system with the instantiated recovery state machine for each iteration of the systematic testing.