One embodiment of the present invention sets forth a technique for performing dynamic randomized controlled testing to evaluate design choices for a consumer electronics (CE) device. A plurality of CE devices may communicate with a server system over a data communications network to be assigned a variant of the design choice. The variant may correspond to a variant of a user interface component of the CE device or a functional component of the CE device. Once assigned, feedback may be collected from the plurality of CE devices specifying one or more performance metrics associated with the design variant assigned to a respective CE device.