Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating neural network architectures. One of the methods includes receiving a request to determine an architecture for a task neural network; maintaining data specifying a plurality of candidate architectures for the task neural network; repeatedly performing operations comprising: selecting one or more candidate architectures in the maintained data to be modified; generating a new candidate architecture from the selected candidate architecture by, for each hyperparameter in the set of hyperparameters, selecting the value for the hyperparameter for the new candidate architecture; and adding data specifying the new candidate architecture to the maintained data; and selecting, as the final architecture for the task neural network, one of the candidate architectures specified in the maintained data.