Neural Architecture Search

Searching automatically for a neural-network design can become expensive because each candidate needs an evaluation. Lilian Weng's 2020 survey explains why training every candidate to completion is a costly way to decide which design to try next.

Cheaper methods train candidates for fewer rounds or on smaller datasets; others reuse weights across designs. These methods reduce the work needed to obtain feedback, while changing how performance is estimated. The search also remains bounded by the operations and connections its designers allow. Assessing an architecture-search result therefore requires understanding both the available designs and how the candidates were compared.

“We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.”

Lilian Weng

“The design of search space usually involves human expertise, as well as unavoidably human biases.”

Lilian Weng

“In the experiments by Zoph & Le 2017 , they were running 800 GPUs in parallel for 28 days and Baker et al. 2017 restricted the search space to contain at most 2 FC layers.”

Lilian Weng

“In their experiments, EA and RL work equally well in terms of the final validation accuracy, but EA has better anytime performance and is able to find smaller models. Here using EA in NAS is still expensive in terms of computation, as each experiment took 7 days with 450 GPUs.”

Lilian Weng

“With a well-designed search space, random search could be a very challenging baseline to beat.”

Lilian Weng