context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
Explains how model architectures can be discovered automatically rather than hand-designed by experts, covering the search space, strategy, and performance estimation that underpin NAS — useful context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → for anyone thinking beyond off-the-shelf architectures.
Key quotes
“We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.”
Lilian Weng
“The design of search space usually involves human expertise, as well as unavoidably human biases.”
Lilian Weng
“In the experiments by Zoph & Le 2017 , they were running 800 GPUs in parallel for 28 days and Baker et al. 2017 restricted the search space to contain at most 2 FC layers.”
Lilian Weng
“In their experiments, EA and RL work equally well in terms of the final validation accuracy, but EA has better anytime performance and is able to find smaller models. Here using EA in NAS is still expensive in terms of computation, as each experiment took 7 days with 450 GPUs.”
Lilian Weng
“With a well-designed search space, random search could be a very challenging baseline to beat.”
