Bryan Catanzaro

Catanzaro earned his Berkeley EECS doctorate in 2011, advised by Kurt Keutzer. Berkeley also records his work with Adam Coates and Andrew Ng at Baidu on deep-learning training systems and end-to-end speech recognition. The connection between these projects is practical computation: how to make learning systems train and operate at useful scale.[2]

Catanzaro leads NVIDIA research applying deep learning to language understanding, graphics and chip design. NVIDIA credits his research with leading to cuDNN and credits him with helping lead the team behind DLSS 2.0. These are different forms of contribution: a library foundation for neural-network computation and team leadership on a graphics system.[1]

Splitting a model across GPUs

The 2019 Megatron-LM paper, which Catanzaro co-authored, tackles models that are too large to fit comfortably on one graphics processor. Its method splits computation within transformer layers across multiple GPUs. The authors implemented the approach in PyTorch with a small number of communication operations, making the distribution strategy available without a new compiler. The contribution is a concrete training method, rather than a claim that larger models always solve a task better.[3]

Combining forms of parallelism

A 2021 follow-up studies how to combine tensor, pipeline and data parallelism. These divide work inside layers, across successive model stages and across training data. Combining them carelessly can leave processors waiting or communicating excessively. The paper analyzes those trade-offs and introduces an interleaved pipeline schedule, giving researchers a way to reason about efficient training across large GPU clusters. Catanzaro is one of its co-authors.[4]