As server rooms fill up with hardware accelerators, computational efficiency at scale becomes a paramount concern when (i) training and (ii) designing deep learning models. First, there is a data parallelism challenge: to keep accelerators fully utilized, the training batch size must be sufficiently large, but a large batch size slows down model convergence due to the less frequent model updates. Second, there is a model parallelism challenge: models with more training parameters achieve better accuracy but incur higher computational costs due to dense floating-point operations.

In this talk, I will first describe Crossbow, a single-server multi-GPU deep learning system that avoids the data parallel training trade-off. Crossbow trains multiple model replicas concurrently on each GPU, thereby avoiding under-utilization even when the preferred batch size is small. For this, Crossbow employs an efficient and scalable model synchronization scheme.

While Crossbow proposes a new parallel training algorithm, it leaves the model architecture unmodified. So, next, I will describe model architecture modifications to the BERT language model, namely GroupBERT, that can deliver the same accuracy but faster. GroupBERT introduces a more efficient Transformer layer based on grouped matrix multiplications. I will conclude the talk with an outlook on ongoing research.