Recent work such as BERT and GPT-2 in unsupervised language modeling demonstrates that training large neural language models advances the state of the art in Natural Language Processing applications. However, for very large models, memory constraints limit the size of models that can be practically trained. Model parallelism allows us to train larger models, because the parameters can be split across multiple processors. Designing such an approach in a simple and efficient way still remains a challenge.

Recently, NVIDIA Research launched project Megatron to enable training state of the art transformer language models with billions of parameters. Join this webinar to learn how NVIDIA researchers created Megatron, the largest Transformer language model ever trained with 8.3 billion parameters at 24x the size of BERT and 5.6x the size of GPT-2. Trained on 174GB of text, this model establishes new state-of-the-art results in tasks such as LAMBADA which tests the ability to model long-term dependencies.

• Understand implementation of efficient model parallelism without any new compiler or model re-writing

• Learn the pre-training process for Megatron

• Know about zero-shot evaluations using Megatron

• Receive an overview of the PyTorch code used to train this model

• Grasp the scope of Megatron’s future uses

Content

content goes here