Generative AI Language Modeling with Transformers

Generative AI Language Modeling with Transformers

This course is part of multiple programs.

Instructors: Joseph Santarcangelo

What you'll learn

Explain the concept of attention mechanisms in transformers, including their role in capturing contextual information.
Describe language modeling with the decoder-based GPT and encoder-based BERT.
Implement positional encoding, masking, attention mechanism, document classification, and create LLMs like GPT and BERT.
Use transformer-based models and PyTorch functions for text classification, language translation, and modeling.

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

6 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

Build your subject-matter expertise

This course is available as part of

When you enroll in this course, you'll also be asked to select a specific program.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV

Share it on social media and in your performance review

There are 2 modules in this course

This course provides you with an overview of how to use transformer-based models for natural language processing (NLP).

In this course, you will learn to apply transformer-based models for text classification, focusing on the encoder component. You’ll learn about positional encoding, word embedding, and attention mechanisms in language transformers and their role in capturing contextual information and dependencies. Additionally, you will be introduced to multi-head attention and gain insights on decoder-based language modeling with generative pre-trained transformers (GPT) for language translation, training the models, and implementing them in PyTorch. Further, you’ll explore encoder-based models with bidirectional encoder representations from transformers (BERT) and train using masked language modeling (MLM) and next sentence prediction (NSP). Finally, you will apply transformers for translation by gaining insight into the transformer architecture and performing its PyTorch implementation. The course offers practical exposure with hands-on activities that enables you to apply your knowledge in real-world scenarios. This course is part of a specialized program tailored for individuals interested in Generative AI engineering. This course requires a working knowledge of Python, PyTorch, and machine learning.

In this module, you will learn the techniques to achieve positional encoding and how to implement positional encoding in PyTorch. You will learn how attention mechanism works and how to apply attention mechanism to word embeddings and sequences. You will also learn how self-attention mechanisms help in simple language modeling to predict the token. In addition, you will learn about scaled dot-product attention mechanism with multiple heads and how the transformer architecture enhances the efficiency of attention mechanisms. You will also learn how to implement a series of encoder layer instances in PyTorch. Finally, you will learn how to use transformer-based models for text classification, including creating the text pipeline and the model and training the model.

What's included

6 videos3 readings2 assignments2 app items1 plugin

6 videosTotal 39 minutes

Course Introduction2 minutesPreview module
Positional Encoding6 minutes
Attention Mechanism7 minutes
Self-attention Mechanism7 minutes
From Attention to Transformers7 minutes
Transformers for Classification: Encoder8 minutes

3 readingsTotal 14 minutes

Course Overview5 minutes
Specialization Overview7 minutes
Summary and Highlights2 minutes

2 assignmentsTotal 30 minutes

Practice Quiz: Positional Encoding, Attention, and Application in Classification15 minutes
Graded Quiz: Fundamental Concepts of Transformer Architecture15 minutes

2 app itemsTotal 105 minutes

Hands-on Lab: Attention Mechanism and Positional Encoding45 minutes
Hands-on Lab: Applying Transformers for Classification60 minutes

1 pluginTotal 2 minutes

Helpful Tips for Course Completion2 minutes

In this module, you will learn about decoders and GPT-like models for language translation, train the models, and implement them using PyTorch. You will also gain knowledge about encoder models with Bidirectional Encoder Representations from Transformers (BERT) and pretrain them using masked language modeling (MLM) and next sentence prediction (NSP). You will also perform data preparation for BERT using PyTorch. Finally, you learn about the applications of transformers for translation by understanding the transformer architecture and performing its PyTorch Implementation. The hands-on labs in this module will give you good practice in how you can use the decoder model, encoder model, and transformers for real-world applications.