Generative AI Advance Fine-Tuning for LLMs

Generative AI Advance Fine-Tuning for LLMs

This course is part of multiple programs.

Instructors: Joseph Santarcangelo

What you'll learn

In-demand gen AI engineering skills in fine-tuning LLMs employers are actively looking for in just 2 weeks
Instruction-tuning and reward modeling with the Hugging Face, plus LLMs as policies and RLHF
Direct preference optimization (DPO) with partition function and Hugging Face and how to create an optimal solution to a DPO problem
How to use proximal policy optimization (PPO) with Hugging Face to create a scoring function and perform dataset tokenization

Details to know

Earn a career certificate

Add to your LinkedIn profile

Assessments

5 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

Build your subject-matter expertise

This course is available as part of

When you enroll in this course, you'll also be asked to select a specific program.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV

Share it on social media and in your performance review

There are 2 modules in this course

Fine-tuning a large language model (LLM) is crucial for aligning it with specific business needs, enhancing accuracy, and optimizing its performance. In turn, this gives businesses precise, actionable insights that drive efficiency and innovation. This course gives aspiring gen AI engineers valuable fine-tuning skills employers are actively seeking.

During this course, you’ll explore different approaches to fine-tuning and causal LLMs with human feedback and direct preference. You’ll look at LLMs as policies for probability distributions for generating responses and the concepts of instruction-tuning with Hugging Face. You’ll learn to calculate rewards using human feedback and reward modeling with Hugging Face. Plus, you’ll explore reinforcement learning from human feedback (RLHF), proximal policy optimization (PPO) and PPO Trainer, and optimal solutions for direct preference optimization (DPO) problems. As you learn, you’ll get valuable hands-on experience in online labs where you’ll work on reward modeling, PPO, and DPO. If you’re looking to add in-demand capabilities in fine-tuning LLMs to your resume, ENROLL TODAY and build the job-ready skills employers are looking for in just two weeks!

In this module, you’ll begin by defining instruction-tuning and its process. You’ll also gain insights into loading a dataset, generating text pipelines, and training arguments. Further, you’ll delve into reward modeling, where you’ll preprocess the dataset and apply low-rank adaptation (LoRA) configuration. You’ll also learn to quantify quality responses, guide model optimization, and incorporate reward preferences. You’ll also describe reward trainer, an advanced training technique to train a model, and reward model loss using Hugging Face. The labs, in this module will allow practice on instruction-tuning and reward models.

What's included

6 videos3 readings2 assignments2 app items1 plugin

6 videosTotal 36 minutes

Course Introduction3 minutesPreview module
Basics of Instruction-Tuning6 minutes
Instruction-Tuning with Hugging Face7 minutes
Reward Modeling: Response Evaluation4 minutes
Reward Model Training 6 minutes
Reward Modeling with Hugging Face7 minutes

3 readingsTotal 15 minutes

Course Overview3 minutes
Specialization Overview10 minutes
Summary and Highlights 2 minutes

2 assignmentsTotal 30 minutes

Different Approaches to Instruction-Tuning21 minutes
Practice Quiz: Instruction-Tuning and Reward Modeling 9 minutes

2 app itemsTotal 150 minutes

Instruction Fine-Tuning LLMs90 minutes
Lab: Reward Modeling60 minutes

1 pluginTotal 5 minutes

Helpful tips for Course Completion5 minutes

In this module, you’ll describe the applications of large language models (LLMs) to generate policies and probabilities for generating responses based on the input text. You’ll also gain insights into the relationship between the policy and the language model as a function of omega to generate possible responses. Further, this module will demonstrate how to calculate rewards using human feedback incorporating reward function, train response samples, and evaluate agent’s performance. You’ll also define the scoring function for sentiment analysis using PPO with Hugging Face. You’ll also explain the PPO configuration class for specific models and learning rate for PPO training and how the PPO trainer processes the query samples to optimize the chatbot’s policies to get high-quality responses. This module delves into direct preference optimization (DPO) concepts to provide optimal solutions for the generated queries based on human preferences more directly and efficiently using Hugging Face. The labs in this module provide hands-on practice on human feedback and DPO.