In Pretraining LLMs you’ll explore the first step of training large language models using a technique called pretraining. You’ll learn the essential steps to pretrain an LLM, understand the associated costs, and discover how starting with smaller, existing open source models can be more cost-effective.
![DeepLearning.AI](https://d3njjcbhbojbot.cloudfront.net/api/utilities/v1/imageproxy/http://coursera-university-assets.s3.amazonaws.com/b4/5cb90bb92f420b99bf323a0356f451/Icon.png?auto=format%2Ccompress&dpr=1&w=28&h=28)
![DeepLearning.AI](https://d3njjcbhbojbot.cloudfront.net/api/utilities/v1/imageproxy/http://coursera-university-assets.s3.amazonaws.com/1b/bdf48065584cbe8e096669d9dd4852/LogoFiles_DeepLearning_Coursera_200x48.png?auto=format%2Ccompress&dpr=1&h=45)
Pretraining LLMs
Recommended experience
What you'll learn
Gain in-depth knowledge of pretraining an LLM, covering data preparation, model configuration, and performance assessment.
Explore model architecture options, including modifying Meta’s Llama models for various sizes and initializing weights randomly or from other models.
Learn innovative pretraining techniques like Depth Upscaling, which can reduce training costs by up to 70%.
Details to know
July 2024
1 assignment
See how employees at top companies are mastering in-demand skills
![Placeholder](https://d3njjcbhbojbot.cloudfront.net/api/utilities/v1/imageproxy/https://coursera_assets.s3.amazonaws.com/images/74c8747e8210831049cf88dd4eefe26c.png?auto=format%2Ccompress&dpr=2&blur=200&px=8&max-w=320)
There is 1 module in this course
In Pretraining LLMs you’ll explore the first step of training large language models using a technique called pretraining. You’ll learn the essential steps to pretrain an LLM, understand the associated costs, and discover how starting with smaller, existing open source models can be more cost-effective.Pretraining involves teaching an LLM to predict the next token using vast text datasets, resulting in a base model, and this base model requires further fine-tuning for optimal performance and safety. In this course, you’ll learn to pretrain a model from scratch and also to take a model that’s already been pretrained and continue the pretraining process on your own data. In detail: 1. Explore scenarios where pretraining is the optimal choice for model performance. Compare text generation across different versions of the same model to understand the performance differences between base, fine-tuned, and specialized pre-trained models. 2. Learn how to create a high-quality training dataset using web text and existing datasets, which is crucial for effective model pretraining. 3. Prepare your cleaned dataset for training. Learn how to package your training data for use with the Hugging Face library. 4. Explore ways to configure and initialize a model for training and see how these choices impact the speed of pretraining. 5. Learn how to configure and execute a training run, enabling you to train your own model. 6. Learn how to assess your trained model’s performance and explore common evaluation strategies for LLMs, including important benchmark tasks used to compare different models’ performance. After taking this course, you’ll be equipped with the skills to pretrain a model—from data preparation and model configuration to performance evaluation.
What's included
1 assignment1 app item
Offered by
Recommended if you're interested in Software Development
Google Cloud
Google Cloud
DeepLearning.AI
DeepLearning.AI
Why people choose Coursera for their career
![](https://d3njjcbhbojbot.cloudfront.net/api/utilities/v1/imageproxy/https://coursera_assets.s3.amazonaws.com/growth_testimonials/passionate_learner/Felipe_Moitta.png?auto=format%2Ccompress&dpr=1&w=64&h=64&fit=crop)
![](https://d3njjcbhbojbot.cloudfront.net/api/utilities/v1/imageproxy/https://coursera_assets.s3.amazonaws.com/growth_testimonials/passionate_learner/Jennifer_John.png?auto=format%2Ccompress&dpr=1&w=64&h=64&fit=crop)
![](https://d3njjcbhbojbot.cloudfront.net/api/utilities/v1/imageproxy/https://coursera_assets.s3.amazonaws.com/growth_testimonials/passionate_learner/Larry_Tao_Wang_1.png?auto=format%2Ccompress&dpr=1&w=64&h=64&fit=crop)
![](https://d3njjcbhbojbot.cloudfront.net/api/utilities/v1/imageproxy/https://coursera_assets.s3.amazonaws.com/growth_testimonials/passionate_learner/Chaitanya_Anand.png?auto=format%2Ccompress&dpr=1&w=64&h=64&fit=crop)
New to Software Development? Start here.
![Placeholder](https://d3njjcbhbojbot.cloudfront.net/api/utilities/v1/imageproxy/https://coursera_assets.s3.amazonaws.com/images/7a1c0e2e779c1ff27cae62480adfe003.png?auto=format%2Ccompress&dpr=2&blur=200&px=8&max-w=120)
Open new doors with Coursera Plus
Unlimited access to 7,000+ world-class courses, hands-on projects, and job-ready certificate programs - all included in your subscription
Advance your career with an online degree
Earn a degree from world-class universities - 100% online
Join over 3,400 global companies that choose Coursera for Business
Upskill your employees to excel in the digital economy
Frequently asked questions
Access to lectures and assignments depends on your type of enrollment. If you take a course in audit mode, you will be able to see most course materials for free. To access graded assignments and to earn a Certificate, you will need to purchase the Certificate experience, during or after your audit. If you don't see the audit option:
The course may not offer an audit option. You can try a Free Trial instead, or apply for Financial Aid.
The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.