We may earn an affiliate commission when you visit our partners.

Pretraining LLMs

In Pretraining LLMs you’ll explore the first step of training large language models using a technique called pretraining. You’ll learn the essential steps to pretrain an LLM, understand the associated costs, and discover how starting with smaller, existing open source models can be more cost-effective.

Pretraining involves teaching an LLM to predict the next token using vast text datasets, resulting in a base model, and this base model requires further fine-tuning for optimal performance and safety. In this course, you’ll learn to pretrain a model from scratch and also to take a model that’s already been pretrained and continue the pretraining process on your own data.

In detail:

1. Explore scenarios where pretraining is the optimal choice for model performance. Compare text generation across different versions of the same model to understand the performance differences between base, fine-tuned, and specialized pre-trained models.

2. Learn how to create a high-quality training dataset using web text and existing datasets, which is crucial for effective model pretraining.

3. Prepare your cleaned dataset for training. Learn how to package your training data for use with the Hugging Face library.

4. Explore ways to configure and initialize a model for training and see how these choices impact the speed of pretraining.

5. Learn how to configure and execute a training run, enabling you to train your own model.

6. Learn how to assess your trained model’s performance and explore common evaluation strategies for LLMs, including important benchmark tasks used to compare different models’ performance.

After taking this course, you’ll be equipped with the skills to pretrain a model—from data preparation and model configuration to performance evaluation.

Enroll now

Or subscribe to Coursera Plus

And get unlimited access to Coursera

Two deals to help you save

Save money when you learn. Here are two deals and offers that may be relevant to this course.

All coupon codes, vouchers, and discounts are applied automatically unless otherwise noted.

Valid until December 2

Annual Subscription

Invest in your career growth and save. Start your annual subscription to Coursera Plus and access 10,000+ programs.

Valid until March 12, 2027

Save 40% Off 3 Months

Access thousands of top-rated learning programs with a Coursera Plus monthly subscription and take 40% off 3 months for a limited time.

What's inside

Syllabus

Traffic lights

Read about what's good

what should give you pause

and possible dealbreakers

Provides an introduction to pretraining, which is a fundamental step in training large language models such as GPT-3 and BERT

Instructs learners in the process of creating a high-quality training dataset, which is essential for effective model pretraining

Guides learners on how to prepare their cleaned dataset for training and package it for use with the Hugging Face library, a popular tool for training and deploying machine learning models

Provides a comprehensive understanding of model configuration and initialization for training, allowing learners to optimize the training process

Enables learners to assess the performance of their trained model and explore common evaluation strategies for LLMs, including important benchmark tasks used to compare different models' performance

Save this course

Create your own learning path. Save this course to your list so you can find it easily later.

Save

Reviews summary

Practical pretraining for llm developers

According to students, this course offers a phenomenal deep dive into pretraining Large Language Models, providing highly relevant content crucial for current industry needs. Learners consistently praise the incredibly practical labs and hands-on approach, which effectively solidify understanding and enable pretraining on custom data, particularly leveraging the Hugging Face library. While the course delivers a solid foundation in data preparation, model configuration, and evaluation, a common point highlighted is that it assumes a certain level of prerequisite knowledge in deep learning and Python, making it challenging for beginners. Nevertheless, it is considered extremely valuable for those with prior machine learning experience.

Explanations are clear, topics are current and essential for LLMs.

"Truly exceptional course that covers pretraining a model from scratch and continuing pretraining. The content is cutting-edge."

"Excellent course! The content is highly relevant to current industry needs."

"The instructors explain complex topics like data tokenization and distributed training very clearly."

"The course really helps in understanding the nuances of how LLMs are built from scratch, or how to continue pretraining."

Reinforces concepts with real-world LLM pretraining tasks.

"The labs were incredibly practical, letting me pretrain a small model on custom data."

"I appreciated the practical demonstrations of pretraining and evaluation metrics. The labs are well-designed and reinforce the concepts."

"The hands-on coding and projects are the strongest part of the course for me; it truly solidified my understanding."

"I particularly enjoyed the section on dataset creation – very practical advice for my own work."

Can be fast-paced, some labs might need more detailed troubleshooting.

"My only minor gripe is that some parts felt a bit fast-paced, especially if you're not already familiar with Hugging Face Transformers."

"The labs were difficult to follow without extensive prior knowledge, and I felt lost at times."

"The labs could be more robust, and sometimes the instructions felt a bit vague, leading to frustration. Needs better support for troubleshooting."

Best suited for learners with existing ML/Deep Learning background.

"This course is a phenomenal deep dive... Highly recommended for intermediate learners."

"Good course, though it assumes a certain level of prerequisite knowledge. It doesn't hold your hand."

"The course felt too advanced for me. I have some ML background but found the jump into LLM pretraining too steep."

"While the course covers essential topics, I struggled with some coding assignments without a strong background."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Pretraining LLMs with these activities:

Python Refresher

Show steps

Brush up on your Python skills before starting the course to ensure a strong foundation and minimize any potential obstacles in your learning journey.

Browse courses on Python

Show steps

Review basic syntax and data structures
Practice coding exercises
Complete online tutorials or refresher courses
Build a small project to apply your skills

Subject Matter Experts

Show steps

Seek out subject matter experts beyond your instructor to gain additional insights and perspectives on the course material, broadening your understanding and expanding your professional network.

Browse courses on Mentorship

Show steps

Identify potential mentors in the field
Reach out and introduce yourself
Schedule meetings or discussions
Seek guidance and advice

Peer-to-Peer Training

Show steps

Engage in peer-to-peer training by mentoring other students in the class, reinforcing your understanding of the course material and developing your leadership and communication skills.

Browse courses on Communication

Show steps

Identify a student who could benefit from your support
Schedule regular sessions to review course material
Provide constructive feedback and guidance
Create practice exercises and assignments

Five other activities

Expand to see all activities and additional details

Show all eight activities

GPT-3 Response Generator

Show steps

Build a GPT-3 based response generator using Hugging Face to practice your text generation skills and improve the quality of your responses in the course.

Browse courses on Hugging Face

Show steps

Install Hugging Face and transformers library
Load a pre-trained text generator model from Hugging Face
Write a function to generate text using the model
Create a simple web app that uses your function to generate responses

Fine-tuning for Specific Tasks

Show steps

Follow guided tutorials to fine-tune pre-trained models for specific tasks, allowing you to apply the concepts learned in the course and enhance your problem-solving skills.

Browse courses on Fine-tuning

Show steps

Identify a specific task and dataset
Select an appropriate pre-trained model
Fine-tune the model on your task
Evaluate the performance of the fine-tuned model

Pretrained Model Performance Evaluation

Show steps

Engage in practice drills to evaluate the performance of different pre-trained models on various benchmark tasks to reinforce your understanding of model evaluation techniques and gain practical experience.

Browse courses on Model Evaluation

Show steps

Create a dataset for model evaluation
Select relevant evaluation metrics
Implement model evaluation code
Evaluate and compare the performance of multiple models
Analyze the results and identify areas for improvement

State-of-the-Art Pretraining Techniques

Show steps

Compile resources and articles on state-of-the-art pre-training techniques to expand your knowledge and stay updated with the latest advancements in the field of Large Language Models.

Show steps

Conduct a literature search for relevant papers and articles
Summarize the key findings and insights
Organize and categorize the resources
Create a presentation or document to share your findings

Contribute to Hugging Face

Show steps

Engage with the open-source community by contributing to Hugging Face, the leading platform for Large Language Models, to gain practical experience and deepen your understanding of the ecosystem.

Browse courses on Open Source

Show steps

Find an issue or feature to work on
Fork the repository
Implement your changes
Submit a pull request

Career center

Learners who complete Pretraining LLMs will develop knowledge and skills that may be useful to these careers:

Reading list

We've selected 50 books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Pretraining LLMs.

Fluent Python

Save

Serves as the ultimate masterclass in idiomatic Python, guiding developers through data models, generators, metaprogramming, and concurrent concurrency patterns.

Pretraining LLMs

Two deals to help you save

What's inside

Syllabus

Traffic lights

Save this course

Reviews summary

Practical pretraining for llm developers

Activities

Career center

Reading list

Share

Similar courses

Coursera Buyer Protection & Guarantee

7-Day Free Trial & Free Audit

14-Day Refund Guarantee

Verified Academic & Industry Partner