We may earn an affiliate commission when you visit our partners.
Course image
Sebastian Witalec

Learn how to build multimodal search and RAG systems. RAG systems enhance an LLM by incorporating proprietary data into the prompt context. Typically, RAG applications use text documents, but, what if the desired context includes multimedia like images, audio, and video? This course covers the technical aspects of implementing RAG with multimodal data to accomplish this.

1. Learn how multimodal models are trained through contrastive learning and implement it on a real dataset.

2. Build any-to-any multimodal search to retrieve relevant context across different data types.

Read more

Learn how to build multimodal search and RAG systems. RAG systems enhance an LLM by incorporating proprietary data into the prompt context. Typically, RAG applications use text documents, but, what if the desired context includes multimedia like images, audio, and video? This course covers the technical aspects of implementing RAG with multimodal data to accomplish this.

1. Learn how multimodal models are trained through contrastive learning and implement it on a real dataset.

2. Build any-to-any multimodal search to retrieve relevant context across different data types.

3. Learn how LLMs are trained to understand multimodal data through visual instruction tuning and use them on multiple image reasoning examples.

4. Implement an end-to-end multimodal RAG system that analyzes retrieved multimodal context to generate insightful answers.

5. Explore industry applications like visually analyzing invoices and flowcharts to output structured data.

6. Create a multi-vector recommender system that suggests relevant items by comparing their similarities across multiple modalities.

As AI systems increasingly need to process and reason over multiple data modalities, learning how to build such systems is an important skill for AI developers.

This course equips you with the key skills to embed, retrieve, and generate across different modalities. By gaining a strong foundation in multimodal AI, you’ll be prepared to build smarter search, RAG, and recommender systems.

Enroll now

What's inside

Syllabus

Traffic lights

Read about what's good
what should give you pause
and possible dealbreakers
Develops expertise and a strong foundation in building multimodal search, RAG systems, and recommender systems
Focuses on practical, immediately-applicable skills and uses real-world applications to illustrate concepts
Taught by Sebastian Witalec, a widely-recognized researcher in the field
Covers core topics like contrastive learning, representation learning, retrieval techniques, and system evaluation for multimodal systems
Provides a solid foundation for professionals working in or aspiring to work in AI system development, search technology, information retrieval, computer vision, natural language processing, and recommendation systems
Involves hands-on, practice-oriented learning through projects and assignments
May not be suitable for complete beginners in AI or deep learning

Save this course

Create your own learning path. Save this course to your list so you can find it easily later.
Save

Reviews summary

Hands-on multimodal rag development

According to learners, "Building Multimodal Search and RAG" is a highly practical course that provides an excellent foundation in a cutting-edge field. Students consistently praise the well-structured content and clear explanations, which effectively demystify complex concepts. The hands-on labs are frequently highlighted as a major strength, allowing learners to implement real-world applications of multimodal RAG systems. While the course is lauded for its relevance and practical depth, prospective students should note that it assumes prior machine learning knowledge, making it most suitable for those with an existing technical background.
Mixed feedback on lab quality, though overwhelmingly positive.
"The labs are really good, well-structured, and incredibly helpful for cementing understanding."
"I found the hands-on labs simply amazing, truly providing valuable practical knowledge."
"I was disappointed because the labs were not well prepared and challenging to follow."
Covers highly relevant and up-to-date topics in multimodal AI.
"The content is very practical and up-to-date, addressing current industry needs effectively."
"I appreciate the extremely relevant and up-to-date content in this rapidly evolving field."
"This course covers a very important and timely topic in AI that will be crucial for future development."
Complex multimodal concepts are explained with clarity and conciseness.
"The instructor explains complex topics simply and makes them incredibly understandable."
"I found the material clear, concise, and incredibly well-structured, making learning easy."
"Instructors are excellent at breaking down challenging multimodal concepts into digestible parts."
Provides excellent hands-on experience for real-world application.
"The hands-on labs are simply amazing, providing practical knowledge and directly applying concepts."
"I loved the hands-on coding and projects; they were the strongest part of the course for me."
"The course truly shines in its practical application, offering great real-world examples and use cases."
Requires a solid background in ML, Python, and RAG concepts.
"It requires a solid background in machine learning and Python programming to fully grasp the concepts."
"I felt a good understanding of LLMs, embeddings, and RAG concepts was essential before starting."
"The pacing might be a bit fast if you are very new to deep learning or ML ops."

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Building Multimodal Search and RAG with these activities:
Review foundational concepts in machine learning
Strengthen the understanding of machine learning algorithms and techniques used in multimodal AI.
Browse courses on Machine Learning
Show steps
  • Revisit key concepts from previous courses or books.
  • Solve practice problems and exercises.
Gather resources on multimodal AI applications
Expand knowledge and stay updated with the latest advancements in multimodal AI.
Show steps
  • Collect articles, research papers, and industry whitepapers.
  • Organize and categorize the resources for easy reference.
Explore open-source multimodal AI libraries and tools
Gain practical experience by working with industry-standard tools and resources.
Show steps
  • Identify and install relevant libraries.
  • Follow tutorials and documentation to understand their functionality.
Five other activities
Expand to see all activities and additional details
Show all eight activities
Practice search and retrieval tasks
Develop proficiency in retrieving relevant context from multimodal data through repetitive practice.
Show steps
  • Define search queries and retrieval criteria.
  • Use provided codebase to implement search and retrieval algorithms.
  • Evaluate results and refine search strategies.
Discuss and share insights on multimodal AI
Engage with peers to exchange ideas and perspectives on the latest advancements in multimodal AI.
Show steps
  • Join online forums or discussion groups.
  • Attend virtual meetups or webinars.
Implement RAG with multimodal data
Enhance understanding of how RAG systems incorporate multimodal data into prompt context.
Show steps
  • Modify provided RAG codebase to handle multimodal inputs.
  • Test and evaluate the performance of multimodal RAG.
Build a multimodal recommender system
Apply multimodal AI concepts to create a practical and useful recommender system.
Show steps
  • Design the system architecture and data structures.
  • Implement the system using provided codebase.
  • Evaluate the system's performance and user experience.
Contribute to open-source multimodal AI projects
Gain practical experience and contribute to the development of multimodal AI technologies.
Show steps
  • Identify open-source projects aligned with interests.
  • Review code and documentation to understand the project.
  • Submit bug reports or feature requests.
  • Contribute code or documentation improvements.

Career center

Learners who complete Building Multimodal Search and RAG will develop knowledge and skills that may be useful to these careers:

Reading list

We've selected 14 books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Building Multimodal Search and RAG.
LangChain widely used framework for developing applications powered by language models, including RAG systems and AI agents. offers a practical guide to using LangChain and LangGraph, providing hands-on experience with implementing RAG and related concepts.
Transformers are the fundamental architecture behind most modern Large Language Models used in RAG. provides a comprehensive guide to working with transformers using the Hugging Face ecosystem. It offers essential background knowledge for understanding the generative component of RAG systems.
Is specifically designed for beginners interested in Retrieval Augmented Generation (RAG). It aims to introduce the core concepts and guide readers in building basic RAG systems. It serves as a good starting point for those with minimal prior knowledge of RAG.
Large Language Models are a core component of RAG. provides a practical, hands-on guide to understanding and working with LLMs, covering transformers, tokenizers, and semantic search. It offers essential background knowledge for anyone building RAG systems and valuable reference for practitioners.
This project-based book helps solidify understanding of Large Language Models and their applications, including the use of Vector Databases and LangChain, which are highly relevant to RAG implementation. It provides practical experience through building various LLM projects.
For those wanting a deep understanding of the generation component in RAG, this book guides you through building an LLM from scratch. It covers the internal workings, limitations, and customization methods. While challenging, it provides a solid foundation in LLM architecture and training.
Aimed at beginners, this book provides a comprehensive roadmap to understanding Retrieval Augmented Generation (RAG) technology. It covers core principles, architecture, training processes, and real-world applications. This valuable resource for those new to RAG looking to gain foundational knowledge.
Explores vector databases and their role in modern data analytics, including applications in AI and generative AI. Understanding vector databases is fundamental to building efficient RAG systems. This book provides a good overview of this essential technology.
Authored by the creator of Keras, this book provides an accessible introduction to deep learning with Python. The third edition includes updated content on generative AI and large language models, making it highly relevant for understanding the generation aspect of RAG. It helps solidify understanding through practical examples.
Focuses specifically on applying deep learning techniques to natural language processing problems. It delves into advanced concepts and models relevant to the generative component of RAG systems. It is suitable for those looking to deepen their understanding of the neural network architectures used in NLP.
Considered a classic in the field, this book provides a broad and deep understanding of Natural Language Processing. While not specifically about RAG, it lays crucial groundwork in areas like text processing, language models (earlier forms), and information extraction, which are prerequisites for understanding RAG. It is commonly used as a textbook in academic institutions.
While not directly about RAG architecture, effective prompt engineering is crucial for optimizing the output of the LLM within a RAG system. provides practical guidance on crafting prompts to achieve reliable results from generative AI models, enhancing the overall performance of a RAG application.
Offers a practical introduction to Natural Language Processing using Python and the NLTK library. It's excellent for beginners to gain hands-on experience with fundamental NLP tasks like text processing and analysis, which are helpful prerequisites for working with RAG systems.

Share

Help others find this course page by sharing it with your friends and followers:

Similar courses

Here are nine courses similar to Building Multimodal Search and RAG.
Leveraging Llama2 for Advanced AI Solutions
Most relevant
Harnessing LLMs & Text-Embeddings API with Google Vertex...
Most relevant
Securing Generative AI
Most relevant
Programming Generative AI: Unit 1
Most relevant
Large Multimodal Model Prompting with Gemini
Most relevant
Build an LLM and RAG-based Chat Application using AlloyDB...
Most relevant
End to End LLMs with Azure
Most relevant
Getting Started with Amazon Bedrock
Most relevant
Generative AI Applications with Amazon Bedrock
Most relevant

Coursera Buyer Protection & Guarantee

Verified partner policies & purchase protections

Accredited University Partner
Free Trial & Audit

7-Day Free Trial & Free Audit

7-day free trial on monthly Coursera Plus subscriptions; free audit option available on most individual courses to explore video lectures and readings at no charge.

Return & Refund Policy

14-Day Refund Guarantee

14-day refund window on annual Coursera Plus subscriptions and single-course purchases prior to certificate issuance. Monthly subscriptions do not offer refunds once billed.

Buyer Protections

Verified Academic & Industry Partner

Accredited university and industry certifications (e.g. Stanford, Google, IBM, Yale), secure SSL payment processing, and verifiable digital certificate links.

Key Purchase Protections
14-day refund window on annual Coursera Plus and single courses
7-day free trial on monthly Coursera Plus subscriptions
Free course audit option for lectures and readings
Verified certificates from top universities & companies
OpenCourser Verified Partner Policy details last reviewed: 2026-10-22
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2025 OpenCourser