Building and Evaluating Data Agents

Learn how to build and evaluate a data agent in “Building and Evaluating Data Agents,” a course created in collaboration with Snowflake, and taught by Anupam Datta, AI Research Lead, and Josha Reini, Developer Advocate at Snowflake.

You’ll design a data agent that connects to data sources (databases, files) and performs web searches to respond to users’ queries. The agent will consist of sub-agents, each specialized in connecting to a particular data source, and other sub-agents that summarize or visualize the results. To answer a particular query, the agent will use a planner that identifies which sub-agents to call and in what order.

You’ll add observability to the agent’s workflow and evaluate the quality of its output. Using an LLM-as-a-judge approach, you’ll assess whether the final answer is relevant to the user’s query and grounded in the collected data. You’ll also evaluate the process by determining whether the agent’s goal, plan, and actions (GPA) are all aligned.

Finally, you’ll apply inline evaluations to evaluate the agent’s performance during runtime. At every retrieval step, you’ll evaluate if the collected data is relevant to the user’s query. The agent will use this evaluation score to decide if it needs to adjust its plan.

What you’ll do, in detail:

Understand what data agents are and how they can be trustworthy when their goal, plan, and actions are properly aligned.

Build a data agent that plans, performs web searches ,and visualizes or summarizes the results, using a multi-agent workflow implemented in LangGraph.

Expand the agent’s capabilities by adding a Cortex sub-agent that retrieves information from structured and unstructured data stored in Snowflake.

Add tracing to the agent’s workflow to log the steps it takes to answer a query.

Evaluate the context relevance of the retrieved results, the groundedness of the final answer, and its relevance to the user’s query.

Measure the alignment of the agent’s goal, plan, and actions (GPA) by computing metrics such as plan quality, plan adherence, logical consistency, and execution efficiency.

Improve the agent’s performance by adding inline evaluations and updating the agent’s prompt.

By the end, you’ll know how to build, trace, and evaluate a multi-agent workflow that plans tasks, pulls context from structured and unstructured data, performs web search, and summarizes or visualizes the final results.

What's inside

Syllabus

Career center

Learners who complete Building and Evaluating Data Agents will develop knowledge and skills that may be useful to these careers:

Artificial Intelligence Engineer

An Artificial Intelligence Engineer is at the forefront of designing, developing, and deploying intelligent systems. This course directly prepares you for the critical work involved in bringing sophisticated AI solutions to life by focusing on building and evaluating data agents that connect to various data sources, perform web searches, and respond to complex user queries. You will gain hands-on experience in architecting multi-agent workflows using tools like LangGraph, expanding capabilities with sub-agents, and integrating with platforms like Snowflake for structured and unstructured data retrieval. A core strength you will develop is the rigorous evaluation of AI agent performance, encompassing context relevance, groundedness, and the alignment of an agent’s goal plan and actions, which is crucial for building trustworthy AI applications.

See salaries and explore the career path for Artificial Intelligence Engineer

Machine Learning Engineer

As a Machine Learning Engineer, you develop and implement scalable machine learning models and systems. This course will significantly enhance your capabilities in building and evaluating data agents, which are advanced applications of machine learning. You will learn to construct agents with sophisticated planning capabilities, integrating various components like web search and data summarization or visualization. The experience of building multi-agent workflows and expanding their functionality, such as connecting to Snowflake for diverse data retrieval, directly mirrors the practical challenges ML Engineers face. Furthermore, the emphasis on adding observability, tracing workflows, and performing detailed evaluations—including inline evaluations and assessing goal plan and action alignment—is vital for ensuring the robustness and reliability of deployed machine learning systems.

See salaries and explore the career path for Machine Learning Engineer

Software Engineer (Artificial Intelligence)

A Software Engineer Artificial Intelligence builds and maintains the core software infrastructure for AI systems. This course equips you with specialized skills for a Software Engineer Artificial Intelligence by focusing on the practical construction and assessment of data agents. You will engage in building multi-agent workflows using frameworks like LangGraph, expanding agent capabilities with sub-agents, and integrating with external data sources, including structured and unstructured data in Snowflake. The emphasis on designing a planner to orchestrate sub-agents and adding tracing for workflow observability are essential software engineering practices. Learning to apply inline evaluations and measure metrics like execution efficiency directly contributes to developing high performance, reliable, and maintainable AI powered software solutions.

See salaries and explore the career path for Software Engineer (Artificial Intelligence)

Artificial Intelligence Quality Assurance Engineer

An Artificial Intelligence Quality Assurance Engineer specializes in testing and validating the reliability, performance, and correctness of AI systems. This course provides highly pertinent expertise for an Artificial Intelligence Quality Assurance Engineer, given its strong emphasis on the detailed evaluation of data agents. You will learn to employ an LLM as a judge approach to assess whether an agent’s final answer is relevant and grounded in data. Crucially, the course delves into measuring the alignment of an agent’s goal plan and actions by computing specific metrics such as plan quality, logical consistency, and execution efficiency. Furthermore, gaining experience with inline evaluations to assess context relevance during runtime is invaluable for ensuring the continuous quality and trustworthiness of AI agent deployments.

See salaries and explore the career path for Artificial Intelligence Quality Assurance Engineer

Artificial Intelligence Researcher

An Artificial Intelligence Researcher explores novel algorithms, models, and paradigms to advance the field of AI. This course is highly relevant for an Artificial Intelligence Researcher, particularly in the realm of agent based AI systems. You will engage in building and evaluating data agents that employ multi agent workflows and sophisticated planning mechanisms. The detailed evaluation methodologies, including assessments of context relevance, groundedness, and the alignment of an agent’s goal plan and actions, provide a strong foundation for developing new metrics and experimental designs for agent performance. This experience can inspire research into more robust, trustworthy, and efficient agent architectures and evaluation techniques, pushing the boundaries of autonomous AI. This role typically requires an advanced degree.

See salaries and explore the career path for Artificial Intelligence Researcher

MLOps Engineer

An MLOps Engineer is responsible for deploying, monitoring, and maintaining machine learning models and systems in production environments. This course provides highly relevant skills for an MLOps Engineer by focusing on the operational aspects of artificial intelligence agents. You will learn to add tracing to an agent’s workflow to log its steps, which is fundamental for observability and debugging in a production setting. The course's detailed approach to evaluating agent performance, including inline evaluations during runtime and measuring metrics like plan quality and execution efficiency, directly translates to monitoring and improving the agents once they are deployed. Understanding how to assess the alignment of an agent’s goal plan and actions also helps ensure operational trustworthiness and reliability, which is paramount for successful MLOps practices.

See salaries and explore the career path for MLOps Engineer

Applied Artificial Intelligence Scientist

An Applied Artificial Intelligence Scientist bridges AI research and practical application, developing innovative AI solutions for real world problems. This course is highly relevant for an Applied Artificial Intelligence Scientist, providing hands on experience in building and evaluating data agents, which are cutting edge AI systems. You will learn to design multi agent workflows that plan tasks, retrieve context from diverse data sources, perform web searches, and synthesize results. The rigorous evaluation methodologies taught, including assessing context relevance, groundedness, and the alignment of an agent’s goal plan and actions, are fundamental for ensuring the practical efficacy and trustworthiness of deployed AI. This understanding is key to translating theoretical AI advancements into robust and impactful applications. This role typically requires an advanced degree.

See salaries and explore the career path for Applied Artificial Intelligence Scientist

Solutions Architect Artificial Intelligence

A Solutions Architect Artificial Intelligence designs and oversees the implementation of complex AI solutions. This course is highly beneficial for a Solutions Architect Artificial Intelligence as it provides a practical framework for building and evaluating sophisticated data agents. You will learn to design multi-agent workflows, identify sub-agents for specific tasks like data connection, summarization, or visualization, and integrate them with various data sources, including structured and unstructured data within Snowflake. This architectural understanding is crucial for creating scalable and effective AI systems. Furthermore, the course teaches you how to add observability and conduct thorough evaluations, assessing metrics such as goal plan and action alignment, which are essential for ensuring the robustness, performance, and trustworthiness of the overall solution architecture.

See salaries and explore the career path for Solutions Architect Artificial Intelligence

Developer Advocate Artificial Intelligence

A Developer Advocate Artificial Intelligence inspires and helps developers utilize AI technologies effectively, often requiring deep technical understanding and excellent communication. This course provides highly relevant, hands on experience for a Developer Advocate Artificial Intelligence by teaching the end to end process of building and evaluating data agents. You will master architecting multi agent workflows, integrating with data sources like Snowflake, and implementing observability and tracing. The ability to articulate how agents plan, perform web searches, summarize, and visualize results, along with the critical methods for evaluating their performance—including goal plan and action alignment and inline evaluations—will empower you to guide other developers. Understanding these technical nuances is essential for creating compelling content and providing robust support to the developer community.

See salaries and explore the career path for Developer Advocate Artificial Intelligence

Natural Language Processing Engineer

A Natural Language Processing Engineer develops systems that enable computers to understand, interpret, and generate human language. This course is directly relevant to a Natural Language Processing Engineer as it involves critical NLP tasks within the context of data agents. You will gain experience in designing agents that respond to users’ queries, summarize results, and use an LLM as a judge approach to assess the quality of generated text. The course's focus on evaluating the groundedness of final answers and their relevance to user queries is fundamental to NLP quality assurance. Additionally, the ability to pull context from various data sources and perform web searches to inform agent responses significantly enhances an NLP Engineer's capacity to build robust language driven applications.

See salaries and explore the career path for Natural Language Processing Engineer

Prompt Engineer

A Prompt Engineer specializes in crafting, refining, and optimizing prompts for large language models to achieve desired outputs and behaviors. This course offers direct and practical experience relevant to a Prompt Engineer by explicitly covering the improvement of an agent’s performance through updating its prompt. You will gain a deep understanding of how prompt design influences an agent’s decision-making and output quality, particularly in scenarios involving planning, web searches, data summarization, and visualization. The course emphasizes evaluating the relevance and groundedness of agent responses using an LLM as a judge approach, which provides a concrete framework for assessing the effectiveness of prompt modifications. This detailed evaluation capability empowers you to systematically iterate and enhance prompts for optimal agent performance.

See salaries and explore the career path for Prompt Engineer

Artificial Intelligence Ethics and Trustworthiness Specialist

An Artificial Intelligence Ethics and Trustworthiness Specialist focuses on ensuring AI systems are developed and deployed responsibly, considering ethical implications and building user trust. This course is particularly valuable for an Artificial Intelligence Ethics and Trustworthiness Specialist because it explicitly addresses how data agents can be trustworthy when their goal plan and actions are properly aligned. You will learn to evaluate the groundedness of an agent’s final answer and its relevance to user queries, which are crucial aspects of transparency and accountability. The course’s detailed methodology for measuring the alignment of an agent’s goal plan and actions, through metrics like plan quality and logical consistency, provides concrete tools for assessing and improving the ethical behavior and reliability of AI agents, fostering greater trust in intelligent systems.

See salaries and explore the career path for Artificial Intelligence Ethics and Trustworthiness Specialist

Artificial Intelligence Product Manager

An Artificial Intelligence Product Manager defines the strategy, roadmap, and features for AI powered products. This course provides a comprehensive understanding of the technical intricacies involved in building and evaluating data agents, which is invaluable for an AI Product Manager. While not focused on coding, knowing how complex multi-agent workflows are designed, how they connect to diverse data sources including structured and unstructured data in Snowflake, and how their outputs are summarized or visualized, allows for more informed product decisions. The emphasis on evaluating agent performance, including trustworthiness, relevance, and groundedness, helps in defining quality metrics and understanding product limitations. This insight enables effective communication with engineering teams and clearer feature definitions for AI products.

See salaries and explore the career path for Artificial Intelligence Product Manager

Data Engineer

A Data Engineer builds and maintains robust data pipelines and infrastructure. While the primary focus of a Data Engineer is often on data movement and storage, this course may be useful by focusing on how data agents interact with diverse data sources. You will learn how agents connect to databases and files, retrieve information from structured and unstructured data stored in Snowflake via a Cortex sub agent, and pull context for user queries. This understanding of how intelligent systems consume and process data provides valuable insight into designing data infrastructure that efficiently serves AI applications. Knowledge of agents’ data retrieval and summarization capabilities helps in optimizing data delivery for AI driven insights and applications.

See salaries and explore the career path for Data Engineer

Data Scientist

A Data Scientist analyzes complex datasets to extract insights, build predictive models, and inform strategic decisions. This course may be useful for a Data Scientist by providing an understanding of how data agents can programmatically gather, process, and summarize information from various sources, including web searches and structured data in Snowflake. While not focused on core statistical modeling, the course's emphasis on evaluating the relevance of collected data and the groundedness of agent outputs is directly applicable to data quality and validation processes a Data Scientist frequently undertakes. Understanding the mechanisms of how agents generate responses can help a Data Scientist better interpret and leverage outputs from AI driven information retrieval systems for their analytical tasks.

See salaries and explore the career path for Data Scientist