Evaluating AI Agents

Learn how to systematically assess and improve your AI agent’s performance in Evaluating AI Agents, a short course built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and Aman Khan, Director of Product.

When you’re building an AI Agent, an important part of the development process is evaluations or evals. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically—rather than relying on trial and error. 

With a systematic approach, you structure your evaluations to assess the performance of each component of the agent, as well as its end-to-end performance. For each component, you select the appropriate evaluators, testing examples, and metrics. This process helps you identify any areas of improvement so you can iterate on your agent during development and in production. 

In this course, you’ll build an AI agent, add observability to visualize and debug its steps, and evaluate its performance component-wise.

In detail, you’ll: 

• Distinguish between evaluating LLM-based systems and traditional software testing.
• Explore the basic structure of AI agents – routers, skills, and memory – and implement an AI  agent from scratch.
• Add observability to the agent by collecting traces of the steps taken by the agent and visualizing the traces.
• Choose the appropriate evaluator – code-based, LLM-as-a-Judge, and human annotations – for each component of the agent. 
• Set up evaluations for the skills and router decisions of the agent example using code-based and LLM-as-a-judge evaluators, by creating testing examples from collected traces and preparing detailed prompts for the LLM-as-a-judge.
• Compute a convergence score to evaluate if the example agent can respond to a query in an efficient number of steps.
• Run structured experiments to improve the performance of the agent by exploring changes to the prompt, LLM model, or the agent’s logic.
• Understand how to deploy these evaluation techniques to monitor the agent’s performance in production.

By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. 

Anyone who has basic Python knowledge and wants to learn to evaluate, troubleshoot, and improve AI agents effectively—both during development and in production. Familiarity with prompting an LLM model would be helpful but not required.