Design rigorous evaluation systems that turn expert judgment into measurable, repeatable AI quality signals.
AI teams can change a prompt, swap a model, add context, or redesign an agent in hours. But without a rigorous evaluation system, they cannot tell whether the change improved the product—or simply moved the failures somewhere less visible.
Traditional software testing asks whether the same input produces the expected output. Generative AI requires a harder question: is this behavior useful, reliable, safe, and good enough for this user and this decision?
Answering that requires representative test cases, explicit quality dimensions, consistent expert judgment, calibrated automated graders, failure analysis, and release thresholds the whole team trusts.
Without those signals, teams debate anecdotes, optimize for demos, and discover regressions through angry users. With them, product, engineering, research, and leadership can make the same decision from the same evidence.
Translate a vague quality goal into explicit product decisions.
Test the experience your users will actually encounter.
Make expert review consistent enough to become a signal.
Automate evaluation without blindly trusting another model.
Turn a score into a prioritized product improvement.
Make quality part of every release—not an occasional audit.
Leave with more than a score. Produce the quality definition, representative dataset, calibrated judge, baseline results, failure diagnosis, and improvement plan your team needs to make a release decision.
Dates are the published cohort schedule.
Reah leads evaluation work on Google models. She brings the operating perspective required to turn ambiguous human judgment into quality signals teams can trust across product development and release decisions.
The course focuses on the difficult parts that simple benchmark tutorials skip: deciding what “good” means, building representative evidence, calibrating automated judges, resolving disagreement, diagnosing regressions, and making evaluation part of the product cadence.
Build each layer live with direct instruction and review.
Define quality, risk, metrics, and decision thresholds.
Create representative cases and consistent scoring criteria.
Compare automated evaluation with human expert judgment.
Run the system, slice results, and prioritize fixes.
Establish dashboards, release gates, monitoring, and cadence.
The Fellowship is built around live, cohort-based learning. You learn directly from the named practitioners, ask questions, complete applied work, and receive the latest insights from people working at the frontier.
Advanced Product Management is included as an on-demand program for strengthening timeless product fundamentals.
Most cohorts are designed for busy working professionals and require only a few hours per week. The exact commitment varies by program. You'll get the most value by attending live, completing the exercises, and making time for your capstone.
Attend live whenever possible because the discussions, feedback, and accountability are important parts of the experience.
However, missing an occasional session won't ruin your progress. You can also retake eligible cohorts during your active membership.
Yes. You can retake eligible cohorts at no additional cost while your membership remains active. This allows you to experience the updated curriculum, refresh your knowledge, and apply what you learn to a new project as AI evolves.
Yes. Many professionals use their learning and development budgets to cover the Fellowship.
We provide a manager-approval email template explaining the practical value to your company. Corporate cohort options are also available for teams. If your company needs an invoice, email support@productfaculty.com and we'll sort it.
Ship enterprise-ready AI products: PRDs, RAG, evals, agents.
Defensible AI strategy: moats, AI P&L, pricing, GTM.
Turn Claude and Claude Code into your daily PM team.
Ace the AI product interview, from product sense to strategy.
Pick your first cohort. Take the rest as they run.
See all 8 courses