An eval is a repeatable test of whether AI output is good by your own standard. For a marketing team, that means writing down what "on brand", "on brief" and "accurate" actually mean, and checking real output against it every week, instead of trusting that a draft looks fine.
Product teams building AI have learned this the hard way. In an episode of Lenny’s Podcast, Hamel Husain and Shreya Shankar, who teach the most popular course on the subject and have trained more than 2,000 product managers and engineers, call evals the hottest new skill for product builders. Their process was designed for people shipping AI products. It transfers almost unchanged to anyone using AI to produce marketing work, and the episode and its transcript are worth an hour of any marketing lead’s time.
Why marketing needs evals now
AI has made drafts cheap. A team that used to write ten social posts a week can now produce a hundred, plus the brief, the reply to the client and the competitor summary. What has not grown is the time anyone has to read them.
The failures that matter in marketing rarely look like failures. A draft reads well and still drifts from the brand’s tone, invents a statistic, ignores a mandatory line from the brief or makes a claim legal would never approve. Nobody notices until a client, a regulator or a customer does. "It looks fine" is not a quality bar when the output is fluent by default.
Start by reading the output, not by writing a rubric
The most useful advice in the episode is the order of operations: always start with error analysis. Before writing a single test, pull a sample of real outputs and read them.
For a marketing team, that might be fifty recent AI drafts: social copy, brief summaries, client replies, product descriptions. One person reads each one and writes a short note on what is wrong with it, in plain words. "Too formal for this brand." "Made up a market size." "Missed the product launch date." Husain and Shankar are firm that a person has to do this first pass; a model cannot tell you what your standard is.
It is tempting to skip this and start from the brand book. Don’t. The brand book says what good looks like. Error analysis tells you how your AI actually fails, which is rarely what you expected.
Turn notes into categories
The method borrows two steps from qualitative research. First, open coding: every note gets a free-form label. Then axial coding: group the labels into a handful of categories, usually five or six. In marketing those categories tend to look like this:
- Tone drifts from the brand voice.
- Invents a fact, number or quote.
- Ignores a mandatory from the brief.
- Uses a banned word or a competitor’s name.
- Writes for the wrong audience or market.
- Breaks a format rule: length, structure, required disclaimer.
Keep reading new outputs until they stop revealing new categories. Researchers call that point theoretical saturation. In practice it often arrives sooner than people expect.
Give one person the final say
Husain and Shankar recommend a "benevolent dictator": one person who decides what counts as a failure. In a marketing team, that is the brand lead for brand questions, or the account lead for a client’s work. A rubric agreed by committee ends up too vague to apply. A rubric owned by one person who knows the brand gets applied the same way every time.
Two kinds of checks
Once you know your failure categories, each one becomes a check. There are two kinds.
Code-based checks are for anything a rule can decide. Character limits for each channel, banned words, required disclaimers and legal lines, mandatory hashtags, UTM parameters in the right format, a competitor’s name in the copy. These are cheap, exact and never get tired. Use them wherever you can.
LLM-as-judge checks are for the things that need judgement: is this on brand, does it answer the brief, does it suit the audience. The episode’s advice is to write one judge per failure mode with a pass or fail answer, rather than one judge that scores everything out of ten. A narrow question gets a reliable answer.
Then check the judge. Take outputs your benevolent dictator has already marked as pass or fail, run the judge on them, and see how often it agrees. Trust a judge only once it agrees with the person who owns the standard.
Your brand book, made runnable
One idea from the episode lands especially well in marketing: well-written eval prompts become living requirements documents. Product teams call them living PRDs. For a marketing team, a set of evals is a brand book you can run. It does not sit in a PDF waiting to be read; it checks every draft against the standard, and it changes when the standard changes.
That also makes onboarding easier. A new team member, a freelancer or a new agency partner can see exactly what "on brand" means for this client, as tests rather than adjectives.
What it costs to keep going
The setup takes a few focused sessions. After that, the guests put the upkeep at around thirty minutes a week: read a fresh sample of outputs, note anything new, and add a check when a new kind of failure appears. That is less time than a single round of client feedback on a draft that should never have been sent.
Where to start this week
- Pick one kind of AI output your team produces often, such as social copy or client replies.
- Pull fifty recent examples and have one person write a note on each.
- Group the notes into five or six failure categories.
- Write code checks for the rule-based categories: length, banned words, disclaimers.
- Write one pass or fail judge for each category that needs judgement, and check it against your own labels.
- Put thirty minutes in the calendar every week to read a new sample.
If you want to go deeper, start with the episode and its transcript, then Lenny’s follow-up guide, Building eval systems that improve your AI product.
