> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://learning.postman.com/llms.txt. # Evaluate AI output with evals *Evals* automatically grade AI-generated output against qualitative criteria, using an AI model as a judge. Use evals to check qualities that don't have a single correct answer, such as friendliness, safety, correctness, and relevance. Each eval returns a score and a short reason, which the rest of your flow can act on. ## Tests compared to evals Postman Flows supports two ways to check the output of a block: * [*Tests*](/docs/tests-and-scripts/tests-and-scripts/) use `pm.test` for deterministic checks that have a definite outcome. A test asserts something that's either true or false, such as whether a response has a 200 status code or whether a field is present. * *Evals* use `pm.eval` or the built-in evals on the **AI Agent** block for criteria that require model judgment. An eval sends the output and a criterion to a judge model, which returns a score and a short reason. Use a test when you can express the check as an assertion. Use an eval when grading the output requires interpretation, such as deciding whether a response is friendly, safe, or relevant to the user's question. ## Eval criteria Both the **AI Agent** block and the **Evaluate** block grade against the same five preset criteria, plus a custom criterion you write yourself. On the **Evaluate** block, each criterion has a matching [`pm.eval`](/flows/reference/blocks/evaluate/#define-evals-with-pmeval) method. | Criterion | `pm.eval` method | What the judge grades | Context the judge needs | | ---------------- | ---------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | **Friendliness** | `pm.eval.friendliness` | Whether the tone is warm, polite, and approachable. | None. | | **Safety** | `pm.eval.safety` | Whether the output is free of harmful or policy-violating content. | None. | | **Non-toxicity** | `pm.eval.nonToxicity` | Whether the output is free of insults, harassment, and profanity. | None. | | **Correctness** | `pm.eval.correctness` | Whether the output is factually accurate and consistent with a known-correct answer. | A known-correct answer: a `reference` on the **Evaluate** block, or the agent context on the **AI Agent** block. | | **Relevance** | `pm.eval.relevance` | Whether the output directly addresses the task or question. | A statement of the task: a `query` on the **Evaluate** block, or the agent prompt on the **AI Agent** block. | | **Custom** | `pm.eval.custom` | Whether the output meets a criterion you write in natural language. | A `criterion`, plus any `context` keys you reference as `{{key}}`. | How the judge gets that context depends on the block, so a similar check can produce different results on each: * On the **AI Agent** block, the block automatically supplies the agent's prompt and inputs to the judge as context. This can give criteria such as **Correctness** and **Relevance** enough grounding to produce a score. If the automatic context isn't enough for a criterion, that criterion skips. * On the **Evaluate** block, you pass the context yourself in the `context`, `query`, and `reference` options of `pm.eval`. A criterion that doesn't get the context it needs skips instead of scoring. Any criterion also skips when the block produces no output to grade. If an eval scores lower than you expect, or skips, check whether the judge has the context it needs. For more on skipped evals, see [Behavior and limitations](#behavior-and-limitations). ## Where to create evals You can create evals in the **AI Agent** block and the **Evaluate** block. Each block has its own configuration options and defaults. For step-by-step instructions, see [Add evals to a flow](/flows/build-flows/ai/add-evals-to-a-flow/). ### AI Agent block The [**AI Agent** block](/flows/reference/blocks/ai-agent/#evals) grades its own output using a separate judge model, distinct from the model the block uses to generate that output. Turn on any of the preset criteria under **Quality presets**, add criteria of your own under **Custom**, or both. The passing threshold is fixed at 80 on a 0–100 scale, and the judge model is fixed at `gpt-4o-mini-2024-07-18`. The block automatically supplies the agent's prompt and inputs to the judge as context, which can give criteria such as **Correctness** and **Relevance** enough grounding to produce a score. ### Evaluate block In the [**Evaluate** block](/flows/reference/blocks/evaluate/#define-evals-with-pmeval), you define evals in a [TypeScript](/flows/reference/typescript/typescript-overview/) script with `pm.eval`. This block gives you more control than the **AI Agent** block: you can set a reference answer, a query, a custom threshold, and a different judge model. For the full option reference and script limits, see [The Evaluate block](/flows/reference/blocks/evaluate/#define-evals-with-pmeval). ## How eval results appear after a flow runs Evals run when the flow runs. You can't rerun them independently of the flow. Before a run, the block shows `No eval results yet. Run the flow to grade this response.` After a run, results appear in the following places: * **Eval summary on the block** — A preview shows a summary, such as `Passed 3/4`, with a row for each eval and its score. * **Per-eval scores and reasons** — Each graded eval returns a score and a short explanation of why the judge assigned it. A failed eval was graded but didn't meet its passing threshold. * **Evals output port** — The block sends structured eval results through a dedicated **Evals** output port, which you can connect to other blocks. Each graded result includes the eval's `status`, `score`, `threshold`, and `reason`. A skipped eval instead has a `skippedReason` that explains why it couldn't be graded. The payload also includes a `summary` with the `graded` and `passed` counts. See [The Evaluate block](/flows/reference/blocks/evaluate/#output). * **All evals tab** — The run log has an **All evals** tab, alongside **All tests**, that lists the eval results for all blocks in the flow that ran evals. To open the run log, click **View run logs** in the toolbar. For more information, see [Troubleshoot Postman Flows](/flows/build-flows/troubleshoot/troubleshoot/#run-logs). If grading can't complete, the flow run still completes. The block shows `Grading failed`, and the **All evals** tab in the run log marks the block as `Grading failed` with no evals graded. ## Behavior and limitations * Evals are opt-in. They don't run unless you configure them. * Each enabled eval is a separate model-judge call that consumes Flows credits, whether it's a preset or a custom eval. Postman prices and validates the whole run before judging starts. If your team can't cover the cost, Postman refuses the entire evaluation up front and doesn't charge for a partial run. See [Credit usage](#credit-usage). * Evals run on the canvas in the desktop and web apps, in local mode, and in deployed and scheduled flows. They aren't supported by the [Postman CLI](/docs/postman-cli/postman-cli-flows/). * Eval failures don't fail the flow. A flow run completes even if one or more evals don't pass. * A *skipped* eval isn't the same as a failing score. The judge skips an eval when it doesn't have the information it needs to grade the output, and skipped evals are left out of the summary counts and the average score. A judge skips an eval when: * The block returned no output to grade. * The judge doesn't have enough grounding or context to evaluate **Correctness**. * No task was provided to grade **Relevance** against. * A custom eval has no criterion. ## Credit usage Credits are consumed on every run of the block, whether an eval passes, fails, or is skipped, and presets and custom evals cost the same. For more information about how Flows uses credits, see [Manage your team's Postman Flows credit usage](/docs/billing/flows-usage/).