Evaluate AI output with evals

Evals automatically grade AI-generated output against qualitative criteria, using an AI model as a judge. Use evals to check qualities that don’t have a single correct answer, such as friendliness, safety, correctness, and relevance. Each eval returns a score and a short reason, which the rest of your flow can act on.

Tests compared to evals

Postman Flows supports two ways to check the output of a block:

• Tests use pm.test for deterministic checks that have a definite outcome. A test asserts something that’s either true or false, such as whether a response has a 200 status code or whether a field is present.
• Evals use pm.eval or the built-in evals on the AI Agent block for criteria that require model judgment. An eval sends the output and a criterion to a judge model, which returns a score and a short reason.

Use a test when you can express the check as an assertion. Use an eval when grading the output requires interpretation, such as deciding whether a response is friendly, safe, or relevant to the user’s question.

Eval criteria

Both the AI Agent block and the Evaluate block grade against the same five preset criteria, plus a custom criterion you write yourself. On the Evaluate block, each criterion has a matching pm.eval method.

How the judge gets that context depends on the block, so a similar check can produce different results on each:

• On the AI Agent block, the block automatically supplies the agent’s prompt and inputs to the judge as context. This can give criteria such as Correctness and Relevance enough grounding to produce a score. If the automatic context isn’t enough for a criterion, that criterion skips.
• On the Evaluate block, you pass the context yourself in the context, query, and reference options of pm.eval. A criterion that doesn’t get the context it needs skips instead of scoring.

Any criterion also skips when the block produces no output to grade. If an eval scores lower than you expect, or skips, check whether the judge has the context it needs. For more on skipped evals, see Behavior and limitations.

Where to create evals

You can create evals in the AI Agent block and the Evaluate block. Each block has its own configuration options and defaults. For step-by-step instructions, see Add evals to a flow.

AI Agent block

The AI Agent block grades its own output using a separate judge model, distinct from the model the block uses to generate that output. Turn on any of the preset criteria under Quality presets, add criteria of your own under Custom, or both. The passing threshold is fixed at 80 on a 0–100 scale, and the judge model is fixed at gpt-4o-mini-2024-07-18. The block automatically supplies the agent’s prompt and inputs to the judge as context, which can give criteria such as Correctness and Relevance enough grounding to produce a score.

Evaluate block

In the Evaluate block, you define evals in a TypeScript script with pm.eval. This block gives you more control than the AI Agent block: you can set a reference answer, a query, a custom threshold, and a different judge model. For the full option reference and script limits, see The Evaluate block.

How eval results appear after a flow runs

Evals run when the flow runs. You can’t rerun them independently of the flow.

Before a run, the block shows No eval results yet. Run the flow to grade this response. After a run, results appear in the following places:

• Eval summary on the block — A preview shows a summary, such as Passed 3/4, with a row for each eval and its score.
• Per-eval scores and reasons — Each graded eval returns a score and a short explanation of why the judge assigned it. A failed eval was graded but didn’t meet its passing threshold.
• Evals output port — The block sends structured eval results through a dedicated Evals output port, which you can connect to other blocks. Each graded result includes the eval’s status, score, threshold, and reason. A skipped eval instead has a skippedReason that explains why it couldn’t be graded. The payload also includes a summary with the graded and passed counts. See The Evaluate block.
• All evals tab — The run log has an All evals tab, alongside All tests, that lists the eval results for all blocks in the flow that ran evals. To open the run log, click View run logs in the toolbar. For more information, see Troubleshoot Postman Flows.

If grading can’t complete, the flow run still completes. The block shows Grading failed, and the All evals tab in the run log marks the block as Grading failed with no evals graded.

Behavior and limitations

• Evals are opt-in. They don’t run unless you configure them.

• Each enabled eval is a separate model-judge call that consumes Flows credits, whether it’s a preset or a custom eval. Postman prices and validates the whole run before judging starts. If your team can’t cover the cost, Postman refuses the entire evaluation up front and doesn’t charge for a partial run. See Credit usage.

• Evals run on the canvas in the desktop and web apps, in local mode, and in deployed and scheduled flows. They aren’t supported by the Postman CLI.

• Eval failures don’t fail the flow. A flow run completes even if one or more evals don’t pass.

• A skipped eval isn’t the same as a failing score. The judge skips an eval when it doesn’t have the information it needs to grade the output, and skipped evals are left out of the summary counts and the average score. A judge skips an eval when:

  • The block returned no output to grade.
  • The judge doesn’t have enough grounding or context to evaluate Correctness.
  • No task was provided to grade Relevance against.
  • A custom eval has no criterion.

Credit usage

Credits are consumed on every run of the block, whether an eval passes, fails, or is skipped, and presets and custom evals cost the same.

For more information about how Flows uses credits, see Manage your team’s Postman Flows credit usage.