Add evals to a flow in Postman Flows

You can add evals to a flow to grade AI-generated output against qualitative criteria, using an AI model as a judge. Add evals on the AI Agent block, where you turn on preset criteria and add custom ones, or in an Evaluate block, where you define evals in TypeScript with pm.eval. Evals are opt-in and consume Flows credits. They aren’t supported by the Postman CLI.

Grade output on the AI Agent block

The AI Agent block grades its own output using a separate judge model, distinct from the model the block uses to generate that output. The passing threshold is fixed at 80 on a 0–100 scale, and the judge model is fixed at gpt-4o-mini-2024-07-18.

To grade an AI Agent block’s output with evals, do the following:

Click the AI Agent block, then click Evals.
Under Quality Presets, turn on each built-in eval you want to run: Friendliness, Safety, Non-toxicity, Correctness, or Relevance.

The AI Agent block passes the agent’s prompt and inputs to the judge automatically, which can give Correctness and Relevance enough grounding to produce a score. For direct control over the reference answer and query, use an Evaluate block.
(Optional) Under Custom, click Add custom eval. Enter a name and a criterion in natural language, then click Add. You can add up to 50 custom evals to an AI Agent block, and the five quality presets don’t count toward this limit.
Run the flow to grade the output.

Grade output in an Evaluate block

In an Evaluate block, you define evals in a TypeScript script with pm.eval. Unlike the AI Agent block, the Evaluate block gives the judge only the context you pass to it, so you can set a reference answer, a query, a custom threshold, and a different judge model.

To grade output in an Evaluate block, do the following:

Add an Evaluate block and connect the block that produces the output you want to grade to the Evaluate block’s input port. The Evaluate block inserts a Select block and assigns the value to a variable named value1.
At the top of the Evaluate block, select TypeScript from the dropdown list.
In the text box, call one or more pm.eval methods on the variable. For example, the following script runs four preset evals and one custom eval:
```
pm.eval.friendliness(value1);
pm.eval.safety(value1);
pm.eval.relevance(value1, { query: "What is the return policy?" });
pm.eval.correctness(value1, {
  reference: "Returns are accepted within 30 days of purchase.",
});
pm.eval.custom(
  "On brand",
  value1,
  "The response stays professional and never promises a refund.",
);
```
The preset pm.eval methods accept an options object with supported fields such as threshold (the pass mark, on a 0–1 scale, default 0.8), reference, query, context, and model. The pm.eval.custom method instead takes a name, the value to grade, and a natural-language criterion. For the full option reference and script limits, see The Evaluate block.
Run the flow to grade the output.

Read the eval results

Evals run when the flow runs. You can’t rerun them independently of the flow.

Before a run, the block shows No eval results yet. Run the flow to grade this response. After a run, you can read the results in the following places:

On the block — A preview shows a summary, such as Passed 3/4, with a row for each eval and its score. You can view the judge’s reason for each graded eval.
In the run log — The run log has an All evals tab that lists the eval results for all blocks in the flow that ran evals. To open the run log, click View run logs in the toolbar.

If grading can’t complete, the flow run still completes. The block shows Grading failed, and the All evals tab in the run log marks the block as Grading failed with no evals graded.

Act on eval results downstream

Both blocks send structured eval results through a dedicated Evals output port that you can connect to other blocks. Each graded result includes the eval’s status, score, threshold, and reason. A skipped eval instead has a skippedReason that explains why it couldn’t be graded. The payload also includes a summary with the graded and passed counts. For the full payload structure, see The Evaluate block.

For example, connect the Evals port to a Condition block that routes the flow one way when all graded evals pass and another way otherwise. Connecting the port assigns the payload to a variable named value1, so you can write a condition like:

value1.summary.graded > 0 &&
value1.summary.passed === value1.summary.graded

Treat the eval set as fully passed only when value1.summary.graded is greater than 0 and value1.summary.passed equals value1.summary.graded. When every eval skips, graded, passed, and score are all 0, so a passed === graded check without the graded > 0 guard would treat that result as a pass.

You can also connect the Evals port to an HTTP Request block that reports failing scores to another service.

Grade output on the AI Agent block

Grade output in an Evaluate block

Read the eval results

Act on eval results downstream

Related pages