Concept 2 min

LLM-as-Judge

One model evaluates another model’s output at scale against criteria that only a human would normally judge, as long as it has a rubric and calibration.

In one sentence

Using an AI model to evaluate another model’s output at scale against subjective criteria that would be too expensive to submit to human judgment case by case.

Before

Subjective criterion: tone, dignity

What it does

Model evaluates with an explicit rubric

After

Score and rationale, at scale

↺calibrate against human judgment

Analogy

The second grader of a college entrance essay. It doesn’t write anything; it only judges, with a rubric in hand. Without the rubric, its score would be worth little.

Example

The criterion "does the letter treat the child with dignity" does not fit into any automated rule. So a judge receives the letter along with an explicit rubric that penalizes:

language of pity or need as the central theme
comparisons that diminish the child
focus on what is lacking rather than what has improved
promises the organization cannot keep

It returns a score from 1 to 5, the rationale, and the quoted excerpt. Cost: R$ 3 for the 200 letters, versus roughly 10 hours of coordination.

The common mistake

Asking "is this good?" and accepting the score. A judge without a rubric gives 4 out of 5 to almost everything, and the suite starts producing nice-looking numbers that measure nothing.

Even worse is never calibrating the judge. Take 30 cases that humans have already evaluated and see whether it agrees with the scores. If it agrees 60% of the time, it is generating a number without evaluating anything.

Third mistake: using the same model that wrote the text to judge what it wrote. It tends to like its own text.

In practice

Explicit rubric, with an example of a score of 1 and one of a score of 5. The real work is in the rubric; the model only applies it.
Structured output: score + rationale + quoted excerpt. The rationale matters more than the score, because it is what you read when the number surprises you.
A different model from the generator, whenever possible.
Recalibrate against human judgment every quarter, or whenever you switch models.

How to make it tangible

A written rubric, with examples of the minimum and maximum scores, plus the call that applies it at scale.

Connects with

Translated from Portuguese with AI assistance.