Comparing and Ranking AI Responses

Pairwise evaluation is not a popularity contest

When comparing two answers, judge both against the same requirements. Avoid deciding that one is “better” before checking each response independently.

A reliable comparison workflow

1. Extract prompt constraints
2. Evaluate Response A independently
3. Evaluate Response B independently
4. Identify the most important differences
5. Compare issue severity, not just issue count
6. Choose A, B, or tie if the rubric allows
7. Write a short evidence-based justification

One major error can matter more than several minor ones

A response with perfect formatting but a critical factual error may be worse than a response with a small style issue.

Justification template

• Decision: Response B is stronger.
• Evidence: It uses age-appropriate language while A relies on technical jargon.
• Impact: It better satisfies the user's audience requirement.

Tie discipline

If both responses satisfy the rubric similarly, do not invent a difference just to avoid a tie. Use the task's tie policy.

Module summary

• Evaluate A and B independently first.
• Compare issue severity.
• Support rankings with specific evidence.
• Use ties only when the rubric permits and the quality is genuinely similar.