LLM performance scores are inflated: A new method shows the truth

As large language models (LLMs) become increasingly sophisticated, ensuring fair and unbiased evaluation has become a critical challenge. Existing evaluation protocols often suffer from benchmark contamination, where models are trained on datasets that include portions of the test benchmarks, leading to artificially inflated results. A recent approach known as Agents-as-an-Evaluator attempts to address this issue by generating new test questions using AI agents. However, this method introduces its own biases, which remain largely unexplored.

Researchers from Hikvision Research Institute, including Meilin Chen, Jian Tian, Liang Ma, Di Xie, Weijie Chen, and Jiang Zhu, propose a new evaluation framework called the Unbiased Evaluator in their study, “Unbiased Evaluation of Large Language Models from a Causal Perspective,” to mitigate these biases.

Their study provides a theoretical framework for evaluation bias and introduces a causality-based evaluation protocol to offer a more comprehensive, unbiased, and interpretable assessment of LLMs.

Challenges with Agents-as-an-Evaluator

While Agents-as-an-Evaluator attempts to reduce benchmark contamination by having AI-generated test questions, the researchers identify two key biases in this method:

1. Data bias: AI-generated test questions tend to favor domains where the model already performs well, leading to an unbalanced assessment.
2. Model bias: During evaluation, AI-generated content aligns more with the model’s strengths, giving it an unfair advantage when assessing itself.

These biases distort the evaluation process, making it difficult to accurately measure a model’s true capabilities.

Introducing the Unbiased Evaluator

To address these issues, the researchers introduce the Unbiased Evaluator, an evaluation protocol based on causal inference principles. This method dynamically evaluates LLMs using controlled interventions, rather than relying solely on static datasets.

At its core, the Unbiased Evaluator utilizes Bags of Atomic Interventions (BOAT)—structured manipulations of test data to assess how LLMs respond to different variations of the same question. This method allows for a systematic evaluation of AI robustness, reducing the impact of pre-existing biases.

Testing the theory: Human, AI, and recursive oversight experiments

To validate their hypotheses, the researchers conducted a series of experiments involving:

• Human-Human oversight: Evaluating whether humans perform better when critiquing critiques rather than directly assessing AI-generated responses.
• Human-AI oversight: Testing if humans can effectively supervise AI by reviewing its self-critiques rather than its raw outputs.
• AI-AI oversight: Assessing whether AI itself can perform effective self-recursive critiques.

Key findings

Human-Human experiments confirmed that reviewing a critique was easier than evaluating a response directly. Higher-order critiques helped increase accuracy while reducing effort.

Human-AI experiments showed that when AI generated recursive critiques, humans could still provide meaningful oversight, even in areas where AI outperformed them.

AI-AI experiments revealed that while AI models could critique their own outputs, their ability to perform higher-order self-critiquing was still limited. Current AI struggles to consistently improve through recursive self-critique, highlighting the need for further advancements in AI alignment.

How recursive self-critiquing works

The researchers formalized a hierarchical critique structure:

• Response Level: The AI generates an answer.
• First-Order Critique (C1): AI reviews its own response, identifying errors or weaknesses.
• Second-Order Critique (C2): AI evaluates multiple first-order critiques, selecting the most valid points.
• Higher-Order Critiques (C3+): AI continues refining critiques recursively, improving accuracy with each level.

The study also introduced two baseline comparison methods:

• Majority voting: Aggregating multiple critiques to see if consensus improves accuracy.
• Naive voting: A control method that simply counts previous judgments without additional analysis.

Findings showed that recursive critiques consistently improved accuracy beyond simple vote aggregation, indicating that the method adds meaningful insight rather than just averaging opinions.

Can recursive self-critiquing solve AI oversight?

The research suggests recursive oversight could be a breakthrough for scalable AI monitoring, but challenges remain.

Strengths

One of the key advantages of recursive self-critiquing is that it allows humans to oversee AI systems without needing to evaluate complex raw outputs. Instead of directly assessing AI-generated content, human reviewers can focus on evaluating AI’s self-critiques, making the process more manageable and efficient.

Another major benefit is that recursive oversight makes AI alignment more scalable. Traditional alignment methods rely heavily on direct human intervention, which becomes impractical as AI capabilities surpass human expertise. By shifting to a system where AI can critique and refine its own outputs, the dependency on human supervision is reduced while maintaining oversight.

Furthermore, recursive self-critiquing introduces a structured approach to AI oversight, resembling hierarchical decision-making in organizations. Just as corporate structures rely on multiple layers of review and feedback, recursive oversight enables AI systems to refine their responses in a structured and logical manner, improving accuracy and interpretability.

Limitations

Despite its potential, recursive oversight has notable limitations. Current AI models struggle with self-critiquing beyond a few levels. While first- and second-order critiques improve oversight, higher-order critiques often fail to produce meaningful refinements, limiting the method’s effectiveness.

Additionally, recursive oversight does not eliminate the risk of reward hacking, where AI models optimize for proxy goals rather than genuine human intent. AI may learn to manipulate its own critique mechanisms to produce favorable evaluations rather than genuinely improving its outputs.

Another critical challenge is ensuring that self-critiquing models do not reinforce their own biases. Without proper safeguards, recursive oversight could lead to AI models amplifying pre-existing errors rather than correcting them. Further research is needed to develop techniques that ensure self-critiquing improves AI alignment rather than reinforcing undesirable patterns.

Experimental results: Unbiased evaluator vs. traditional methods

The study compared state-of-the-art proprietary models like GPT-4, Gemini 2.0, and Claude with open-source models like Llama, Qwen, Yi, and Mistral under both traditional evaluation benchmarks and the Unbiased Evaluator.

Results showed that:

• All models performed worse when evaluated using the Unbiased Evaluator, suggesting that previous evaluation methods overestimated AI performance.
• Proprietary models like GPT-4 and Gemini 2.0 exhibited the least performance drop, indicating stronger generalization.
• Open-source models showed greater performance declines, suggesting more room for improvement in robustness.

This research highlights significant biases in current AI evaluation methodologies and proposes the Unbiased Evaluator as a new solution.

Featured image credit: Kerem Gülen/Midjourney