Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

Overview

• Background: Originates from the growing reliance on LLMs for evaluating human-level text generation. Despite their utility, proprietary LLMs like GPT-4 present significant drawbacks:

  1. Closed-source nature: Limited transparency hinders collaborative research.

  2. Uncontrolled versioning: Frequent updates undermine reproducibility.

  3. High costs: Evaluating models at scale becomes financially burdensome.

• Objective: PROMETHEUS is an open-source 13B model designed to deliver evaluation capabilities comparable to GPT-4, leveraging user-defined score rubrics and reference answers, while addressing cost and accessibility concerns.

Contributions

1. FEEDBACK COLLECTION Dataset:

   • Includes 1K user-defined score rubrics, 20K instructions, and 100K responses and feedback.

   • Designed for training models with diverse, fine-grained evaluation criteria and reference answers.

2. PROMETHEUS Development:

   • Fine-tuned Llama-2-Chat-13B to achieve evaluation capabilities similar to GPT-4.

   • Achieved a Pearson correlation of 0.897 with human evaluators (GPT-4 scores 0.882).

3. Extensive Experiments:

   • Validated on benchmarks like MT Bench and Vicuna Bench, demonstrating high accuracy and capability in adhering to customized criteria.

4. Open-Source Availability:

   • Code, dataset, and model are made publicly available for research and practical applications.

The Feedback Collection Dataset

Previous studies demonstrated that fine-tuning LLMs with feedback improves their evaluator functionality. However, existing datasets do not directly address the goal of making LLMs into precise evaluators. Therefore, the authors developed the Feedback Collection Dataset to fine-tune open-source evaluator LLMs.

Key considerations in dataset construction:

1. Include as much reference material (e.g., reference answers and scoring rubrics) as possible.

2. Maintain uniform lengths of reference answers for each score (1–5) to avoid length bias.

3. Ensure a uniform score distribution to prevent decision biases.

4. Restrict the range of instructions and responses to realistic scenarios for user interactions with LLMs.

Methodology

• Dataset Construction:

  • Step 1: Curated 50 seed rubrics for evaluation tasks.

  • Step 2: Expanded to 1,000 rubrics via GPT-4 with in-context learning and paraphrasing.

  Prompt for Paraphrasing as a New Score Rubric

  We are brainstorming criteria with which to grade a language model on its responses in diverse situations.
  A ‘criteria‘ is some useful, real-world objective, and associated rubric for scores 1-5, that tests a capability.
   
  Here you will see 4 examples of ‘criteria‘, and their scoring rubrics, formatted as JSON.
  Criteria 1:
  {JSON LIST 1}
   
  Criteria 2:
  {JSON LIST 2}
   
  Criteria 3:
  {JSON LIST 3}
   
  Criteria 4:
  {JSON LIST 4}
   
  Please brainstorm a new criteria and scoring rubrics.
  Be creative and create new but useful criteria that people in different settings or industries might find practical.
  Please format the output as same as the above examples with no extra or surrounding text. Write [END] after you are done.
   
  New Criteria:Please paraphrase the sentences inside the dictionary below.
  Each paraphrase should not change the meaning or substance of the original sentence, be naturally written, but sufficiently diverse from one another.
  Diversity can come from differences in diction, phrasing, sentence structure, formality, detail, and/or other stylistic changes.
   
  The dictionary:
  {CRITERIA}
   
  Respond with only dictionary (same format as the given dictionary) with no extra or surrounding text.
  Write [END] after you are done.
   
  Dictionary with Paraphrased Sentences:

  • Step 3: Created 20K task instructions tailored to score rubrics.
  Prompt for Generating an Instruction and Reference Answer

  Your job is to generate a new novel problem and a response that is related to the given score rubric.
   
  The score rubric:
  {CRITERIA}
   
  * Problem
  - The problem should inherently be related to the score criteria and score rubric given above. Specifically, the score criteria should be the core attributes required to solve the problem.
  - The problem itself should not be too generic or easy to solve.
  - If the score rubric is related to logical abilities, generate problems that require math or coding abilities.
  - Try to make the person who might solve the problem not notice the existence of the score rubric by not explicitly mentioning it, and also provide additional inputs and options if needed.
  - Assume a situation where a user is interacting with an AI model. The user would try to ask in a first-person point of view, but not using terms like ”I”, ”A User” or ”You” in the first sentence.
  - Do not give a role to the AI, assume that the user is asking a question from his point of view.
  - Do not include any phrase related to AI model in the problem.
   
  * Response
  - The response should be a response that would get a score of 5 from the score rubric.
  - The response should be as detailed as possible unless the score rubric is related to conciseness or brevity. It should consist of multiple paragraphs, a list of items, or a step-by-step reasoning process.
  - The response should look like how a well-prompted GPT-4 would normally answer your problem.
   
  * Format
  - DO NOT WRITE ANY GREETING MESSAGES, just write the problem and response only.
  - In front of the problem, append the phrase ”Problem:” and in front of the response, append the phrase ”Response:”.
  - Write in the order of ”Problem” - ”Response”, where the two items are separated by the phrase ”[NEXT]”.
  - Write [END] after you are done.
   
  Data Generation:

  • Step 4: Generated responses and feedback for each instruction.
  Prompt for Generating Responses and Feedback

  Your job is to generate a response that would get a score of {SCORE} and corresponding feedback based on the given score rubric. For reference, a reference response that would get a score of 5 is also given.
   
  Instruction:
  {INSTRUCTION}
   
  The score rubric:
  {CRITERIA}
   
  Reference response (Score 5):
  {REFERENCE}
   
  * Response
  - The quality of the score {SCORE} response should be determined based on the score rubric, not by its length.
  - The score {SCORE} response should have the same length as the reference response, composed of {SENT NUM} sentences.
  - Do not explicitly state the keywords of the score rubric inside the response.
   
  * Feedback
  - The score {SCORE} feedback should each be an explanation of why the response would get a score of {SCORE}. It should be written based on the generated response and score rubric.
  - The score {SCORE} feedback shouldn’t just copy and paste the score rubric, but it should also give very detailed feedback on the content of the corresponding response.
  - The score {SCORE} feedback should include the phrase ”So the overall score is {SCORE}” in the last sentence.
   
  * Format
  - DO NOT WRITE ANY GREETING MESSAGES, just write the problem and response only.
  - In front of the response, append the phrase ”Response:” and in front of the feedback, append the phrase ”Feedback:”.
  - Write in the order of ”Response” - ”Feedback”, where the two items are separated by the phrase ”[NEXT]”.
  - Write [END] after you are done.
   
  Data Generation:

• Fine-tuning:

  • Trained PROMETHEUS to provide feedback and scores (1-5) based on user-defined rubrics and reference answers.

  • Included markers (e.g., "[RESULT]") to improve model inference.

  • Using the Feedback Collection Dataset, the authors fine-tuned Llama-2-Chat (7B and 13B) to create PROMETHEUS. The fine-tuning process mirrors Chain-of-Thought (CoT) Fine-tuning, where feedback is generated sequentially, followed by a score.

  Prompt for Prometheus

  ###Task Description:
  An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing an evaluation criterion is given.
  1. Write a detailed feedback that assesses the quality of the response strictly based on the given score rubric, not evaluating in general.
  2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
  3. The output format should look as follows: Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5) ̈
  4. Please do not generate any other opening, closing, and explanations.
   
  ###The instruction to evaluate:
  {instruction}
   
  ###Response to evaluate:
  {response}
   
  ###Reference Answer (Score 5):
  {reference answer}
   
  ###Score Rubrics:
  [{ criteria description }] 
  Score 1: {score1 description} 
  Score 2: {score2 description} 
  Score 3: {score3 description} 
  Score 4: {score4 description} 
  Score 5: {score5 description}
   
  ###Feedback:

Experimental Settings

1. MT Bench

• Specialization: Multi-turn instruction evaluation.

• Composition:

  • Multi-turn conversation prompts with logical and alignment-based criteria.

  • Final turn of each conversation is used as the evaluation input, along with previous dialogue context.

• Usage: Tests PROMETHEUS’s ability to evaluate conversational agents and handle complex multi-step interactions.

2. Vicuna Bench

• Specialization: Instruction-following evaluation.

• Composition:

  • 80 handcrafted prompts.

  • Includes reference answers and rubrics for fine-grained evaluation.

• Usage: Focuses on how well PROMETHEUS evaluates single-turn instruction-based tasks, like creative writing or factual answers.

3. Flask Eval

• Specialization: Fine-grained evaluation across multiple NLP skillsets.

• Composition:

  • Prompts sourced from NLP datasets and instruction datasets.

  • Includes 12 coarse-grained rubrics like:

    • Logical Thinking

    • Background Knowledge

    • User Alignment

    • Problem Handling

• Usage: Tests PROMETHEUS’s ability to generalize to common but broader evaluation tasks.

4. HHH Alignment

• Specialization: Measures alignment with human preferences.

• Composition:

  • 221 pairs of responses scored on:

    • Helpfulness

    • Harmlessness

    • Honesty

    • General preference.

• Usage: Evaluates PROMETHEUS’s accuracy in ranking responses based on human-aligned criteria.

5. MT Bench Human Judgment

• Specialization: Human preference-based evaluation.

• Composition:

  • Modified MT Bench dataset where responses are scored based on human preferences.

• Usage: Tests PROMETHEUS’s ability to predict human preferences without explicit ranking training.

Experimental Results

The study evaluated the correlation between human annotators and model evaluations across three benchmark datasets:

• FEEDBACK BENCH (Unseen Score Rubric subset)

• MT Bench (Zheng et al., 2023)

• Vicuna Bench (Chiang et al., 2023)

For each dataset, 45 unique custom scoring rubrics were used. PROMETHEUS demonstrated performance on par with GPT-4, achieving a Pearson correlation coefficient of 0.897, compared to GPT-4's 0.882 and GPT-3.5-Turbo's 0.392.

To assess whether PROMETHEUS generates useful and meaningful feedback beyond scoring decisions, human annotators were asked to select the better feedback in a pairwise comparison. Results showed:

• PROMETHEUS feedback was preferred over GPT-4 58.62% of the time.

• PROMETHEUS feedback was preferred over GPT-3.5-Turbo 79.57% of the time.

These findings highlight that PROMETHEUS provides valuable and actionable feedback, making it a competitive evaluator.

Analysis of Feedback Preferences

In addition to pairwise comparisons, human annotators analyzed reasons for their preferences by selecting from six comprehensive options.

• GPT-4 Feedback: Often perceived as generic or abstract, leading to lower preference.

• PROMETHEUS Feedback: While generally detailed and critical, it was sometimes seen as overly harsh, which detracted from its appeal.

This difference indicates that GPT-4 tends to provide neutral and high-level feedback, whereas PROMETHEUS explicitly articulates its evaluation of a response's quality, often expressing clear opinions on whether the response meets the criteria.

Can PROMETHEUS Simulate GPT-4 Evaluation?

PROMETHEUS surpassed other models (e.g., GPT-3.5-Turbo and Llama2-Chat-70B) in correlation with GPT-4 evaluation scores across multiple datasets. PROMETHEUS exhibited strengths in aligning with unseen rubrics but performed slightly below GPT-4 in scenarios where instruction length and specificity varied.

Key Findings from Table 3

1. LLAMA-2-CHAT 13B Performance:

   • Surprisingly, the 13B version of LLAMA-2-CHAT showed decreased performance compared to the 7B version.

   • While the 70B model achieved a slight improvement, increasing model size alone does not necessarily enhance evaluation capability in LLMs.

2. PROMETHEUS Performance:

   • Seen Rubric Set: PROMETHEUS 13B outperformed LLAMA-2-CHAT 13B by a Pearson correlation improvement of +0.420.

   • Unseen Rubric Set: It showed a +0.397 improvement.

   • PROMETHEUS consistently outperformed LLAMA-2-CHAT 70B, GPT-3.5-TURBO-0613, and various GPT-4 versions.

3. Reason for PROMETHEUS’s Performance:

   • The alignment of the training data (Feedback Collection) with the test data distribution appears to play a critical role. PROMETHEUS excels at scenarios where LLM interaction mimics real-world usage, even with unseen rubrics.

   • Models trained on coarse-grained score rubrics (e.g., LLAMA-2-CHAT 13B + COARSE) showed minimal improvements, emphasizing the importance of fine-grained training for handling diverse rubrics.

Key Findings from Table 4

• Scaling Trends:

  • Across various sizes of LLAMA-2-CHAT, performance trends remained consistent, indicating that larger model sizes alone do not guarantee improved evaluation capabilities.

• PROMETHEUS Benchmark Results:

  • Compared to LLAMA-2-CHAT 13B, PROMETHEUS achieved Pearson correlation gains of:

    • +0.255 on Vicuna Bench

    • +0.493 on MT Bench

    • +0.202 on Flask Eval

• Comparison with Other Models:

  • PROMETHEUS surpassed LLAMA-2-CHAT 70B and GPT-3.5-TURBO-0613 in evaluation tasks.

  • However, it still lags behind GPT-4, likely due to differences in instruction length and specificity between the training and evaluation datasets.

    • Feedback Collection: Instructions tend to be long and detailed (e.g., "I am an urban planner... looking for solutions to address traffic congestion and air quality issues caused by population growth").

    • Evaluation Datasets: Instructions are shorter (e.g., "Explain quantum mechanics").

• Specialized Training with Flask Eval:

  • LLAMA-2-CHAT 13B + COARSE, trained specifically on the Flask Eval dataset, outperformed PROMETHEUS in that domain.

  • This underscores the benefit of tailoring training to specific evaluation datasets for optimal performance.

Can PROMETHEUS Function as a Reward Model?

PROMETHEUS improved ranking performance on HHH Alignment and MT Bench Human Judgment datasets compared to the baseline. These results suggest that PROMETHEUS can serve as a reward model without being explicitly trained for ranking tasks.

Why Include Reference Materials?

Evaluating responses without reference materials is highly challenging. When only an instruction and response are provided, an evaluator LLM must:

1. Identify the critical aspects of the instruction.

2. Internally hypothesize an ideal answer to the instruction.

3. Evaluate the response's quality based on the above two steps.

By incorporating reference materials, the evaluation process is simplified, enabling the model to focus solely on assessing the quality of the response rather than resolving the instruction. The roles of the key components are as follows:

• Score Rubric: Provides explicit information on the critical aspects needed to address the instruction. Without this, the evaluator LLM would need to infer these details from the instruction itself, leading to potential inaccuracies.

• Reference Answer: Separates the process of hypothesizing the ideal answer and evaluating the response. When a reference answer is provided, the evaluator can focus entirely on comparing the response to this benchmark, avoiding issues that arise when the model cannot solve the problem itself.

Ablation Studies

• Training Ablation:

  Experiments removing individual reference materials confirmed their independent contribution to PROMETHEUS’s superior performance.

  • Reference Answer Exclusion: Removing reference answers caused the largest performance drop, affirming their critical role in enabling evaluators to focus on assessment rather than hypothesizing solutions.

  • Score Rubric Exclusion: Excluding the score rubric had less impact on datasets like FEEDBACK BENCH but led to significant performance drops on Vicuna Bench. This supports the hypothesis that rubrics are vital for generalization across datasets.

• Model Ablation:

  • Models fine-tuned with both supervised learning and RLHF (Reinforcement Learning from Human Feedback) showed the best performance.

  • While using different base models (e.g., Llama-2, Vicuna, Code-Llama) did not severely degrade performance, Code-Llama had slight advantages in evaluating code-related domains.

Narrowing the Gap with GPT-4 Evaluation

PROMETHEUS consistently outperformed GPT-4 in human-like evaluation tasks, particularly on datasets like FEEDBACK BENCH. Several reasons are proposed for this:

1. Direct Fine-Tuning Alignment: PROMETHEUS was trained on datasets closely aligned with evaluation tasks, naturally outperforming GPT-4, which was not fine-tuned on similar data.

2. Consistency Issues in GPT-4: GPT-4's self-consistency may have been diluted due to augmented instructions and instances used for comparison.

3. Exclusions in Human Evaluations: Tasks involving coding or mathematics—areas where GPT-4 excels—were omitted in human evaluation settings, potentially favoring PROMETHEUS.

4. Critical Evaluation Style: PROMETHEUS demonstrated a more critical approach compared to GPT-4, likely stemming from its specialized fine-tuning for evaluation purposes. This characteristic enables PROMETHEUS to identify areas for improvement more explicitly than GPT-4.