✨ CHARM Findings ✨

Prompt Strategy

We selected 5 commonly used prompt strategies, and assessed the performance of the 19 LLMs on CHARM reasoning task. We evaluated the currently commonly used LLMs, which can be divided into two categories:

Examples of prompt strategies

We tested the combinations of the 19 LLMs and the 5 prompt strategies in CHARM reasoning tasks. The results are shown in the table below.

Averaged accuracy on CHARM reasoning tasks

💡 Results showed that LLMs' orientation and the task's domain affect prompt strategy performance, which enriches previous research findings.

From the LLM dimension, it's clear that various LLMs prefer different prompt strategies: XLT consistently excels for English LLMs among the 5 strategies, while for Chinese-oriented LLMs, despite some complexity, ZH-CoT generally performs best.

From the commonsense domain dimension, strategies that use English for reasoning (like XLT, Translate-EN, etc.) are suitable for the global domain; however, ZH-CoT generally performs better in the Chinese domain.

The conclusion here differs from previous studies (Huang et al., 2023a, Zhang et al., 2023a, Shi et al., 2022), which suggested that employing English for non-English reasoning tasks was more effective than using the native language.

Integrated Reasoning vs Memorization

We evaluated the correlation between integrated reasoning and memorization on the MRI tasks. Here is the average performance of the LLMs on the 4 MRI tasks.

Averaged accuracy across the 4 MRI tasks in the Chinese commonsense domain.

As shown in the figure, the 19 LLMs can be roughly divided into three types:

• Type I: Low memorization and low integrated reasoning ability. We found that apart from OpenAI's GPT series, all other English LLMs belong to this type.
• Type II: High memorization and medium integrated reasoning ability. GPT3.5 and all Chinese-oriented LLMs below 30B belong to this type. Some LLMs have high memorization performance but relatively poor integrated reasoning ability.
• Type III: Ultra-high memorization and high integrated reasoning ability. This category includes GPT4 and the three Chinese-oriented LLMs exceeding a size of 30B.

Memorization-Independent Reasoning

We propose two methods to compare the LLMs' memorization-independent reasoning on the MRI tasks: Mono-LLM-Memorization (FRMM) and Memorization-Independent Battles among LLMs (MIB). For specific details about the FRMM and MIB methods, please refer to the paper. Here are the results of the FRMM and MIB methods.

Leaderboard on the MRI tasks.

💡 If Language Models (LLMs) provide incorrect responses to retained reasoning questions, these can be termed as memorization-independent reasoning errors. We analyzed these errors by manually reviewing the reasoning process and categorized them into four main types:

• Understanding Error: In this case, the LLM was unable to accurately comprehend the question, including misunderstanding the content, ignoring or even modifying important information in the premise, and failing to grasp the core query of the question.
• Knowledge Error: The LLM incorporated inaccurate knowledge during the reasoning process. It's important to highlight that the knowledge pieces related to the reasoning question were previously examined in the related memorization questions, which the LLM answered correctly. However, the LLM output incorrect information during the reasoning phase.
• Logical Error: The LLM made logical reasoning errors, such as mathematical reasoning errors, inability to reach the correct conclusion based on sufficient information, or reaching the correct conclusion but outputting the wrong option.
• Other Errors: These are other scattered, relatively rare types of errors.

💡 Distribution of the Memorization-Independent Reasoning Errors

Istribution of the memorization-independent reasoning errors

💡 Examples of the 3 types of memorization-independent reasoning errors of LLMs

Examples of the 3 types of memorization-independent reasoning errors of LLMs.

🖊️ Citation

    @misc{sun2024benchmarking,
          title={Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations}, 
          author={Jiaxing Sun and Weiquan Huang and Jiang Wu and Chenya Gu and Wei Li and Songyang Zhang and Hang Yan and Conghui He},
          year={2024},
          eprint={2403.14112},
          archivePrefix={arXiv},
          primaryClass={cs.CL}
    }