AuRoRA: Augmented Reasoning and Refining with Task-Adaptive Chain-of-Thought Prompting

Anni Zou, Zhuosheng Zhang, Hai Zhao
Shanghai Jiao Tong University
Demo Video Code

AuRoRA is an augmented reasoning and refining system with task-adaptive chain-of-thought prompting. AuRoRA has the characteristics of task self-adaptation and process automation. It enhances the reasoning ability of the system from multiple dimensions such as knowledge retrieval, knowledge refinement and knowledge correction, and adapts to different large language models (LLMs). The system provides user-friendly visual analysis of results, which has important academic and application value in the era of LLMs.

Website Demos

System Workflow

AuRoRA is based on the chain-of-thought (CoT) prompting, with the features of task self-adaptation and process automation, effectively adapting the LLMs to diverse task scenarios. The workflow of the system is divided into six steps:

  1. Self-identification: the user inputs a question, and the system automatically identifies the type of question and selects the question-adapted prompts.
  2. Self-construction: the system clusters and classifies the questions in the data pool according to the question type, and automatically constructs samples to ensure the high efficiency of the method and the diversity of the constructed samples.
  3. Self-retrieval: the system extracts relevant knowledge from multi-source knowledge bases according to the content of the input problem, providing diversified knowledge bases and effectively alleviating the problem of hallucination.
  4. Self-refinement: the system fuses and refines knowledge from multiple sources in order to achieve double-checking and filtering between different sources of knowledge and to improve the reliability of knowledge enhancement.
  5. Self-revision: the system rewrites and corrects the original CoT according to the automatically extracted high-quality knowledge to improve its accuracy and logic.
  6. Self-consistency: the system adopts a self-consistency decoding strategy, sampling the CoT (both initial and corrected) multiple times during the process. The output is automatically organized by mimicking human divergent thinking and the most self-consistent answer is selected as the final answer.

Experimental Results

Compared to the zero-shot setting, our system (AuRoRA) significantly boosts the performance and improves the method's interpretability.

Related Works

Recent work has enhanced the CoT system in three main aspects:

  • Sample Selection: Auto-CoT and Self-Prompting guide LLMs through various types of reasoning by automatically constructing diverse, task-adapted prompt samples.
  • Reasoning Enhancement: Knowledge augmentation methods such as CRITIC, Verify-and-Edit and Multi-Chain Reasoning (MCR) correct the inference chain by interacting with external knowledge.
  • Decoding Strategy: Self-Consistency generates diverse reasoning paths for aggregation by sampling from the decoder.

BibTeX

@misc{aurora-web,
title={AuRoRA: Augmented Reasoning and Refining with Task-Adaptive Chain-of-Thought Prompting},
author={Anni Zou and Zhuosheng Zhang and Hai Zhao},
url={https://anni-zou.github.io/aurora-en.github.io/},
year={2023}
}