ReACT介绍与llama_index ReActAgent实践

Agent是大模型的重要应用方向，而ReACT是学术界提出的重要方法，本文介绍ReACT论文，然后通过llama_index ReActAgent来分析ReACT的执行过程。

ReACT

《REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS》，由Shunyu Yao等人撰写，发表在2023年的ICLR会议上。论文探讨了如何在大型语言模型（LLMs）中结合推理（reasoning）和行动（acting）的能力，以提高模型在语言理解和交互式决策制定任务中的性能。

主要观点：

大型语言模型（LLMs）在语言理解和交互式决策制定任务中表现出色，但它们的推理（例如链式思维提示）和行动（例如行动计划生成）能力通常被分开研究。
作者提出了一种新的方法ReAct，它通过交错生成推理轨迹和特定于任务的行动，使两者之间产生更大的协同效应。推理轨迹帮助模型诱导、跟踪和更新行动计划，以及处理异常情况，而行动则允许它与外部资源（如知识库或环境）接口并收集额外信息。

具体步骤：

ReAct（Reasoning and Acting）是一种结合了推理（reasoning）和行动（acting）的方法，旨在提高大型语言模型（LLMs）在解决复杂任务时的性能和可解释性。ReAct的具体步骤如下：

定义任务和目标：首先，明确模型需要解决的任务和目标。这可能包括问题回答、事实验证、文本游戏或网页导航等。
生成推理轨迹：模型生成一系列推理轨迹，这些轨迹是模型内部的思考过程，用于解决问题。这些推理轨迹可能包括分解任务目标、提取重要信息、执行常识推理、指导搜索重构、跟踪进度和处理异常等。
执行行动：模型根据推理轨迹执行一系列行动。在ReAct中，行动是通过与外部环境（例如Wikipedia或其他知识库）的交互来完成的。行动可以是搜索信息、选择产品、选择选项或购买等。
交替推理和行动：在解决任务的过程中，模型会交替进行推理和行动。这意味着模型在执行每个行动后可能会生成新的推理轨迹，然后再执行新的行动，以此往复。
更新上下文：每次行动后，模型会更新上下文信息，这包括之前的行动、观察结果和新生成的推理轨迹。这种上下文更新帮助模型跟踪任务的当前状态，并为未来的推理和行动提供信息。
生成任务解决轨迹：通过上述步骤，模型生成一个包含行动、观察和推理轨迹的任务解决轨迹。这个轨迹不仅展示了模型如何解决问题，而且提供了模型决策的透明度，使得人类用户可以理解和评估模型的行为。
评估和调整：在实际应用中，模型生成的任务解决轨迹可能会被人类用户评估和调整。用户提供的反馈可以用来进一步指导模型的行为，或者在模型自身无法正确解决问题时进行干预。

ReAct的核心在于通过交错的推理和行动步骤，使模型能够在执行任务时动态地进行推理和行动，从而提高任务解决的准确性和效率。这种方法特别适用于需要与外部环境交互并从中获取信息以支持决策的任务。

以llama_index ReActAgent来看ReAct 执行过程

我们编写一个简单的ReActAgent程序，计算乘法，同样的没有openai的账号，我们用google的Gemini。

from llama_index.core.agent import ReActAgent

from llama_index.core.tools import FunctionTool

from llama_index.llms.gemini import Gemini  

# define sample Tool

def multiply(a: int, b: int) -> int:

    """Multiply two integers and returns the result integer"""

    return a * b  

multiply_tool = FunctionTool.from_defaults(fn=multiply)  

# initialize llm

llm = Gemini(api_key="AI...", transport="rest")  

# initialize ReAct agent

agent = ReActAgent.from_tools([multiply_tool], llm=llm, verbose=True)  

resp = agent.query("计算85乘以9")  

print(resp.response)

第一步，将tools放入prompts，要求LLM按照要求做Thought，然后出书符合要求的response：

[ChatMessage(role=<MessageRole.SYSTEM: 'system'>, content='\nYou are designed to help with a variety of tasks, from answering questions     to providing summaries to other types of analyses.\n\n## Tools\nYou have access to a wide variety of tools. You are responsible for using\nthe tools in any sequence you deem appropriate to complete the task at hand.\nThis may require breaking the task into subtasks and using different tools\nto complete each subtask.\n\nYou have access to the following tools:\n> Tool Name: multiply\nTool Description: multiply(a: int, b: int) -> int\nMultiply two integers and returns the result integer\nTool Args: {"type": "object", "properties": {"a": {"title": "A", "type": "integer"}, "b": {"title": "B", "type": "integer"}}, "required": ["a", "b"]}\n\n\n## Output Format\nPlease answer in the same language as the question and use the following format:\n\n```\nThought: The current language of the user is: (user\'s language). I need to use a tool to help me answer the question.\nAction: tool name (one of multiply) if using a tool.\nAction Input: the input to the tool, in a JSON format representing the kwargs (e.g. {"input": "hello world", "num_beams": 5})\n```\n\nPlease ALWAYS start with a Thought.\n\nPlease use a valid JSON format for the Action Input. Do NOT do this {\'input\': \'hello world\', \'num_beams\': 5}.\n\nIf this format is used, the user will respond in the following format:\n\n```\nObservation: tool response\n```\n\nYou should keep repeating the above format until you have enough information\nto answer the question without using any more tools. At that point, you MUST respond\nin the one of the following two formats:\n\n```\nThought: I can answer without using any more tools. I\'ll use the user\'s language to answer\nAnswer: [your answer here (In the same language as the user\'s question)]\n```\n\n```\nThought: I cannot answer the question with the provided tools.\nAnswer: [your answer here (In the same language as the user\'s question)]\n```\n\n## Current Conversation\nBelow is the current conversation consisting of interleaving human and assistant messages.\n\n', additional_kwargs={}), ChatMessage(role=<MessageRole.USER: 'user'>, content='计算85乘以9', additional_kwargs={})]

上述数据没有格式化，我们格式化后来看:

![[Pasted image 20240313134946.png]]

大体上就是让LLM做COT思考，然后根据用户当前的question，选择合适的tool进行计算。

LLM 思考后，回复：

assistant: Thought: The current language of the user is: chinese. I need to use a tool to help me answer the question.

Action: multiply

Action Input: {"a": 85, "b": 9}

可以看到，给出了Action的tool，以及action的input

llama_index 会对上面的输出进行解析process_actions，得到：

[ActionReasoningStep(thought='The current language of the user is: chinese. I need to use a tool to help me answer the question.', action='multiply', action_input={'a': 85, 'b': 9}), ObservationReasoningStep(observation='765')]

得到action step后，会进行执行，调用函数计算

cur_step_output = 765

然后，开始next_step:

[TaskStep(task_id='1d8db1d0-f5e3-4bfa-bcef-8fc405f958ca', step_id='91e9a97a-5ae1-4900-bf77-690865976902', input=None, step_state={'is_first': False}, next_steps={}, prev_steps={}, is_ready=True)]

来看step如何执行，同样的拼接prompt，更新上下文，然后让大模型COT:

[ChatMessage(role=<MessageRole.SYSTEM: 'system'>, content='\nYou are designed to help with a variety of tasks, from answering questions     to providing summaries to other types of analyses.\n\n## Tools\nYou have access to a wide variety of tools. You are responsible for using\nthe tools in any sequence you deem appropriate to complete the task at hand.\nThis may require breaking the task into subtasks and using different tools\nto complete each subtask.\n\nYou have access to the following tools:\n> Tool Name: multiply\nTool Description: multiply(a: int, b: int) -> int\nMultiply two integers and returns the result integer\nTool Args: {"type": "object", "properties": {"a": {"title": "A", "type": "integer"}, "b": {"title": "B", "type": "integer"}}, "required": ["a", "b"]}\n\n\n## Output Format\nPlease answer in the same language as the question and use the following format:\n\n```\nThought: The current language of the user is: (user\'s language). I need to use a tool to help me answer the question.\nAction: tool name (one of multiply) if using a tool.\nAction Input: the input to the tool, in a JSON format representing the kwargs (e.g. {"input": "hello world", "num_beams": 5})\n```\n\nPlease ALWAYS start with a Thought.\n\nPlease use a valid JSON format for the Action Input. Do NOT do this {\'input\': \'hello world\', \'num_beams\': 5}.\n\nIf this format is used, the user will respond in the following format:\n\n```\nObservation: tool response\n```\n\nYou should keep repeating the above format until you have enough information\nto answer the question without using any more tools. At that point, you MUST respond\nin the one of the following two formats:\n\n```\nThought: I can answer without using any more tools. I\'ll use the user\'s language to answer\nAnswer: [your answer here (In the same language as the user\'s question)]\n```\n\n```\nThought: I cannot answer the question with the provided tools.\nAnswer: [your answer here (In the same language as the user\'s question)]\n```\n\n## Current Conversation\nBelow is the current conversation consisting of interleaving human and assistant messages.\n\n', additional_kwargs={}), ChatMessage(role=<MessageRole.USER: 'user'>, content='计算85乘以9', additional_kwargs={}), ChatMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content="Thought: The current language of the user is: chinese. I need to use a tool to help me answer the question.\nAction: multiply\nAction Input: {'a': 85, 'b': 9}", additional_kwargs={}), ChatMessage(role=<MessageRole.USER: 'user'>, content='Observation: 765', additional_kwargs={})]

大模型回复：

assistant: Thought: I can answer without using any more tools. I'll use the user's language to answer

Answer: 765

上述的回复已经没有新的action了，因此解析不出来action，next_steps就为空了，计算结束，返回答案
这里的计算比较简单，一步action就搞定了，对于需要多步action的，会按照下面的流程图，递归的执行，直到得到Final Answer
- 思考Thought
- 得到要执行的Action
- 执行Action，得到观测结果 Observation
- 将上下文，连同Observation，让大模型继续思考Thought
- 直到没有Action，给出最后的Filnal Answer