Guided Reasoning: A New Approach to Improving Multi-Agent System Intelligence

Gregor Betz from Logikon AI, KIT introduces Guided Reasoning. A system with more than one agent is a Guided Reasoning system if one agent, called the guide, mostly works with the other agents to improve their Reasoning. A multi-agent system with a guide agent and at least one client agent is called a Guided Reasoning system if the guide works with the clients in a planned and main way to get them to reason in a way that follows a certain method M. One way to describe the reasoning method M is with standards and criteria, clear examples, or detailed rules and directions. Guided Reasoning methods include a coach helping a business unit do a SWOT analysis, a child helping their grandmother solve a crossword problem, and a Socratic dialogue.

At first glance, the case for AI-AI Guided Reasoning is based on these assumptions:

AI should give the right answers and explain them. 

AI systems can only honestly explain their answers if they are based on clear thinking. 

Bad Reasoning makes it harder for AI systems to give the right replies.

Strong experts in a field don’t always know how to use advanced thinking techniques.

The cognitive specialization principle says that to make AI systems that can be explained and are accurate; more AI experts should be added for reasoning methods (meta-reasoning specialists) who can work with experts in other domains. Guided Reasoning is a good design technique for advanced GenAI apps because it makes it easy to divide the cognitive work.

Logikon’s standard way of using Guided Reasoning mentions that when client agents are faced with a decision problem, they are told to look into and carefully weigh both the pros and cons reasons.

Step 1: The Guided Reasoning method is started when the user query is sent. This might be done immediately by the client model calling a tool-use method or if the user specifically asks for it to be done. 

Step 2: The client presents the problem statement to the guide. The guide’s crucial role is to meticulously organize the steps of thinking that will be used to find the answer, providing a clear structure to the process.Step 3: The guide may ask the client questions. 

Step 4: The guide gets the client’s answers. 

Step 5: The answers are further processed and reviewed. 

The guide sets the rules for the thinking process and manages the flow of work, either statically or dynamically. The guide rewrites the problem differently after getting the problem statement (in step 2). Steps 3 and 4 let the client answer the different problem statements without relying on each other. This is called the “chain of thought.”  The guide compares the possible answers to determine if the client understands the problem and what they should say in response. The client is given a properly written explanation and a summary of the thinking process (protocol). If the AI hasn’t developed consistent lines of Reasoning and answers to similar problem formulations, the client may respond to the first user question.

After receiving the problem statement, the guide tells the client to think of different ways to solve the problem and list the pros and cons of each possible solution. The guide uses the thinking trace made in this way as a starting point for further analysis. In particular, through a series of steps outlined below, it creates an informal argument map that makes the different arguments put forward during brainstorming clear and shows how they are connected to the competing answer choices directly or indirectly. 

A single claim shows each case for the informal argument map. 

Next, the guide uses the argument map to get the client to evaluate the arguments in a planned way.

The client is tasked with evaluating the persuasiveness of claim C by examining all the pros and cons that have been deemed reasonable.

This backward, argument-by-argument review starts with the argument map’s leaf nodes and ends with a check of how plausible the main claim(s) are.

The above figure shows users’ steps to put together a controversial argument as a loose (fuzzy) argument map. This is how Logikon normally does direct Reasoning by weighing the pros and cons. Each step in the Logikon Python program is matched with a different analyst class. The analyst classes mostly use internal LLM processes to make the needed logical artifacts. 

The IssueBuilder takes the rough thinking reasoning trace and, with the help of expert LLMs, describes the main issue the text is about, which is usually a new way of stating the original problem. 

The ProsConsBuilder uses the thinking traces to build a list of pros and cons with multiple roots that address the main issue that was already identified. There are several steps to this method itself: First, from the reasoning trace, all reason statements relevant to the problem are taken out, no matter their valence. In the second step, these reasons are combined in one or more lists of pros and cons. This is the only step where the core root claims are found and added. The final lists of pros and cons are checked for duplicates and thoroughness (based on the reasons given at the start) and changed if needed. 

The RelevanceNetworkBuilder uses a set of prompt templates to determine how likely it is that any two reason statements are relevant to each other and any pair of a reason statement and a core claim. This makes a full graph of all the reason statements and main claims, with weighted support and attack relationships. (Any two root claims are thought to contradict each other maximally.) 

The FuzzyArgmapBuilder takes the entire graph and uses an optimal branching method to create a tree that connects all the argument nodes with the strongest edges. It then adds more edges with weights higher than a certain level. This process results in a fuzzy argument map, which is then exported in various useful formats. The purpose of the FuzzyArgmapBuilder is to provide a comprehensive and visually intuitive representation of the argumentation process, making it easier to understand and analyze.

Check out the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and LinkedIn. Join our Telegram Channel. If you like our work, you will love our newsletter..

Don’t Forget to join our 50k+ ML SubReddit

Dhanshree Shenwai is a Computer Science Engineer and has a good experience in FinTech companies covering Financial, Cards & Payments and Banking domain with keen interest in applications of AI. She is enthusiastic about exploring new technologies and advancements in today’s evolving world making everyone’s life easy.