AgentFlow

Outcome-driven reinforcement learning has advanced reasoning in large language models (LLMs), but prevailing tool-augmented approaches train a single, monolithic policy that interleaves thoughts and tool calls under full context; this scales poorly with long horizons and diverse tools and generalizes weakly to new scenarios. Agentic systems offer a promising alternative by decomposing work across specialized modules, yet most remain training-free or rely on offline training decoupled from the live dynamics of multi-turn interaction.

We introduce AgentFlow, a trainable, in-the-flow agentic framework that coordinates four modules (planner, executor, verifier, generator) through an evolving memory and directly optimizes its planner inside the multi-turn loop. To train on-policy in live environments, we propose Flow-based Group Refined Policy Optimization (Flow-GRPO), which tackles long-horizon, sparse-reward credit assignment by converting multi-turn optimization into a sequence of tractable single-turn policy updates. It broadcasts a single, verifiable trajectory-level outcome to every turn to align local planner decisions with global success and stabilizes learning with group-normalized advantages.

Across ten benchmarks, AgentFlow with a 7B-scale backbone outperforms top-performing baselines with average accuracy gains of 14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on scientific tasks, even surpassing larger proprietary models like GPT-4o. Further analyses confirm the benefits of in-the-flow optimization, showing improved planning, enhanced tool-calling reliability, and positive scaling with model size and reasoning turns.

AgentFlow is a general-purpose tool-integrated agentic framework for solving complex reasoning tasks through fine-grained planning and effective tool use. It comprises four specialized modules—Planner $\mathcal{P}$, Executor $\mathcal{E}$, Verifier $\mathcal{V}$, and Generator $\mathcal{G}$—coordinated by shared memory $M$ and a toolset $K$. We formalize AgentFlow's problem-solving process as a multi-turn Markov Decision Process (MDP): given query $q$ and toolset $K$, the planner $\mathcal{P}$ (a trainable policy $\pi_\theta$) produces an action $a^t \sim \pi_\theta(a^t \mid q, K, M^t)$ that formulates a sub-goal, selects a tool $k \in K$, and retrieves relevant context from memory $M^t$. The executor $\mathcal{E}$ invokes tools according to $a^t$, yielding execution results $e^t \sim \mathcal{E}(e^t \mid a^t, K)$. The verifier $\mathcal{V}$ evaluates $e^t$, producing a binary verification signal $v^t \sim \mathcal{V}(v^t \mid q, e^t, M^t)$. If $v^t = 0$, the memory is updated deterministically: $M^{t+1} = f_{\text{mem}}(M^t, a^t, e^t, v^t)$. This process repeats until $v^t = 1$ (termination) or a maximum turn budget is reached. Upon termination at turn $T$, the generator $\mathcal{G}$ produces the final solution $o \sim \mathcal{G}(o \mid q, M^T)$. After $T$ turns, the trajectory $\tau = \{(a^t, e^t, v^t)\}_{t=1}^T$ records planning, execution, and verification steps. The joint generative process is:

To comprehensively evaluate tool-use capabilities of AgentFlow, we conduct experiments on four types of reasoning tasks: (1) Knowledge-intensive search including Bamboogle, 2Wiki, HotpotQA, and Musique; (2) Agentic reasoning such as GAIA (where we adopt the textual split); (3) Logic-dense mathematical reasoning including AIME 2024, AMC 23, and Game Of 24; and (4) Scientific reasoning including GPQA and MedQA.

Accuracy comparison on search-intensive and agentic tasks. 7B-Base refers to Qwen-2.5-7B-Base and 7B-Inst refers to Qwen-2.5-7B-Instruct. AutoGen and our AgentFlow method are agentic systems, which use Qwen-2.5-7B-Instruct for the LLM-powered agents and tools for fair comparison. We visualize the gains of AgentFlow to each baseline in the Δ columns.

Baselines: We compare against four categories of baselines: (1) Open-source LLMs: Qwen-2.5 (7B, 14B, 32B) and Llama-3.3-70B; (2) Proprietary LLMs: GPT-4o-mini and GPT-4o; (3) Tool-integrated reasoning LLMs: Supervised Fine-Tuning (SFT), Iter-RetGen, Search-R1, ZeroSearch, ReSearch, StepSearch, and VerlTool; (4) Training-free agentic system: AutoGen.

Accuracy comparison of mathematical and scientific reasoning tasks. We visualize the gains of AgentFlow to each baseline in the Δ columns.

Baselines: We compare against four categories of baselines: (1) Open-source LLMs: Qwen-2.5 (7B, 14B) and Llama-3.3-70B, Llama-3.1-405B; (2) Proprietary LLMs: GPT-4o-mini and GPT-4o; (3) Reasoning LLMs: Supervised Fine-Tuning (SFT), SimpleRL-reason, Open-Reasoner-Zero, General-Reasoner, and Luffy; (4) Tool-integrated reasoning LLMs: TIR and ToRL; (5) Training-free agentic system: AutoGen.

We conduct comprehensive analyses to understand the effectiveness of Flow-GRPO and the behavior of AgentFlow across various dimensions.

Impact of Planner Training Strategies. Experiments demonstrate that training the planner with the online reinforcement learning method, Flow-GRPO, yields a significant 17.2% performance improvement, whereas traditional offline Supervised Fine-Tuning (SFT) results in a catastrophic 19.0% performance collapse.

Optimized and Adaptive Tool Selection. After optimization with Flow-GRPO, the planner learns to select the most appropriate tools for different tasks, such as increasing the use of Google Search for the broad-knowledge 2Wiki task while shifting to the more specialized Wikipedia and Web Search for the domain-specific MedQA task.

Enhanced Tool-Calling Reliability. The Flow-GRPO training process enhances tool-calling reliability, as evidenced by a consistent decrease in the tool-calling error rate across all tasks, with a reduction of up to 28.4% on the GAIA task.

Superior Training Efficiency and Stability. Analysis of training dynamics reveals that Flow-GRPO not only continuously increases rewards (accuracy) while shortening response length but also achieves more stable and sustained performance growth compared to traditional monolithic methods like ToRL.

Consistent Gains Across Model Scales. Flow-GRPO's online fine-tuning method delivers consistent and effective performance gains on AgentFlow, regardless of whether the backbone model scales from 3B to 7B parameters.

Performance Scaling with Inference Turns. During the inference phase, increasing the maximum allowed interaction turns from 3 to 10 enables AgentFlow to conduct deeper reasoning, leading to continuous improvements in final performance across all tasks.

Adaptability to Upgraded Tool Engines. The trained AgentFlow system demonstrates strong adaptability, as its overall performance significantly improves when its internal tool engines are upgraded from Qwen-2.5-7B-Instruct to the more powerful GPT-4o.