StableToolBench

Benchmarks are designed to consistently evaluate the performance of various models over time. To test this consistency of ToolBench, we reproduce the model performances and record any variations. As depicted in Figure 1, a notable decline in the performance of all methods over time is observed, which raises concerns about the stability of ToolBench as a benchmark. Click here to overview the detailed analysis of the impacts of API status and the evaluation system in ToolBench on stability of the benchmark.

×

Instability of API Status

ToolBench exihibits significant instability on the API status (Figure 1). This instability was primarily due to issues such as expired APIs, network issues, or failed authentication. Only 44.4% of API calls were successful, with the rest being mostly unavailable due to various errors, some not authorized, and others suffering from parsing errors and parameter changes. This instability in API status led to considerable variability in model performance assessments and had a significant impact on the stability of the benchmark (Figure 2). When APIs failed or behaved unpredictably, it directly affected the tools' ability to function correctly and consistently, thereby impacting the effectiveness of the tools and the overall reliability of the benchmark

Instability of Evaluation

The evaluation metrics used in ToolBench, such as Pass Rate (PR) and Win Rate (WR), are subject to randomness, particularly when dealing with tasks labeled as "unsolvable" or "unsure". This randomness contributes to inconsistencies in model evaluations. For example, despite using the same tasks, the results varied significantly between different strategies like Chain-of-Thought (CoT) and Depth First Search (DFS), and the discrepancy in evaluation was noted to be a problem.

The Virtual API Server

To stabilise the API server, we propose to use the virtual API server. It comprises two primary components: a caching system and an API simulator. The caching system stores responses from all API calls, ensuring consistency and reducing latency. It's populated with data from both the training and test phases and continuously updated to maintain scalability and quality. The API simulator, powered by a large language model (gpt-4-turbo), simulates API responses that aren't in the cache or are unavailable. It utilizes documentation and real API call examples as few-shot prompts to ensure the simulated responses closely mimic real API behavior. Together, these components work under specific calling rules, initially checking the cache for a response before attempting a real API call and, if necessary, resorting to the simulated response. This integrated approach aims to balance stability and reality in API behaviors, significantly enhancing the benchmark's reliability and effectiveness.

The Stable Evaluation System

Solvable Tasks Filtration. Since the solvablility of tasks in original ToolBench induces siginificant instability, we filter out the unsolvable tasks in advance. This process is executed using GPT-4, Gemini Pro, and Claude 2. Each task from the dataset is evaluated by these models to determine its solvability through majority voting. A task is classified as solvable if it provides all the necessary and valid information required for completion and can be resolved with the available tools. Human evaluation shows that these models can effectively filter out unsolvable tasks, ensuring the stability of the benchmark.

Table 1: Summary of Task Statistics before and after filtration

Metrics (SoPR and SoWR). Due to the limitation of gpt-3.5-turbo-16k in tool learning, we uniformly adopt gpt-4-turbo-preview as the automatic evaluator. SoPR is in essence PR with all tasks solvable and only assesses the answers using the same prompt in ToolBench. The evaluator assigns outcomes of answers categorised as Solved, Unsolved, or Unsure, which respectively contribute scores of 1, 0.5, and 0 to the overall SoPR calculation. As for SoWR, when one is solved and the other is unsolved, the solved one wins. Under other circumstances, gpt-4-turbo-preview will be used to make a win-lose decision.

Stability of Our System

Table 2: Solvable Pass Rate scores. In this experiment, we run all models once, evaluate three times and take the average results.

Table 3: Solvable Win Rate scores. We run all models once against GPT-3.5-Turbo-0613 + CoT and evaluate three times. We follow the ToolBench implementation to take the most frequent result for each query during evaluation.