DOI QR코드

DOI QR Code

Value-Based Adaptive MCTS for Efficient LLM Reasoning

가치 기반 적응형 MCTS를 통한 LLM 추론 효율화

  • Received : 2025.08.25
  • Accepted : 2025.12.02
  • Published : 2025.12.30

Abstract

Large Language Models (LLMs) improve reasoning by scaling test-time computation using techniques like Chain-of-Thought prompting and Monte Carlo Tree Search (MCTS). While recent methods such as ReST-MCTS* leverage learned process rewards and per-step value estimates, they rely on fixed branching and iteration budgets, which can be inefficient across tasks of varying difficulty. This work proposes an adaptive tree search framework that dynamically adjusts branching factors and search iteration based on difficulty which is measured by the node value. This approach allocates resources more effectively to complex tasks, improving accuracy and reducing runtime without extra training. We further evaluate the method under various configurations, including different value thresholds and maximum iteration limits. Experiments on the MATH and GSM8K dataset show consistent improvements over fixed-budget baselines across these settings, highlighting the robustness and efficiency benefits of adaptive resource allocation in LLM reasoning.

Keywords

Ⅰ. Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, especially when combined with test-time inference strategies that scale computational resources to improve accuracy. Techniques such as Chain-of-Thought (CoT) prompting[1], tree-based reasoning[2], and Monte Carlo Tree Search (MCTS)[3] have been developed to elicit more structured and interpretable reasoning from LLMs. These approaches generate multiple candidate reasoning paths and select among them using heuristics or model-based evaluations, enabling models to "think more" before answering and improving performance without additional training.

Recent advancements in self-training have extended this line of work by using LLM-generated reasoning traces as supervision. Notably, ReST-MCTS* introduces a reinforced self-training framework that integrates MCTS with a learned process reward function to guide reasoning trace generation[4]. Instead of relying solely on final-answer correctness, ReST-MCTS* estimates per-step value: the likelihood that a given step contributes to arriving at the correct answer. These inferred values are used both to improve the reward model and to select higher-quality traces for policy training, enabling iterative self-improvement. The approach achieves state-of-the-art performance across multiple reasoning tasks, outperforming methods like Best-of-N[5], Tree-of-Thought[2], ReSTEM[6], and Self-Rewarding LM[7] under fixed computational budgets.

However, existing methods, including ReST-MCTS*, typically operate under fixed branching factors and iteration limits during tree search. This uniform allocation of compute fails to account for the diverse reasoning complexity across questions. Some problems require deeper exploration or more diverse reasoning paths, while others may be solvable with minimal effort. Treating all instances equally can lead to wasted computation on easy cases and insufficient search on harder ones.

This work proposes an adaptive tree-based reasoning framework that dynamically adjusts the tree search configuration—branching factor and iteration limit—based on per-step estimates of value or uncertainty. By allocating resources in proportion to expected information gain, our method seeks to improve both sample efficiency and final performance. This adaptive extension of ReST-MCTS* aims to better match search effort to problem difficulty, pushing LLM reasoning further toward both scalability and efficiency.

Ⅱ. Related Works

2.1 Scaling Test-time Compute

Effectiveness of scaling test-time compute has been demonstrated in several recent studies[8–10], showing that allocating more inference-time resources can yield greater performance gains than simply scaling model parameters. Methods such as chain-of-thought prompting[1], which encourages the model to produce explicit intermediate reasoning steps, and self-consistency[11], which samples multiple reasoning paths and selects the majority answer, have further enhanced reasoning performance by improving logical coherence and reducing variance in outputs. Another promising direction is the use of verifiers, including process reward models, to guide reasoning. These can be manually designed using labeled step-level correctness[5, 12] or learned without human labels by estimating per-step correctness from rollouts[4, 13], enabling scalable reasoning supervision without costly annotation.

Search-based strategies have also been explored to improve reasoning quality. Best-of-N[5] selects the most promising final answer from multiple independent completions. Beam search[14] expands multiple partial solutions in parallel, maintaining the top candidates at each step. Tree-structured approaches[2, 15] explicitly branch reasoning paths based on heuristics, allowing exploration of diverse solution trajectories. Monte Carlo Tree Search (MCTS)[3, 4] balances exploration and exploitation in this reasoning space by simulating possible continuations and prioritizing high-value paths. In particular, recent work on ReST-MCTS*[4] integrates process-level verification into MCTS to enable self-training without human annotations, demonstrating strong gains in mathematical problem-solving tasks.

2.2 Adaptive Reasoning in LLMs

As scaling test-time compute becomes increasingly impactful[8-10], efficiency during reasoning has emerged as a critical concern[16]. This has spurred a range of adaptive reasoning techniques that dynamically modulate computational effort according to input difficulty, such as learning to predict the optimal amount of thinking time[17], switching between fast and slow reasoning modes[18, 19], combining long and short chain-of-thought strategies[20], dynamically selecting problem-solving strategies[21], and rolling back erroneous reasoning paths for correction[22]. In our work, we leverage this adaptive reasoning trend by integrating a verifier into the MCTS framework, enabling the system to allocate more search compute to harder instances and less to easier ones effectively, thereby optimizing both computation and outcome quality.

Ⅲ. Preliminaries

3.1 Monte Carlo Tree Search

Monte Carlo Tree Search (MCTS) is a search algorithm used to explore decision spaces in a best-first manner. In the context of ReST-MCTS* [4], MCTS is adapted to guide large language model (LLM) reasoning by selecting, evaluating, and refining multi-step reasoning paths. Each node in the search tree represents a partial solution (i.e., a sequence of reasoning steps), and edges represent candidate next steps generated by an LLM. The objective is to find a reasoning path that leads to a correct final answer, guided by a learned value model that estimates the quality of partial solutions.

ReST-MCTS* uses a modified MCTS variant called MCTS*, which operates in four stages.

3.1.1 Node Selection

Starting from the root node (the initial problem prompt), the algorithm recursively selects child nodes using an Upper Confidence Bound (UCB) formula:

\(\begin{align}U C B(C)=v_{C}+\epsilon \sqrt{\frac{\ln n_{\text {parent }}}{n_{C}}}\end{align}\)

where vC is the predicted quality of node C by the value model, nC is the visit count of C, and ∊ is an exploration constant. This balances exploitation (following high-value paths) and exploration (visiting less explored nodes).

3.1.2 Expansion

When a leaf node is reached, the LLM generates b candidate next steps, forming new child nodes. Each new node is evaluated by the value model to obtain an initial quality score.

3.1.3 Simulation (Rollout)

From the most promising newly expanded node, the algorithm performs a limited rollout: it greedily selects the next step with the highest predicted value at each stage, simulating a full reasoning trajectory. The maximum value observed during this simulation is backpropagated to improve the parent node's estimate.

3.1.4 Backpropagation

The values of nodes along the path from the expanded node back to the root are updated using a weighted average based on children's visit counts. This ensures that frequently visited and high-value paths influence future decisions more strongly.

To further improve efficiency, ReST-MCTS* introduces a self-critic mechanism that allows the LLM to predict whether a reasoning path has reached a final answer. If the model outputs an ‘<end>’ signal, expansion and rollout are skipped for that node, preventing unnecessary computation.

Ⅳ. Method

This work proposes an adaptive inference framework that augments the ReST-MCTS* algorithm [4] by dynamically adjusting its branching factor and iteration budget during test-time tree search. The key intuition is to allocate computational resources more efficiently based on either (a) the reasoning step index k, or (b) the estimated node value vC, which reflects the probability that a partial reasoning trace will lead to a correct final answer. Our method consists of two complementary components: adaptive branching and adaptive iteration control.

4.1 Adaptive Branching

In standard MCTS, the branching factor—i.e., the number of children expanded per node—is typically fixed throughout the search. This work introduces dynamic branching strategies that adapt this factor based on either the node's step index or its estimated value.

4.1.1 Step-Based Branching

This strategy adjusts the branching factor according to the step index k, which represents the depth of the node C in the reasoning trace [s1, …, sk].

(1) Increasing Branching

For early steps (k < K), the branching factor is set to a minimal value (e.g., 1) to limit exploration when little context is available. As reasoning progresses (k ≥ K), the branching factor increases to its maximum value bmax to promote diverse reasoning continuations.

(2) Decreasing Branching

This variant begins with a high branching factor for early steps to explore broadly when reasoning is uncertain, then gradually reduces it as k increases, focusing compute resources on refining promising paths as more context becomes available.

4.1.2 Value-Based Branching

Here, the branching factor is adapted based on the estimated value vC of the current node C.

(1) Increasing Branching

If vC < V, the node is deemed unpromising, and the branching factor is reduced to avoid over-exploration. When vC ≥ V, the node is considered promising, and the branching factor is increased to bmax to explore more continuations.

(2) Decreasing Branching

Alternatively, this variant allocates a high branching factor when vC is low, under the hypothesis that broad exploration may recover useful paths. As vC increases, indicating higher promise, the branching factor is reduced to focus on a smaller set of high-quality directions.

4.2 Adaptive Iteration Control

Traditional MCTS uses a fixed number of iterations for all queries, regardless of their complexity. This work uses a dynamic scheme that adjusts the iteration count based on diagnostics from early exploration and intermediate value signals.

4.2.1 Warm-Up Phase: Initial Iteration Estimation

The search begins with a brief warm-up phase, during which a small number of iterations are performed uniformly. This serves two purposes. First, the value estimates vC of early leaf nodes help assess the structure of the search space. Second, if promising (high vC) paths are found early, the query is likely easy, whereas if only low-value paths are found, the query is likely difficult.

This phase does not commit to a final answer but informs the subsequent iteration strategy. If promising paths are already identified, further search may be curtailed. Otherwise, more search iterations are allocated to compensate for the lack of strong candidates.

4.2.2 Final Iteration Adjustment

Let vbest denote the value of the highest-scoring leaf node found after the warm-up. This signal is used to determine whether further search is warranted. If vbest ≥ V, the search concludes early or proceeds with a reduced iteration count. If vbest < V, the algorithm proceeds up to the full iteration limit T to refine the reasoning trajectory.

This dynamic iteration policy ensures that compute resources are allocated proportionally to the difficulty of each problem: simpler instances receive minimal exploration, while complex or ambiguous ones benefit from deeper search.

Ⅴ. Experiments

5.1 Experimental Setup

The adaptive inference method is evaluated on the MATH dataset [23] and GSM8K [24], which consist of diverse and challenging mathematical reasoning problems. First, we perform a hyperparameter search analysis (Section 5.4) using a subset of 100 randomly selected problems from the MATH test set. For the final evaluation (Section 5.5), we randomly sample 200 problems from each of the MATH and GSM8K datasets. This sampling is performed uniformly at randomto ensure representativeness while keeping computational costs manageable. All models operate under the ReST-MCTS* framework[4], with modifications applied to the branching and iteration behaviors as described in Section 4.

LLaMA-3-8B model[25] is used as the policy model to generate reasoning steps. For value estimation, Skywork o1 Open-PRM-Qwen-2.5-7B process reward model[26] is employed. This 7B-parameter reward model, built on the Qwen-2.5-7B backbone, is fine-tuned on partial reasoning traces to predict step-level process rewards, which is the likelihood that an intermediate step contributes toward a correct final answer, enabling value-guided search within the ReST-MCTS* framework.

Two primary metrics are reported: Pass@1 accuracy, reflecting the percentage of problems correctly solved using the top final answer returned by MCTS, and average search time per sample, measuring computational efficiency.

5.2 Adaptive Branching Configuration

Two adaptive branching strategies are evaluated—based on reasoning step index k and node value estimate vC. In both cases, the branching factor is dynamically adjusted between a minimum of 1 and a maximum of bmax = 3.

5.2.1 Step-Based Branching

A fixed threshold at step index K = 3 determines the branching factor. Under increasing branching, nodes with k < 3 expand 1 child, and nodes with k ≥ 3 expand bmax children. In the decreasing branching variant, this configuration is reversed.

5.2.2 Value-Based Branching

The branching factor is adapted based on the estimated node value vC. Unless explicitly stated otherwise, the default value threshold used in the experiments is set to V = 0.5. For increasing branching, nodes with vC < 0.5 expand 1 child, while those with vC ≥ 0.5 expand bmax children. The decreasing branching variant expands bmax children for low-value nodes and 1 child for high-value nodes.

5.3 Adaptive Iteration Configuration

A two-stage dynamic iteration control scheme is employed to adjust the total number of search iterations based on early signals from the search tree.

5.3.1 Warm-Up Phase

Each query begins with a fixed warm-up phase consisting of T0 = 5 iterations. This phase gathers diagnostic signals from early leaf nodes to estimate problem difficulty, as described in Section 4.

5.3.2 Iteration Adjustment

Based on the highest observed node value vbest after the warm-up, the remaining iteration budget is adjusted. The value threshold for iteration adjustment is also set to V = 0.5 if not otherwise specified. If vbest ≥ 0.5, a smaller number of additional iterations is allocated (Tlow = 5, totaling T = 10). If vbest < 0.5, a larger total budget of T = 20, 30, or 50 is allocated. These thresholds are varied to examine the tradeoff between performance and computational cost.

5.4 Hyperparameter Search Analysis

We first conduct a preliminary hyperparameter search to investigate the effects of different settings for the iteration limit, branching increase/decrease strategies, and thresholds.

5.4.1 Adaptive Branching by Step

Table 1 shows how step-based branching improves performance under tight iteration budgets. Decreasing branching performs best at 20 iterations, while increasing branching is more stable at 30 iterations, suggesting that step-based strategies help early exploration but may underperform with extended search.

<Table 1> Step-Based Adaptive Branching Results

DGTSA8_2025_v21n4_77_7_t0001.png 이미지

5.4.2 Adaptive Branching by Value

Table 2 shows the performance of value-based branching. Decreasing branching, which prioritizes expansion of low-value nodes, consistently achieves the highest accuracy across budgets, indicating that broader exploration under uncertain value estimates is advantageous. Increasing branching performs worse at higher iteration limits, likely due to premature pruning of ambiguous nodes.

<Table 2> Value-Based Adaptive Branching Results

DGTSA8_2025_v21n4_77_8_t0001.png 이미지

To further examine the decreasing-branch method, we conducted experiments varying the value threshold, with the maximum iteration limit fixed at 30. Figure 1 plots accuracy and runtime for different thresholds, with blue and red dashed lines marking the accuracy and runtime of the fixed-branch baseline, respectively. All thresholds yield lower runtimes than the baseline. A threshold of 0.9 achieves the highest accuracy (44%), indicating that reducing branching only for highly certain nodes (value> 0.9) can improve overall performance.

<Figure 1> Accuracy and Time per Different Value Thresholds for Adaptive Decreasing Branching

DGTSA8_2025_v21n4_77_8_f0001.png 이미지

5.4.3 Adaptive Iteration Control

Table 3 shows that adaptive iteration reduces runtime while improving accuracy by allocating search effort based on problem difficulty. After an initial warm-up phase, further search is conditionally applied depending on the value of the most promising partial solution, reducing computation on easier problems. This strategy is particularly effective at moderate iteration limits of 30, offering a balance between efficiency and performance.

<Table 3> Adaptive Iteration Control Results

DGTSA8_2025_v21n4_77_8_t0002.png 이미지

We also tested different value thresholds for the adaptive iteration mechanism (Figure 2). The blue dashed line denotes the fixed-iteration baseline accuracy, and the red dashed line its runtime. While higher thresholds generally lead to longer runtimes, they still remain below the baseline runtime. A threshold of 0.5 yields the highest accuracy, suggesting that more iterations are not always beneficial and that moderate thresholds may balance exploration and efficiency most effectively.

DGTSA8_2025_v21n4_77_9_f0001.png 이미지

<Figure 2> Accuracy and Time per different value Thresholds for Adaptive Iteration

5.5 Results

To strengthen the evaluation, we extend our experiments to two benchmarks, MATH and GSM8K. For each dataset, all experiments are conducted over three independent trials to account for variability. We adopt the settings identified as optimal in Section 5.4 under an iteration limit of 30. Specifically, we use a value threshold of 0.9 for adaptive decreasing branching and a threshold of 0.5 for adaptive iteration control.

Table 4 shows the evaluation of our adaptive inference methods on the MATH and GSM8K datasets. Each experiment was repeated three times, and mean with standard error (SE) is reported. Both adaptive branching and adaptive iteration method consistently outperform the baseline in terms of accuracy. Notably, the improvements exceed the standard errors, indicating that the observed gains are statistically consistent across trials. In addition to accuracy improvements, adaptive iteration also reduces computation time on MATH, while adaptive branching maintains comparable runtime, demonstrating that the proposed adaptive mechanisms enhance LLM reasoning efficiency reliably and with controlled computational cost.

<Table 4> Results with Mean and Standard Error

DGTSA8_2025_v21n4_77_9_t0001.png 이미지

Ⅵ. Conclusion

This work explores adaptive mechanisms for improving efficiency and performance in LLM reasoning using MCTS. While ReST-MCTS* relies on fixed search parameters, adaptive branching and iteration control strategies dynamically adjust computation based on reasoning step or value estimates.

Experiments show that these adaptive strategies can outperform fixed-budget baselines, particularly when moderate search budgets are available. Step-based branching improves early-stage efficiency, while value-based branching helps guide exploration based on step node value. Adaptive iteration control further enhances search efficiency by allocating compute based on estimated difficulty, improving accuracy while reducing average runtime.

These results demonstrate the potential of lightweight, value-guided adaptivity in scaling test-time reasoning, opening new directions for more flexible and resource-aware inference with large language models.

The practical applications of such adaptive mechanisms extend far beyond academic testing. In industry, this could lead to AI tutors that tailor problem difficulty to individual students, legal AI that dedicates more analytical power to ambiguous cases, and medical diagnostic tools that apply greater computational resources to complex patient scenarios.

As future work, we plan to improve the robustness of adaptivity by developing methods that are less sensitive to noisy early-step value estimates, which can cause suboptimal branching or premature termination. We also aim to move beyond simple threshold-based heuristics toward learned or more principled control policies that adapt more flexibly across tasks and domains. Finally, we will extend evaluation to a wider set of reasoning benchmarks to better assess generality and real-world applicability.

References

  1. J. Wei, X. Wang, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, "Chain-of-thought prompting elicits reasoning in large language models," arXiv preprint arXiv:2201.11903, 2022.
  2. Y. Yao, D. Fu, and J. Tang, "Tree of thoughts: Deliberate problem solving with large language models," arXiv preprint arXiv:2305.10601, 2023.
  3. X. Feng, Z. Wan, M. Wen, S. M. McAleer, Y. Wen, W. Zhang, and J. Wang, "AlphaZero-like tree-search can guide large language model decoding and training," arXiv preprint arXiv:2404.00518, 2024.
  4. T. Zhao, H. Zhang, Z. Qi, and M. Yang, "ReST-MCTS*:"ReST-MCTS*: LLM self-training via process reward guided tree search," arXiv preprint arXiv:2407.01495, 2024.
  5. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, "Let's verify step by step," arXiv preprint arXiv:2305.20050, 2023.
  6. A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, J. Harrison, J. Lee, K. Xu, et al., "Beyond human data: Scaling self-training for problem-solving with language models," arXiv preprint arXiv:2312.06585, 2023.
  7. W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston, "Self-rewarding language models," Proceedings of the International Conference on Machine Learning (ICML), PMLR, vol. 235, 2024, pp. 57905–57923.
  8. C. Snell, J. Lee, K. Xu, and A. Kumar, "Scaling LLM test-time compute optimally can be more effective than scaling model parameters," arXiv preprint arXiv:2408.03314, 2024.
  9. Y. Wu, Z. Sun, S. Li, S. Welleck, and Y. Yang, "An empirical analysis of compute-optimal inference for problem-solving with language models," arXiv preprint arXiv:2408.00724, 2024.
  10. B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini, "Large language monkeys: Scaling inference compute with repeated sampling," arXiv preprint arXiv:2407.21787, 2024.
  11. X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, "Self-consistency improves chain of thought reasoning in language models," International Conference on Learning Representations, 2023.
  12. J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins, "Solving math word problems with process- and outcome-based feedback," arXiv preprint arXiv:2211.14275, 2022.
  13. P. Wang, L. Li, Z. Shao, R. X. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui, "Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations," arXiv preprint arXiv:2308.09138, 2023.
  14. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, "PAL: Program-aided language models," arXiv preprint arXiv:2211.10435, 2022.
  15. S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu, "Reasoning with language model is planning with world model," arXiv preprint arXiv:2305.14992, 2023.
  16. M. Alomrani, "A survey of adaptive and controllable test-time compute," arXiv preprint arXiv:2507.02076, 2025.
  17. M. Damani, I. Shenfeld, A. Peng, A. Bobu, and J. Andreas, "Learning how hard to think: Input-adaptive allocation of LM computation," International Conference on Learning Representations, 2025.
  18. J. Pan, Y. Zhang, Y. Sun, T. Qin, and T. Liu, "Learning adaptive parallel reasoning with language models," Empirical Methods in Natural Language Processing, 2025.
  19. X. Zhang, Z. Chen, Y. Lin, Y. Zhang, and T. Liu, "When to continue thinking: Adaptive thinking mode switching for efficient reasoning," arXiv preprint arXiv:2505.15400, 2025.
  20. H. Luo, Z. Wang, X. Chen, and M. Zhang, "AdaR1: From long-CoT to hybrid-CoT via bi-level adaptive reasoning optimization," arXiv preprint arXiv:2504.21659, 2025.
  21. J. Zhou, Z. Liu, and W. X. Zhao, "Adaptive-solver framework for dynamic strategy selection in large language model reasoning," arXiv preprint arXiv:2310.01446, 2023.
  22. S. Chen, "Thought rollback (TR): Adaptive reasoning with rollback," arXiv preprint arXiv:2412.19707, 2024.
  23. D. Hendrycks, C. Burns, S. Basart, et al., "Measuring mathematical problem solving with the MATH dataset," Advances in Neural Information Processing Systems, vol. 34, 2021.
  24. K. Cobbe, V. Kosaraju, M. Bavarian, et al., "Training verifiers to solve math word problems," arXiv preprint arXiv:2110.14168, 2021.
  25. Meta AI, "LLaMA 3: Open foundation and chat models," arXiv preprint arXiv:2407.21783, 2024.
  26. Skywork AI, "Skywork Open PRM: Skywork-o1-Open-PRM-Qwen-2.5-7B," https://huggingface.co/SkyworkAI/Skywork-o1-Open-PRM-Qwen-2.5-7B, 2024.