Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics

2026-07-14 · arXiv:2607.06820 · code and data

AI4Math research focuses on Lean autoformalization and theorem proving. But how do we come up with new conjectures? In our work we study how well can LLM agents use computation to create hypotheses and solve research-level mathematical problems?

Experimental setup

We evaluated 15 frontier models under two matched conditions on 133 curated research-level problems from RealMath. In the tool-free baseline, a model reasons directly. In the agentic setup, the same model enters a ReAct loop with SageMath for executable symbolic computation.

SageMath is an open-source computer algebra system with a unified Python interface to mathematical software such as GAP, Singular, and PARI/GP. It supports symbolic algebra, numerical experiments, polynomial ideals, graph computations, and number theory. Agents generate Sage code in an isolated sandbox, inspect verifiable outputs, and iteratively refine their hypotheses. Context7 provides current SageMath documentation and code examples.

Each run has a maximum of 15 tool calls. Across both settings this produced more than 3,990 independent runs. Answers pass through symbolic equivalence checking and, when that check is inconclusive, a three-model LLM-as-a-Judge panel.

sage@cell:~$ Try running your Sage code remotely on SageMathCell.

output

Main solve-rate observations

SageMath access improved every evaluated model, by +9.7 percentage points on average. The effect was highly model-dependent: Qwen 3.7-Max gained +27.8 pp, MiniMax M3 +17.3 pp, and DeepSeek 3.2 +15.8 pp. GPT-5.5 reached the highest solve rate, improving from 67.7% to 75.2%; Opus 4.8 followed at 73.7%. The strongest open-weight agents approached the tool-free performance of the best closed models.

Fugu-Ultra, a multi-agent orchestration system rather than a single LLM, reached 72.9%, matching Opus 4.7. A weak tool-free baseline did not automatically imply a large tool gain: Grok 4.3 started close to Qwen but improved by only +3.8 pp, while Kimi gained +1.5 pp despite issuing 1,421 tool calls. In both cases, incorrect tool interaction obscured much of the potential benefit of the CAS.

Figure 2. Solve rate for each model in zero-shot and agentic settings; hover over a model to highlight both bars and inspect its tool-use profile. Error bars show 95% Wilson confidence intervals.

Token usage efficiency

Tool-enabled agents used between roughly 34k and 556k tokens per problem, but larger budgets did not produce proportionally higher accuracy. GPT-5.5 occupied the best point on the accuracy-cost frontier: the highest solve rate at the lowest token usage among tool-enabled configurations. MiniMax used 16× more tokens per problem while solving 19.6 pp fewer problems.

The nearest configurations to GPT-5.5 were Fugu-Ultra at 72.9% with 70k tokens, Opus 4.7 at 72.9% with 97k, and Opus 4.8 at 73.7% with 119k. Models from the same provider often occupied nearby regions, suggesting related post-training or tool-use policies. Low consumption alone was not efficiency: Grok was the second-cheapest agent because its degenerate one-line code produced little output, while Kimi spent 205k tokens per problem for a gain of only +1.5 pp.

Figure 3. Accuracy-cost trade-off. Circles denote zero-shot runs and squares denote agentic runs; hover over a marker to inspect its solve rate, token usage, and model profile.

Solve rate by mathematical category

The impact of SageMath follows the computational structure of a field. Combinatorics gained the most (+18.7 pp), followed by classical analysis (+12.0 pp) and rings and algebras (+10.7 pp). Number theory improved modestly from a strong baseline, probability was nearly saturated, and PDEs benefited least. Group theory remained difficult despite extensive Sage/GAP support, while algebraic topology showed the largest disagreement between models.

No model exceeded 44% in group theory, and GPT-5.5 was the only model whose category score decreased with tool access. In algebraic topology, the mean tool-enabled solve rate was only 23%, yet Fugu-Ultra, GPT-5.5, and Opus 4.7 solved more than half of the problems while seven models remained at 0%. Tool-call intensity was only weakly correlated with success: probability reached 86% with 4.4 Sage calls per problem, whereas group theory and algebraic topology prompted almost twice as many calls without comparable gains.

Figure 6. Agentic solve rates by model and arXiv category. Hover over a cell to compare it with the zero-shot baseline and reveal the tool lift.

Trace-level behavior

Aggregate accuracy hides how agents actually compute, so we propose a tiled visualization for tool traces. Each vertical stack represents one model solving one problem; tiles encode successful Sage calls, code errors, timeouts, crashes, documentation queries, and the final verdict. This makes repeated failures, recovery, and unproductive search visible at a glance.

The closed systems shown here—Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro, and Fugu-Ultra—have broadly similar profiles: comparatively short traces dominated by successful calls, with failures usually isolated. Open-model behavior differs substantially. Qwen performs extended empirical probing, DeepSeek V4-Pro often produces long sequences of successful checks, Kimi mixes errors and timeouts throughout long traces, and Grok alternates between short successes and clusters of code errors.

Tiled visualization of tool-call traces across models and problems — Figure 15. Our trace visualization for 50 selected problems. Every column is a problem and every stack is a sequence of tool interactions.

Sage execution failures

We treat exceptions, 45-second timeouts, and low-level Sage crashes as failed calls. Exception volume ranged from 31–38 for the strongest agents to 459 for DeepSeek 3.2. Four classes—TypeError, NameError, AttributeError, and SyntaxError—accounted for 87% of all exceptions. The profiles are diagnostic: NameError often reveals loss of state between sandbox calls, while 248 of 336 syntax errors came from Grok’s repeated semicolon-joined one-liners.

TypeError was the most common failure with 786 instances, primarily reflecting incorrect handling of Sage’s object system. Of 664 NameError instances, DeepSeek 3.2 produced 151 and Kimi 122 because each call runs in a fresh interpreter and variables do not persist. Opus 4.7 and Fugu-Ultra produced no state-loss errors, while Opus 4.8 produced one. The four strongest models had similar profiles: roughly half of their small number of exceptions were Sage-specific coercion errors.

Failed Sage calls by exception type and model — Figure 13. Failed Sage calls grouped by Python exception type (left) and by model (right).

Tool-use depth

Agentic runs are strongly bimodal. Most models either finish within 3–4 interactions or consume the full 15-call budget; traces of intermediate length are uncommon. The strongest systems cluster in the short mode, while DeepSeek 3.2, DeepSeek V4-Flash, and Kimi reach the final bin on 86–90 of 133 problems.

Opus 4.7 had the shortest traces in the study, with a median of 3 and a mean of 4.8 turns; GPT-5.5, Opus 4.8, and Fugu-Ultra also completed more than half of all problems within four rounds. DeepSeek 3.2 and DeepSeek V4-Flash instead reached the final bin on 90 and 89 problems, with mean traces approaching the tool budget. Qwen and MiniMax had flatter distributions, while Gemini 3.1 Pro combined a strong 3–4 turn peak with a long tail.

Distribution of agentic trace lengths across models — Figure 7. Number of problems completed in each range of agent-tool interactions.

Sage usage statistics

Function-level traces show that agents adapt computation to mathematical domains. GCD dominates number theory; combinations and permutations are concentrated in combinatorics; rank, basis, dimension, and determinant computations spread across algebraic categories; and integration and ODE solvers cluster in analysis. Symbolic solving, expansion, and simplification are the most domain-independent operations, typically serving as intermediate steps before the final answer.

arith.gcd appeared in 1,274 traces, including 1,042 in number theory. itertools.combinations appeared 1,041 times, concentrated in combinatorics, group theory, and spectral graph theory, while 285 of 360 permutation traces came from combinatorics. Matrix rank appeared 895 times across representation theory, algebraic geometry, algebraic topology, classical analysis, and spectral graph theory, showing how frequently agents reduced research questions to rank, basis, dimension, or kernel computations.

Function-level Sage usage by arXiv category — Figure 16. Frequently used computational functions by arXiv primary category, excluding constructors and generic programming utilities.

Sage execution time

Pure sandbox time varied more than fivefold: from 29 seconds per problem for Opus 4.7 and 37 seconds for GPT-5.5 to 142–161 seconds for Sonnet, GLM, and Kimi. The slowest profiles were driven largely by infeasible enumerations reaching the timeout, not by productive computation. For most models, Sage accounted for only 10–26% of total solve time; model reasoning remained the dominant cost.

The sandbox share varied independently of absolute runtime. It reached 55% for Grok 4.3 because the model performed little reasoning around short code snippets, but only 4% for Qwen because long reasoning traces dominated its total time. Closed frontier models generally produced fast-executing code; open models more often attempted to solve the problem through extended computation rather than using SageMath to verify a reasoning-derived candidate.

Mean SageMath sandbox execution time by model — Figure 8. Mean cumulative SageMath runtime per problem, excluding model inference.

Case study: conjecture discovery

We examine one GPT-5.5 solution trajectory for a problem from Himeno and Teragaito’s paper on twisted torus knots. The task asks for closed forms of two torsion orders of K = T(p, kp + 1; 2, 1), neither of which is directly implemented in SageMath. The trajectory reproduces the main stages of a computational mathematician’s workflow:

Capability discovery. The agent constructs a BraidGroup and checks which Alexander-polynomial and Burau-matrix operations SageMath provides.
Experiment and pattern extraction. It computes Alexander polynomials for small parameters, fixes an exponent-representation error, converts exponent gaps into staircase lengths, and conjectures Ord(K) = p − 1.
Independent computational test. It builds presentation matrices over 𝔽₂[U], computes Smith normal forms, and obtains evidence for Ord′(K) = floor((p − 2) / 2).
Stability testing. Repeating the computations for several values of k shows that both patterns remain unchanged before larger instances reach SageMath’s computational limits.

SageMath supplied finite-instance evidence rather than a general proof. The agent interpreted intermediate computations, recognized patterns, formulated conjectures, and tested them across parameters—closely emulating an experimental-mathematics workflow.

Conclusion

Computer algebra gives LLM agents a verifiable substrate for computational and experimental mathematics. Across 15 models, SageMath improved research-level problem solving and substantially narrowed the open–closed model gap. The main separator was not the amount of computation, but whether an agent could choose useful experiments, recover from feedback, and stop once the evidence was sufficient. These capabilities point beyond answer generation toward automated mathematical experimentation and conjecture discovery.