Programmatic Tool Calling Matches JSON Tool Calling Across LLMs

What changed

A recent research paper introduces and evaluates programmatic tool calling (PTC) as an alternative to traditional JSON tool calling for LLM agents. In the PTC paradigm, models generate Python scripts that utilize typed Python stubs to invoke tools. These scripts are then executed in a single agent turn, with results handled internally. This contrasts with native JSON tool calling, where models emit rigid JSON objects to call functions.

The study systematically compared PTC against JSON tool calling on the BFCL v4 benchmark, utilizing a 309-entry subset across eight task categories. Fourteen language models, released between November 2024 and July 2026, were evaluated. The research focused on three key areas where JSON tool calling has shown limitations: sequential chaining, parallel fan-out, and context rot (adversarial context load).

Key Findings:

• Overall Performance: Programmatic tool calling matched or exceeded native JSON tool calling accuracy in 11 out of 14 evaluated models on the BFCL v4 benchmark. Performance differences largely aligned with model generation, with newer models generally showing greater benefits from PTC.
• Sequential Chaining: PTC's accuracy advantage increased with chain length, reaching an 18.8% absolute gap over JSON tool calling for chains of 12 or more. This is attributed to the extra inference turn required by JSON tool calling for each step in a chain.
• Parallel Fan-out: JSON tool calling exhibited a hard structural limit, with models like Claude Sonnet 5 dropping tool calls above a fan-out threshold of 70-72. In contrast, PTC maintained 100% enumeration accuracy even at a fan-out of 100.
• Context Rot Robustness: Under adversarial context loading, PTC remained stable, while JSON tool calling degraded by an average of 2.3%. A filesystem-discovery approach used as a reference point degraded by 32% in the same conditions.
• Specific Model Improvements: The GPT-5.6 family, for instance, achieved a 10.6% improvement over the JSON tool calling baseline. All five Anthropic models and the three newest GPT generations matched or exceeded baseline accuracy with PTC.

Why it matters for builders

This research offers a compelling case for adopting programmatic tool calling, moving beyond the constraints of rigid JSON formats. For AI builders, this means potentially more capable and reliable agents that can handle complex, multi-step tasks and resist adversarial inputs more effectively. The ability to execute code directly offers a more natural and powerful way for LLMs to interact with external tools and services.

Practical impact

AI builders should consider integrating programmatic tool calling into their agent architectures. This approach may lead to improved performance, especially in scenarios involving complex function chaining or parallel execution of multiple tools. The demonstrated robustness under context rot suggests PTC could be a more reliable choice for agents operating in dynamic or potentially manipulated environments. Developers can explore frameworks that support code execution as a native tool-use interface, such as those emerging in platforms like Cloudflare Agents or through libraries that treat code composition as the default interface.

Caveats and source limits

The study was conducted on the BFCL v4 benchmark, and results are specific to the 14 models evaluated, which span releases from November 2024 to July 2026. While the findings suggest PTC is a viable alternative, direct comparisons on other benchmarks or with models released after July 2026 are not provided. The research focuses on accuracy and robustness, with less emphasis on computational cost or latency differences between the two paradigms. The source does not provide specific pricing information or details on the implementation of PTC within specific commercial LLM offerings beyond general trends.