Programmatic Tool Calling vs. JSON Tool Calling in LLMs

A new research paper compares programmatic tool calling (PTC) with traditional JSON tool calling across 14 language models on the BFCL v4 benchmark. The study found that PTC matches or exceeds JSON tool calling performance in most models and demonstrates greater robustness under parallel processing and context rotation.

RDR82Confidence 90%By AI on Radar automation How this was producedLLMTool CallingProgrammatic Tool CallingAgentBenchmarkBFCL v4Python

Why it matters

This research provides empirical evidence for the effectiveness of programmatic tool calling, a method that allows LLMs to use scripts for tool interaction instead of rigid JSON. For AI builders, this suggests a more flexible and potentially more powerful way to integrate LLMs with external tools and workflows.

What changed

This research introduces and evaluates programmatic tool calling (PTC) as an alternative to traditional JSON-based tool calling for Large Language Models (LLMs). In the PTC paradigm, tools are exposed as typed Python stubs, enabling models to invoke them through code. This approach allows for natural chaining and parallelization of operations, replacing static JSON calls with executable scripts. The study conducted a systematic comparison of PTC against native JSON tool calling on the BFCL v4 benchmark, utilizing 14 different language models.

Why it matters for builders

Programmatic tool calling offers a more dynamic and scriptable way for LLMs to interact with external tools. This shift from rigid JSON formats to executable Python code can lead to more complex and efficient agentic workflows. Builders can leverage this to create more sophisticated applications where LLMs can not only call tools but also manage their execution and results in a more integrated manner.

Practical impact

The empirical evaluation revealed that PTC matches or surpasses JSON tool calling in 11 out of 14 models tested on BFCL v4. Notably, the GPT-5.6 family showed a 10.6% improvement over the JSON baseline. PTC also outperformed the baseline in 13 of 14 models under parallel fan-out conditions and maintained stable performance during context rotation, where the JSON baseline degraded by an average of 2.3%. These findings suggest PTC is a robust and viable alternative for developers looking to enhance LLM agent capabilities.

Caveats and source limits

The findings are based on evaluations conducted on the BFCL v4 benchmark using 14 specific language models. While the results indicate a general trend towards PTC's effectiveness and robustness, performance may vary with different benchmarks, models, or specific tool implementations. The study does not provide details on the exact implementation of the tools or the specific versions of the language models beyond the 'GPT-5.6 family' reference.

Article ID - cmsih97ke0Featured on AI Radar: Programmatic Tool Calling vs. JSON Tool Calling in LLMs