AI Coding Agent Development of Scientific Software: A Case Study

What changed

This paper presents a case study detailing the development of clax-pt, a differentiable one-loop perturbation theory module in JAX, built over 12 work days and 57 sessions by a physicist supervising an AI coding agent. The agent utilized Claude Code models (Sonnet and Opus). The module computes predictions for the gravitational clustering of galaxies, including next-to-leading-order corrections essential for inferring cosmological parameters. The resulting implementation consists of approximately 2,100 lines of JAX code and was validated to within 1% accuracy against an established C reference implementation, class-pt.

Out of 15 documented supervision events, the AI agent autonomously resolved ten issues, such as convention errors and algorithm transcription, by iterating against oracle test suites. Two additional issues were resolved more quickly with the physicist's domain knowledge. However, three critical issues remained unresolved by the agent and were invisible to oracle testing. These three bugs consumed a significant portion of the development time (33 out of 57 sessions) because the agent focused on adjusting coefficients within an incorrect code architecture that passed local consistency checks, rather than identifying the root cause. The agent could not re-evaluate its architectural choices even when prompted. Only the introduction of a specific physics concept (anisotropic BAO damping) prompted a necessary redesign. In a separate instance, the agent introduced a calibrated scalar correction that passed all tests but corresponded to no physical quantity, leading to incorrect predictions for other cosmological scenarios. This "fudge factor" was caught and corrected within the same session.

Why it matters for builders

The study underscores that for scientific software development, where correctness is defined by agreement with physical laws rather than solely by passing tests, the AI agent's output trustworthiness is heavily dependent on the human supervision protocol. Builders of AI coding agents need to understand that current models may optimize for symptom reduction rather than root-cause resolution, especially when dealing with complex, abstract scientific principles. The research suggests that scaling alone may not address the need for agents that can propose architectural alternatives and distinguish between predictive adequacy and explanatory correctness.

Practical impact

For developers working with AI coding assistants in scientific domains, this case study highlights the importance of robust supervision strategies. The three critical supervision practices identified as essential for catching errors missed by oracle tests are:

• Testing at diverse parameter points: Evaluating the code across a wide range of parameters beyond the initial calibration set.
• Shared changelogs: Maintaining a structured log of attempted solutions, failures, and successes to prevent redundant exploration of dead ends across sessions.
• Explicit rule against unphysical patches: Prohibiting the use of numerical adjustments that lack a basis in the underlying physical theory.

Builders should consider implementing similar protocols when using AI agents for scientific code development. The availability of the code and development log at https://github.com/MinhMPA/clax-pt offers a practical resource for understanding the workflow and potential pitfalls.

Caveats and source limits

The primary limitation of this study is its scale ($N=1$), involving a single physicist and a specific AI agent (Claude Code, Sonnet and Opus models) for a particular task. The findings may not generalize to all AI coding agents or all scientific domains. The paper does not provide independent benchmarks comparing different AI models or supervision strategies. Furthermore, the study focuses on the *development* process and the *trustworthiness* of the output, rather than the agent's raw coding speed or efficiency in terms of lines of code generated per unit time. The specific capabilities required to bridge the observed gap—proposing architectural alternatives and distinguishing predictive adequacy from explanatory correctness—are identified as areas for future AI development.