Can AI Fix Your Rust Code?

by

Author: Denis Avetisyan

Researchers introduce a new benchmark and agentic framework to dramatically improve automated bug fixing in Rust programs.

Large language model agents operate on a foundational framework encompassing perception, planning, and action, where the model receives input, formulates a plan leveraging its internal knowledge and reasoning capabilities, and then executes actions within an environment, iteratively refining its approach based on observed outcomes - a cycle inevitably complicated by the realities of production systems and emergent technical debt.
Large language model agents operate on a foundational framework encompassing perception, planning, and action, where the model receives input, formulates a plan leveraging its internal knowledge and reasoning capabilities, and then executes actions within an environment, iteratively refining its approach based on observed outcomes – a cycle inevitably complicated by the realities of production systems and emergent technical debt.

This paper details Rust-SWE-bench and RustForger, a system leveraging dynamic tracing and large language models for automated program repair.

Despite the increasing promise of large language models in software engineering, evaluating their efficacy in realistically complex, repository-level issue resolution has remained a significant challenge. This paper, ‘Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents’, addresses this gap by introducing Rust-SWE-bench, a benchmark comprising 500 real-world Rust issues, and demonstrating that a novel agentic framework, RUSTFORGER-integrating automated testing with dynamic tracing-significantly outperforms existing approaches, achieving a 34.9% improvement over the strongest baseline. By uniquely solving 46 previously intractable tasks, RUSTFORGER showcases the potential of dynamic analysis to overcome the limitations of LLMs in navigating Rust’s strict semantics; but can these techniques be generalized to further enhance automated repair across diverse programming languages and codebases?

Rust’s Debugging Dilemma: When Safety Bites Back

Rust’s dedication to memory safety and fearless concurrency, while providing significant benefits, inherently introduces debugging challenges distinct from those encountered in languages with garbage collection or less strict memory models. The compiler’s rigorous checks, designed to prevent common errors like data races and dangling pointers, shift the focus of debugging from runtime crashes to compile-time errors and subtle logical flaws. Furthermore, concurrent code, even when correctly compiled, can exhibit non-deterministic behavior, making it difficult to reproduce and isolate issues. These complexities demand developers adopt new strategies and tools, moving beyond traditional breakpoint-based debugging to leverage techniques like static analysis, fuzz testing, and sophisticated tracing mechanisms to effectively navigate the intricacies of modern Rust applications.

Conventional debugging techniques, while effective for simpler programs, frequently prove inadequate when confronted with the complexities inherent in modern software engineering projects. These methods often rely on manual inspection of code and step-by-step execution, becoming exponentially more difficult to apply to large-scale Rust applications characterized by intricate ownership models, lifetimes, and concurrent operations. The very features designed to enhance safety – Rust’s borrow checker and thread safety guarantees – can ironically introduce subtle bugs that are challenging to isolate using traditional tools. Consequently, developers face increasing difficulty in diagnosing and resolving performance bottlenecks, memory leaks, and data races, necessitating the development of more automated and intelligent debugging solutions capable of navigating these complexities.

As Rust adoption expands beyond early adopters and into larger, more complex projects, maintaining codebases presents escalating challenges. The language’s strengths – memory safety and fearless concurrency – ironically contribute to debugging difficulties, as traditional methods struggle with the nuanced errors that can arise from borrow checking and multi-threaded interactions. Consequently, a demand is growing for tools and techniques that move beyond manual inspection and breakpoint-driven analysis. Automated approaches, including static analysis, fuzzing, and intelligent code completion, are becoming increasingly vital to proactively identify potential issues, streamline refactoring, and ensure the long-term health of Rust applications. These advanced systems aim to alleviate the burden on developers, allowing them to focus on innovation rather than being consumed by the intricacies of code maintenance.

The Rust-SWE-bench benchmark suite covers a diverse range of task categories, as illustrated by its distribution.
The Rust-SWE-bench benchmark suite covers a diverse range of task categories, as illustrated by its distribution.

LLMs to the Rescue? The Rise of Agentic Debugging

Agentic approaches in software engineering leverage Large Language Models (LLMs) as autonomous entities capable of interacting with their environment to achieve specific goals. This involves a cyclical process of perception, analysis, and action; the LLM receives input regarding the software project-such as code repositories, test results, or bug reports-analyzes this information to identify issues or opportunities, and then executes actions like modifying code, creating tests, or submitting pull requests. These systems typically employ tools and APIs to interface with the development environment, enabling the LLM to not just suggest changes, but to implement and validate them automatically, effectively functioning as software engineering agents.

Agentic systems employing Large Language Models (LLMs) are designed to automate core software engineering tasks historically completed by human developers. This automation encompasses static and dynamic code analysis to identify potential bugs and vulnerabilities, automated test case generation and execution to verify functionality, and code repair capabilities, including bug fixing and refactoring. These LLM-driven agents leverage techniques such as program synthesis and semantic understanding to not only detect issues but also propose and implement solutions, potentially reducing development time and improving code quality. The scope of automated repair currently varies, ranging from simple fixes to more complex refactoring, and often requires iterative testing and validation to ensure the proposed changes do not introduce regressions.

Agentic frameworks like SWE-agent, OpenHands, and Agentless each employ distinct methodologies for LLM integration within the software development lifecycle. SWE-agent focuses on autonomous code generation and modification through a loop of planning, execution, and verification, utilizing tools like GitHub Copilot and unit testing. OpenHands utilizes a hierarchical tree of LLM-driven tools, enabling complex task decomposition and execution, with an emphasis on observability and control. Agentless distinguishes itself by operating without a central planner or explicit memory, relying on iterative prompting and direct interaction with the environment and tools, prioritizing simplicity and adaptability. These frameworks differ in their architectural choices, tool selection, and approaches to task management, representing varied strategies for leveraging LLMs to automate software engineering processes.

Across all large language models, agents consistently resolve more cumulative tasks as the number of edited lines per fixed patch increases.
Across all large language models, agents consistently resolve more cumulative tasks as the number of edited lines per fixed patch increases.

Rust-SWE-bench: A Realistic Testbed for Automated Debugging

Rust-SWE-bench is a newly developed benchmark consisting of 500 individual tasks representative of real-world Rust software engineering challenges. These tasks were extracted from 34 publicly available and actively maintained Rust repositories, ensuring a diverse range of coding styles, project structures, and problem domains. The selection criteria prioritized repositories with significant community involvement and a history of practical application, aiming to create a benchmark that accurately reflects the types of issues developers encounter in professional Rust projects. The benchmark encompasses a wide spectrum of task complexities, from simple bug fixes and code refactoring to the implementation of new features and the resolution of complex logical errors.

Rust-SWE-bench tasks are not simple code completion or unit test generation; they necessitate a comprehensive understanding of Rust semantics, including ownership, borrowing, lifetimes, and error handling. The benchmark includes tasks such as bug fixing, feature implementation, code refactoring, and performance optimization, all derived from genuine issues encountered in established Rust projects. These tasks often require multi-step reasoning, navigating complex codebases, and adapting to varied coding styles, thereby demanding problem-solving skills beyond basic syntactic manipulation. The complexity is further increased by the inclusion of tasks requiring external crate integration and configuration adjustments.

The Rust-SWE-bench benchmark was employed to quantitatively assess the capabilities of multiple agentic frameworks when confronted with practical Rust programming challenges. Evaluation focused on the frameworks’ success rates in resolving authentic issues extracted from real-world Rust projects, providing a comparative analysis of their performance across a diverse range of software engineering tasks. This involved submitting each framework to the 500 tasks comprising the benchmark and measuring the percentage of tasks successfully completed, allowing for a direct comparison of their ability to handle complex Rust code and problem-solving requirements.

Rust-SWE-bench is constructed through a process involving data collection, test case generation, and execution to evaluate software engineering workflows.
Rust-SWE-bench is constructed through a process involving data collection, test case generation, and execution to evaluate software engineering workflows.

RustForger: Dynamic Tracing for a Smarter Debugger

RustForger is an agentic framework designed to automate the debugging process in Rust projects. It integrates automated testing workspaces, allowing for isolated execution and modification of code, with cross-project dynamic tracing capabilities. This tracing functionality leverages the Abstract Syntax Tree (AST) to monitor program execution and data flow across multiple project dependencies. By combining these elements, RustForger aims to provide a comprehensive and automated approach to identifying and resolving software defects, moving beyond traditional static analysis and logging methods.

Dynamic tracing in RustForger leverages the Abstract Syntax Tree (AST) to provide detailed runtime analysis of Rust code. By examining the AST, the framework can track the execution path and data flow with precision, enabling identification of the precise code locations responsible for errors or unexpected behavior. This differs from traditional debugging methods by moving beyond simple stack traces to provide a contextual understanding of the program’s state during execution. The AST-based approach allows RustForger to correlate runtime values with the original source code, facilitating pinpoint accuracy in identifying root causes, even in complex scenarios involving multiple interacting components and dependencies.

The integration of Cargo, Rust’s package manager, is central to RustForger’s ability to reliably reproduce and resolve complex Rust issues. Cargo facilitates dependency management, ensuring a consistent build environment across different execution attempts and enabling the agent to accurately recreate the conditions necessary for problem reproduction. By leveraging Cargo’s features for building, testing, and managing Rust projects, RustForger avoids issues stemming from inconsistent dependencies or build configurations, which are common obstacles in debugging complex software. This consistent environment directly contributes to the agent’s improved performance in both reproducing reported errors and successfully applying fixes.

Evaluation on the Rust-SWE-bench benchmark indicates RustForger achieves a 28.6% resolution rate for identified issues. This represents a 34.9% performance increase compared to the strongest existing baseline system. Notably, RustForger successfully resolved 46 tasks that were not resolved by any of the other LLM-based systems tested. Furthermore, the framework demonstrated a 67.3% Reproduction Success Rate – the percentage of failed tests successfully reproduced – exceeding the 55.5% rate achieved by the OpenHands system.

The RustForger framework enables the creation of realistic forged components through a simulation-based approach.
The RustForger framework enables the creation of realistic forged components through a simulation-based approach.

The pursuit of automated issue resolution, as demonstrated by RustForger and benchmarked with Rust-SWE-bench, feels predictably ambitious. It’s a familiar cycle: a framework emerges, promising elegant solutions to the chaos of production code. The system strives for perfection, attempting to automatically mend errors before they escalate. As Brian Kernighan observed, “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.” This rings true; the very act of automating repair introduces a new layer of complexity, a new surface for failures to manifest. It’s not a matter of eliminating bugs, but shifting where they reside-from the application logic to the automated repair mechanism itself. The benchmark offers a snapshot of progress, but the true measure lies in how gracefully the system degrades when reality inevitably deviates from the ideal.

What’s Next?

The introduction of Rust-SWE-bench is, predictably, a move toward quantifying a problem production has been solving with duct tape and late nights for decades. Any benchmark that necessitates a ‘forger’ suggests the bar for ‘solved’ is set rather low. It will be instructive to observe how quickly the benchmark itself becomes the bottleneck, and how many edge cases the testing framework conveniently omits. The gains demonstrated by RustForger are encouraging, of course, until one considers the overhead of ‘dynamic tracing’ in a system already contending with the complexities of Rust’s borrow checker.

The pursuit of ‘agentic frameworks’ feels less like progress and more like a sophisticated re-implementation of existing debugging practices. One anticipates a future where these agents spend more time explaining why they failed than actually fixing code. A truly robust system would be one that gracefully accepts its own limitations, a concept conspicuously absent from most current research.

Ultimately, the field will likely converge on a frustrating truth: automated repair is rarely elegant, often introduces new problems, and generally delays the inevitable need for a human to actually read the code. Better one well-understood, monolithic codebase than a hundred microservices patched together by optimistic LLMs. The logs, as always, will have the final say.

Original article: https://arxiv.org/pdf/2602.22764.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-02-28 10:57