Measuring the Reliability of Software Engineering Agents Across Repeated Runs: An Issue-Level Study with Independent Patch Verification

Rakesh Singh

Measuring the Reliability of Software Engineering Agents Across Repeated Runs: An Issue-Level Study with Independent Patch Verification

Author(s)	Mr. Rakesh Singh
Country	India
Abstract	Abstract: A software engineering agent may resolve an issue on one attempt and produce an incomplete repair when asked to solve it again. Repeated runs reveal this variation, but the apparent reliability still depends on how patches are judged. This article presents a paired evaluation that applies benchmark and independent acceptance criteria to the same frozen submissions. It tracks each issue across three observed states: never solved, sometimes solved and always solved. A constructed eight-issue example demonstrates the analysis. With five attempts per issue, benchmark acceptance is 23/40 (57.5%), while verified acceptance is 15/40 (37.5%). Three issues change state after verification, including one that moves from always solved to never solved. These values illustrate the analysis and describe no observed agent performance. The article specifies task qualification, clean-run controls, independent verification, paired uncertainty estimates and resource reporting for a subsequent repository study. Keywords:Software engineering agents, repeated runs, issue resolution, patch verification, SWE-bench, reliability
Keywords	Abstract: A software engineering agent may resolve an issue on one attempt and produce an incomplete repair when asked to solve it again. Repeated runs reveal this variation, but the apparent reliability still depends on how patches are judged. This article presents a paired evaluation that applies benchmark and independent acceptance criteria to the same frozen submissions. It tracks each issue across three observed states: never solved, sometimes solved and always solved. A constructed eight-issue example demonstrates the analysis. With five attempts per issue, benchmark acceptance is 23/40 (57.5%), while verified acceptance is 15/40 (37.5%). Three issues change state after verification, including one that moves from always solved to never solved. These values illustrate the analysis and describe no observed agent performance. The article specifies task qualification, clean-run controls, independent verification, paired uncertainty estimates and resource reporting for a subsequent repository study. Keywords:Software engineering agents, repeated runs, issue resolution, patch verification, SWE-bench, reliability
Field	Computer Applications
Published In	Volume 8, Issue 6, November-December 2026
Published On	2026-11-10

View / Download PDF File

About IJFMR Fees & Payment Current Issue Publication Archive	Submit Research Paper Track Submission Status Publication Guidelines Publication Ethics Peer Review & Plagiarism	Join as a Reviewer Editors & Reviewers Reviewer Referral Program Get Reviewer Membership Certi.	Website/Journal Policies Usage Policy Content Policies Privacy Policy

Contact Us		+91-9687-828-838	editor@ijfmr.com

International Journal For Multidisciplinary Research

A Widely Indexed Open Access Peer Reviewed Multidisciplinary Bi-monthly Scholarly International Journal

Measuring the Reliability of Software Engineering Agents Across Repeated Runs: An Issue-Level Study with Independent Patch Verification

Share this