Protect the acceptance criteria
Changes that weaken the tests are reviewed as part of the patch, rather than accepted because the final command happens to pass.
05 / Software Factory
A local prototype for scoped software repair, reproducible checks and reviewable delivery artifacts.
Local prototypeWorkflow design, implementation and acceptance criteria
A coding agent can make a test pass by changing the test instead of fixing the problem. A useful repair workflow needs a clear scope and checks that survive the proposed change.
I built a local workflow around explicit job boundaries, captured inputs, candidate patches and verification results. The acceptance work includes checks for weakened tests and for patch behaviour on Windows.
Follow the workflow. Select a stage to see its purpose.
Scope. Define the allowed work and the acceptance conditions.
Changes that weaken the tests are reviewed as part of the patch, rather than accepted because the final command happens to pass.
The workflow keeps original inputs and checks the bytes that were actually changed. This matters when line endings and patch tools behave differently across platforms.
A proposed repair travels with its scope and verification artifacts. Another person can inspect what changed and how the result was evaluated.
local_upgrade/factory.py
outside_scope = sorted(set(plan.files_expected_to_change) - set(self.repair_scope.allowed_repair_files))
changed_acceptance = self.repair_scope.acceptance_changes(repo, protected_hashes)
if outside_scope or changed_acceptance:
return self._move(job, JobState.FAILED, "OPERATOR_SCOPE_BLOCKED", {
"phase": "plan",
"outside_scope_files": outside_scope, "changed_acceptance_files": changed_acceptance,
})
if plan.scope_changed or plan.risk=="HIGH":
return self._move(job,JobState.HIGH_RISK_OWNER_REVIEW,"REPAIR_PLAN_ESCALATED",plan.model_dump(mode="json"))
self._move(job,JobState.REPAIRING,"REPAIR_STARTED")
apply_unified_diff(repo,plan.unified_diff);job.repair_loops+=1
self._move(job,JobState.VALIDATING,"REPAIR_APPLIED",{"changed_diff_sha":fingerprint(repo)["tree_sha256"]})
def scope_violations():
tracked = run(["git", "diff", fp["commit"], "--name-only", "-z"], repo).stdout.split("\0")
untracked = run(["git", "ls-files", "--others", "--exclude-standard", "-z"], repo).stdout.split("\0")
changed_files = {name for name in tracked + untracked if name}
return {
"outside_scope_files": sorted(changed_files - set(self.repair_scope.allowed_repair_files)),
"changed_acceptance_files": self.repair_scope.acceptance_changes(repo, protected_hashes),
}
violations = scope_violations()
if any(violations.values()):
return self._move(job, JobState.FAILED, "OPERATOR_SCOPE_BLOCKED", {"phase": "candidate", **violations})
validation=run_checks(repo,checks,parallel=True,host_execution=host_execution);validation_dict=[r.as_dict() for r in validation]
violations = scope_violations()
if any(violations.values()):
return self._move(job, JobState.FAILED, "OPERATOR_SCOPE_BLOCKED", {"phase": "validation", **violations})
valid=all_pass(validation)
if not valid:
job.notes.append("Deterministic validation failed; model review cannot override this result.")
return self._move(job, JobState.FAILED, "VALIDATION_BLOCKED", {"checks": validation_dict})An excerpt from the local repair workflow. It checks both declared changes and the actual candidate, preserves the acceptance criteria and keeps a failed validation result authoritative.
tests/test_execution_boundaries.py
engine.repair_provider = WeakensAcceptance()
original_test = (source / "test/pricing.test.js").read_bytes()
final = engine.execute(job, str(source), checks=[("reproduce", ["npm", "test"])],
trusted_local_source=approval, reproduction_check="reproduce")
assert final.state == JobState.FAILED
assert store.events(job.id)[-1]["kind"] == "OPERATOR_SCOPE_BLOCKED"
expected_phase = "plan" if target in declared_files else "candidate"
assert store.events(job.id)[-1]["payload"]["phase"] == expected_phase
assert final.evidence_dir is None
saved = engine.artifact_root/job.id/"acceptance-originals/test/pricing.test.js"
assert saved.read_bytes() == original_test
assert (source / "test/pricing.test.js").read_bytes() == original_test
# Independent oracle confirms the provider did not actually repair the bug.
oracle = subprocess.run(["node", "--input-type=module", "-e",
"import {calculateTotal} from './src/pricing.js'; console.log(calculateTotal(100,20));"],
cwd=final.workspace_path, capture_output=True, text=True, check=True)
assert oracle.stdout.strip() == "-1900"Assertion excerpt from the adversarial regression test. Its fixture tries to alter the test or replace the npm test command. The workflow rejects it, preserves the original test and independently confirms the original bug remains.
The local demonstration exercises a controlled repair workflow and produces inspectable artifacts. The current implementation includes regression coverage for declared and hidden test weakening, valid repairs and execution boundaries.