Removing a TODO rarely comes with a test for it. Ask your agent for one

A TODO or FIXME comment is a note from a developer admitting that the code is suboptimal. An agent can clear a long list of them in an afternoon. A study published today in the ESEM 2026 proceedings suggests that people already pay down this kind of debt without checking the fix. An agent can repeat that habit unless you ask for something different.

What the study found

Suzuka Yoshimoto and four co-authors studied self-admitted technical debt, meaning TODO and FIXME comments, in eight open-source Java projects. They measured test coverage for 784 debt instances in the latest releases and followed 5,175 debt removal events over the projects’ histories.

Their main results:

• 60.7% of the code flagged as debt is covered by the existing test suite.
• When a team removes the debt, the test that targets it is usually missing. Manual inspection found that 3.4% of removal commits add new tests aimed at the resolved debt.
• 12.5% of removal commits add some test in the same commit, whether or not it targets the flagged code.

The authors conclude that developers treat paying down this debt as an ordinary code change and not as high-risk maintenance. Their abstract puts it this way: “most debt removals proceed without targeted verification, despite the developer’s own prior flag that the code is suboptimal.”

The paper has a caveat worth knowing. Over short and medium windows, debt fixes with and without new tests were later linked to bugs in the same method at almost the same rate. The authors say those bugs were not necessarily caused by the fix. Over the longest window, the group that added tests reached a slightly higher cumulative defect alignment probability (6.32% against 4.37%). The authors suggest this may be because people choose to test the more complex parts of the code. So the study does not show that a targeted test prevents later bugs. Its finding is narrower: the person who wrote the TODO flagged the code as suspect, and the fix almost never comes with a check of what they flagged.

Why this matters more with an agent

The study looks at the commit histories of open-source Java projects. It does not study agents. Still, an agent asked to “resolve the TODOs in this module” receives the same instruction a developer would, and the study shows what usually follows in human-written history: a code change with no test aimed at the debt.

There is some encouraging evidence on agents and testing. A second ESEM 2026 paper, which followed 669 developers across 103 repositories, found small improvements on all three of its testing measures after developers adopted agents. The same study found that reviews got faster, less detailed and more permissive, but that happened in repositories without agents too. Testing improved a little, and review standards slipped across the board. Neither paper measured whether an agent writes a test aimed at a TODO when it removes one, so that check is still yours to set.

A rule to give your agent: fail before, pass after

When you hand an agent a debt cleanup, add one rule to the task:

For each TODO or FIXME you remove, write or name a test that fails on the code before your change and passes after it. If you cannot write that test, leave the comment and tell me why.

That rule gives you three useful results:

1. A real fix comes with proof. The failing run on the old code shows the test is checking what the comment flagged. A test that passes on both versions is not about the debt.
2. A comment the agent cannot test is worth seeing. It may describe a performance concern, a design preference, or a note that no longer applies. You want that sorted before the code changes.
3. The 60.7% stops being the safety net. Coverage means some test runs those lines. It does not mean any test checks what the comment worried about.

Give the agent one exception to keep the rule practical: a TODO that is only a note, such as “rename this,” can be removed without a test, but the agent should list it separately in its summary so you can see what it skipped.

Measure your own baseline first

You can check how your repository handles this today. In git, the -S option finds commits that changed how many times a string appears:

git log -S "TODO" --since="6 months ago" --oneline -- src/
git log -S "FIXME" --since="6 months ago" --oneline -- src/

Some of these commits will have added comments rather than removed them, so open each one. Pick ten that removed a comment and, for each, ask one question: did this commit add or change a test that checks the behavior the comment described? Write down how many did. That number is your starting point, and you can compare it with the study’s 3.4% and 12.5%. Test folders often sit outside src/, so look at the whole commit, not only the filtered path.

What to do this week

• Run the two git log -S commands and score ten debt removals for a targeted test.
• Add the fail-before, pass-after rule to the instruction file your coding agents read.
• On your next agent cleanup, require a separate list of every TODO removed without a test, with a reason for each.
• In review, run the agent’s new test against the parent commit once and confirm it fails.

If you want help with one decision about how your agents should handle this debt, a Consult block is one way to start: you bring the decision, coaches run the method on it with you, and you leave with the map and a first move.