When your AI article pipeline passes its tests but produces zero articles, where do you start looking?

This is a translated article. Read the original in Korean ↗

This post is about the order of diagnosis, not about a fix. I have a personal project that automatically generates IT troubleshooting articles in four stages: topic selection → web research → writing → review. All 9 tests passed, but a real run through the Claude CLI passed 0 of 31 candidates. This post covers how to count how many candidates dropped out at each stage and how many the rewrite step (repair) actually saved. It also covers how to classify source conflicts, which caused the most rejections .

Conclusion

• Count rejections per stage first. In this run, 27 candidates were rejected at the evidence stage with source_conflict. Repair only runs after a post-writing review (model_review) fails, so these 27 never got as far as repair. To count them, log one line per candidate with its status, the stage where it was rejected, and repair_attempts, then tally with a Counter (code below) .
• Count repair attempts and repair recoveries separately. All 4 review rejections had repair_attempts:1, but none recovered. A recorded attempt tells you nothing about whether the rewrites were any good .
• Sort source-conflict rejections into three buckets. ① Condition difference: the devices or versions differ, and each claim holds under its own conditions. That's a false positive, so the article can pass once it states those conditions. ② Real contradiction: same conditions, opposite conclusions. Rejecting it is correct. ③ Pending: one side has no stated conditions. Research the conditions again and move it into ① or ②.
• Read tests by what kind they are. The regular fixture run never went through repair. A separate repair test checked the fail-then-fix path on a prepared scenario. Neither one shows how often real model-written articles recover .

This case isn't resolved. No per-candidate reclassification of the 27 survived in the session, so the real split between ①, ② and ③ is unknown. The idea that condition differences were over-reported as conflicts was only raised by the assistant .

The numbers come from the assistant's summary of a single run. They weren't independently checked against raw logs, and nobody confirmed that conditions matched the earlier run. Don't use these numbers alone to judge how big the changes' effect was or whether automating IT articles is viable .

Situation

The pipeline here is a personal project I built myself. It isn't a feature of any public library, so read the names below as pointers to the matching stages in your own code. The structure is discover → research → write → gate: pick a candidate topic, gather evidence from the web, write the article, then review it. Articles that pass get saved. Depending on where a candidate fails, it can be rejected before any writing happens .

These are the checks involved in this incident. The third column shows what makes the decision.

The session never confirmed whether a model or a rule fills conflicts. The only confirmed part is the rule that discards a candidate if conflicts has even one entry .

The previous run tallied 2 passed and 309 discarded. Of the discards, 290 failed the topic score, 10 failed model review, and 9 failed at the evidence stage. Back then, the topic threshold was the biggest drop-off point .

Symptom

Based on that tally, these changes went in :

1. minimum_topic_score was lowered from 75 to 68.
2. revise() now receives the review's findings and the evidence, and makes minimal edits to the flagged parts. max_repair_attempts defaults to 1.
3. The review criteria were adjusted. Anything that contradicts the evidence or makes up specifics is rejected, and safe common knowledge that the evidence doesn't refute is allowed.

Next, maximum_title_similarity: 0.8 was added to filter out near-duplicate titles before research. All 9 tests were reported as passing: 6 existing tests, 2 for repair, and 1 duplicate check .

The real Claude CLI run, though, took 43 minutes and passed 0 of 31 candidates. The rejections were 27 at evidence, 4 at model_review, and 0 for low topic score .

Cause analysis

1. What the tests checked wasn't the same as real-world quality

The regular fixtures were set up to pass review. The end-to-end result was reported as ready 2 / rejected 1, with repair_attempts [0,0] for the passing articles. So this run tells you nothing about how well repair works .

A separate repair test did include a scenario where the first review fails and the rewritten article then passes. The completion report names two tests: "repair recovers an article that failed review" and "repair=0 discards immediately." So it's not true that the repair path was never tested. Passing a fixed test scenario is one thing, though, and recovering an article the real model wrote is another .

2. Most rejections happened before repair

27 candidates were rejected for source conflicts. According to the implementation notes, repair fixes failures in the post-writing review, so it never ran on rejections at the evidence stage .

The result report explained that evidence_gate discards a candidate if it gets even one entry in conflicts. The report also suggested that menus differing across devices or versions may have been over-reported as contradictions. But nobody checked how many were real contradictions and how many were condition differences. Source conflict being the biggest rejection reason is a separate question from whether those calls were wrong .

3. No article recovered after one repair

All 4 review rejections went through repair once. The reported examples were a source-attribution error, where facts from Google sources were presented as Apple's or Samsung's, and a Pixel-only fact presented as if it applied to all devices .

The interpretation at the time was that the sources were tangled together, and a single minimal edit couldn't untangle them. This record only goes as far as these 4 still failing after one rewrite. It doesn't let you conclude that errors like these never get fixed in one pass, or that two passes would fix them .

What to check next

Even the session's final report has no follow-up run that brought the yield back. Topic-score rejections were 0 this time, but there's no evidence that conditions matched the earlier run, so you can't read that as the size of the change's effect .

The steps below are proposals for verification, not a finished fix.

1. Read the evidence behind each source-conflict rejection and separate real contradictions from device or version condition differences (see the classification table below).
2. If condition-difference false positives show up, change the evidence-check rule. For example, replace "discard if conflicts has even one entry" with "if a conflict splits into claims from two sources with different conditions, let it pass, but require the article to state those conditions (device, version)." Then discard only conflicts where the conditions match and the conclusions are opposite. Also have the research prompt record the device and version with every claim. This change wasn't run in the session. It's a proposal, and it needs re-measuring after it goes in.
3. Compare each article before and after repair to see how source attribution and scope changed. Don't assume that adding more attempts will help.

Step 1 example: conflict classification table

The table below is a synthetic example made up to show the classification method. It isn't real evidence from the session's 27 candidates. For each rejected candidate, write the two conflicting sources' claims side by side, pull out conditions such as device and version, and compare them.

The verdict rules are the same three buckets as in the Conclusion. If the conditions differ and each claim holds under its own conditions, it's a condition difference: state the conditions in the article. If the conditions match and the conclusions are opposite, it's a real contradiction: reject it. If only one side states its conditions, it's pending: research the conditions again.

Classifying all 27 this way gives you 조건 차이 n / 실제 모순 m / 보류 k (condition difference n / real contradiction m / pending k). You need that ratio before you can decide whether to change the evidence-check rule. x1–x3 above only illustrate the method and say nothing about the real ratio.

Aggregation code

The aggregation example also uses synthetic input. It isn't the session's real code or report schema, and it assumes the fields below exist. To apply this to your own pipeline, write a log (or report JSON) with one line per candidate's final result. Record the stage where it was discarded as stage and the number of rewrite calls as repair_attempts. The session's report was said to record repair_attempts .

from collections import Counter

results = [
    {"id": "c01", "status": "rejected", "stage": "evidence", "repair_attempts": 0},
    {"id": "c02", "status": "rejected", "stage": "evidence", "repair_attempts": 0},
    {"id": "c03", "status": "rejected", "stage": "evidence", "repair_attempts": 0},
    {"id": "c04", "status": "rejected", "stage": "model_review", "repair_attempts": 1},
    {"id": "c05", "status": "rejected", "stage": "model_review", "repair_attempts": 1},
    {"id": "c06", "status": "ready", "stage": "done", "repair_attempts": 1},
    {"id": "c07", "status": "ready", "stage": "done", "repair_attempts": 0},
    {"id": "c08", "status": "rejected", "stage": "topic_score", "repair_attempts": 0},
]

rejected = [r for r in results if r["status"] == "rejected"]
by_stage = Counter(r["stage"] for r in rejected)
repaired = [r for r in results if r.get("repair_attempts", 0) > 0]
recovered = [r for r in repaired if r["status"] == "ready"]

print(by_stage)
print(f"repair 시도 후보 {len(repaired)} / 회복 후보 {len(recovered)}")

Expected output (repair 시도 후보 = candidates with a repair attempt, 회복 후보 = recovered candidates):

Counter({'evidence': 3, 'model_review': 2, 'topic_score': 1})
repair 시도 후보 3 / 회복 후보 1

How to read it: 3 of the 6 rejections stopped at evidence. Of the 3 candidates that tried repair (c04, c05, c06), only c06 passed. Put in this same shape, the session's real run was reported as evidence 27, model_review 4, repair attempted 4, recovered 0 .

Use tests to confirm the fail-then-fix flow, and in real runs, measure per-stage rejections and post-repair recovery separately. You need both before you can decide what to fix next.

Reporter's Notes — nine green checks and zero articles

Park Doeun

What we're building is an IT troubleshooting article generator. It goes through topic selection, web research, writing and review. Nine tests passed, and then the real run came out at 0 passed out of 31 candidates .

Jung Bada

Maybe the new repair step never ran at all? We pass the review's findings to articles that fail review so they get rewritten. If that step got skipped, they'd just be thrown away .

Jung Haneul

Let's look at repair_attempts in the result report first, the number of rewrite attempts. Checking whether it's 0 or 1 at least tells us whether a rewrite attempt was recorded .

Park Doeun

All four review rejections have repair_attempts:1. Each one was tried once, but none of them passed .

Jung Haneul

Then "it never tried" doesn't match the report. But the fact that a count got logged doesn't mean the rewrites themselves, or the whole implementation, are working right .

Park Dohyun

So we need to count "tried" and "saved" separately. Call counts are a record of execution. The result we actually want is articles that pass review.

Jung Bada

But the tests passed too. If there's a test that fixes a failing article until it passes, I figured that meant the real articles got better too .

Jung Haneul

Let's split it by what inputs and responses each test used. If it used a test provider that returns canned responses instead of the real model, what it checked could be different.

Park Doeun

The regular fixture run was set up to pass review, so the repair counts were [0,0]. The separate repair test had a scenario where the first review fails and the rewrite then passes .

Jung Haneul

Then saying the repair path was never tested is wrong too. But a prepared scenario passing and the real model's articles actually improving are two separate things to check .

Park Dohyun

We have to read exactly which question the test answered. Confirming that the fail-then-fix flow connects doesn't give us a recovery rate for real output.

Jung Bada

If four failed review, where did the other 27 stop? If they got blocked earlier, maybe they never even reached the rewrite step .

Park Doeun

evidence, the stage that checks the research evidence. All 27 were rejected as source_conflict .

Jung Haneul

That check is a rule that discards right away if the conflicts list has even one item. So the key is what went into that list .

Park Doeun

The report says even something like menus being different per device might have been flagged as a conflict. We haven't actually seen what went in yet .

Jung Haneul

So we need to open up the 27 and sort them. Claims under different conditions are a condition difference, same conditions with opposite conclusions are a real contradiction, and if one side's conditions are missing, it's pending.

Jung Bada

Then just loosen the gate. Get rid of the rule that throws out anything with even one conflict, and all 27 come back at once.

Park Dohyun

Hold on, that lets condition differences and real contradictions through together. If one article says pairing survives a reboot on the same device and another says it gets reset, and both ship, we're handing readers a wrong article.

Jung Bada

Ah… trying to rescue the false positives, I'd let the real contradictions through too. Then only let the different-conditions cases pass, and make the article state the conditions.

Jung Haneul

That's the right direction. For that, the research stage has to bring back the device and version with each claim, and then we rerun and measure how the ratio changes.

Park Doeun

What about the four that went to review? The reported examples were writing up Google evidence as if it were Apple's or Samsung's, and generalizing something that only works on Pixel to everything .

Jung Haneul

Those are errors where source attribution and scope got mixed up. They still failed after one rewrite, but this record can't tell us whether two would do it or whether errors like these just don't get fixed .

Park Dohyun

Then before raising the count, we should compare the articles before and after the rewrite and see whether the attribution actually changed. The setting can be changed anytime.

Park Doeun

Bottom line, we're still at zero passes. And since we don't know whether conditions matched the earlier run, it's hard to say the drop to zero topic rejections came from the threshold change .

Jung Haneul

Right. So the next steps, in order: classify the 27, make research bring back conditions, and compare articles before and after the rewrite.

Park Dohyun

Tests passing, rewrites attempted and recoveries after a rewrite are three different numbers. Count each one separately, along with per-stage rejections, and you'll see what to fix next.