Dev Investigation Notes
When an AI content pipeline keeps searching the web for nearly identical topics (filtering duplicates against published articles and within the same batch)
This is a translated article. Read the original in Korean ↗
I was working on an automated content pipeline (content-loop-mvp) that picks topics, runs web research, writes articles, submits them for model review, and saves the ones that pass . What the implementation report calls “published articles” here are articles in storage used for comparison; automatic publication to an external blog was not confirmed . Recent runs had produced 2 accepted articles and 309 rejections, and the rejection list kept showing topics that rephrased the same subject . This article covers filtering candidates similar to published articles or topics that have already passed in the same batch, before web research. The record did not establish whether this prevents topics researched and rejected in a previous run from being researched again in the next run.
Conclusion
Put duplicate detection at the cheapest stage (topic selection). If you run web research, writing, and review before throwing an article away because “we already covered this,” all those calls go to waste.
The implementation report for this pipeline (discover → research → write → review) said a title similarity gate was added immediately after the topic score gate, before the research call . The reported setup was:
normalized_title()/title_similarity()— compare titles for similarity after removing whitespace and symbolsready_titles()— a list of front matter titles from already published articles- Initialize
known_titleswith published article titles, then keep adding the titles of candidates that pass the topic gate - Reject a candidate as
duplicate_topicbefore research if its similarity is at leastmaximum_title_similarity(configured as0.8) - The existing
exists()check rejects exact topic matches asalready_generatedfirst, so the new gate handles only near duplicates
The comparison set consists only of published articles and candidates that passed earlier in the same batch. There is no report of topics rejected in previous runs being included in that list . Unconfirmed: whether this prevents previously rejected topics from being researched again in the next run therefore remains unverified. To prevent that, merge the rejection history into known_titles and run the same candidate list twice in succession, checking that the second run makes zero research calls (this is only a proposal; it was not implemented or verified in this case).
The flow looks roughly like this. This example reconstructs the reported behavior; it is not the original code.
known_titles = storage.ready_titles() # Published article titles
for topic in sorted(candidates, key=score, reverse=True):
if storage.exists(topic): # Exact match
reject(topic, "already_generated"); continue
if topic.score < config.minimum_topic_score:
reject(topic, "topic_score"); continue
if any(title_similarity(topic.title, t) >= config.maximum_title_similarity
for t in known_titles): # Near duplicate → reject before research
reject(topic, "duplicate_topic"); continue
known_titles.append(topic.title) # Register before research
evidence = provider.research(topic) # Expensive calls start here
...
The similarity algorithm is not in the record. The session record contains only the summary “compare titles after removing whitespace and symbols” and a similarity result of 0.919; neither the algorithm nor the input titles could be confirmed . That means the original score cannot be reproduced. To apply this yourself, you can build a function along the same lines using only the standard library, as below. This is an example implementation proposed by this article, not the original.
import re
from difflib import SequenceMatcher
def normalized_title(t: str) -> str:
return re.sub(r"[\W_]+", "", t.casefold()) # Remove whitespace and symbols, lowercase
def title_similarity(a: str, b: str) -> float: # 0.0 (completely different) ~ 1.0 (identical)
return SequenceMatcher(None, normalized_title(a), normalized_title(b)).ratio()
Because this compares individual characters, changes in word order or substantially different wording produce lower scores. You need to run the function on your own title data to find out what scores it returns.
Verify not just “was the duplicate filtered out?” but “was the research call skipped?” The test test_near_duplicate_topic_rejected_before_research reportedly supplied two candidates with a similarity of 0.919 and confirmed ready 1 / rejected 1(duplicate_topic) with one research call . The actual test input titles are not in the record, so create your own test with the same structure. Below is test pseudocode. You will need to prepare Topic, provider, evidence, pipeline, and the result fields to match your project.
def test_near_duplicate_skips_research():
calls = []
provider.research = lambda topic: calls.append(topic.title) or evidence
candidates = [
Topic("PS5 듀얼센스 연결 안 될 때 단계별 해결법", score=90),
Topic("PS5 듀얼센스 연결 안 될 때 해결 방법 단계별", score=85), # Same topic, different wording
Topic("갤럭시버즈 한쪽 소리가 안 날 때", score=80), # Different topic (to check for false positives)
]
report = pipeline.run(candidates)
assert len(calls) == 2 # One variant rejected before research
assert report.rejected_reasons == ["duplicate_topic"]
assert "갤럭시버즈 한쪽 소리가 안 날 때" in calls # Different topic passes
First, print the two variants’ actual similarity score using the function above to check whether it exceeds maximum_title_similarity. If it does not, make the test titles more similar or adjust the threshold.
Situation
This pipeline runs in the order discover (generate topic candidates) → research (gather evidence through web searches) → write (write the article) → review (model review). The topic score gate sits immediately after discover and only compares scores . An exact-match check (exists()) was already in place, but even slightly different wording could get through .
Symptom
- Recent runs had produced 2 accepted articles and 309 rejections .
- More than 10 variants of
ps5-듀얼센스-연결appeared repeatedly in the rejection list. The assistant diagnosed it as “discover keeps generating nearly identical topics, running them through web search and writing, then throwing them away” . However, the same report broke the rejections down by stage: 290 at topic scoring, 10 at model review, and 9 at evidence validation . Topic score rejections are free rejections that happen before research, so the record does not show which stages actually rejected those 10-plus variants. The claim that they reached research and writing was the assistant’s diagnosis, not a verified count. - Discover output from other runs also repeated the same subject with different wording: “Step-by-step fixes when a PS5 DualSense controller won’t connect” and “How to reset and re-pair a PS5 DualSense when it won’t connect to the console” , for example.
Cause
Costs differ by stage. In the same diagnosis, the assistant broke down rejection costs by stage .
| Stage | Cost of rejection |
|---|---|
| Topic score rejection | Effectively free. Only compares scores and rejects the candidate, without web search, writing, or review |
| Model review rejection | Expensive. Uses discover + research (web search) + write + review, then rejects the article at the end |
If a duplicate topic gets through the cheap gate unchecked, it incurs all the expensive downstream costs (for the reasons above, the actual number of such cases here is unconfirmed). Duplicate detection therefore needs to move ahead of research to be effective. In the implementation report’s words, the design prevents wasted calls by “moving duplicate detection ahead of web research and writing, up to the topic stage” .
Fix
The reported gate targets two sources of wasted calls .
- Variants within the same batch — candidates are processed in score order, so only the highest-scoring variant proceeds to research; the rest are filtered out before research.
- Duplicates of published articles — when a topic similar to an already published article appears again, research and writing do not run.
The policy of registering titles before research. Following the implementation report’s “add when the topic passes the topic gate,” the reconstructed example above places known_titles.append before the research call . That means the title is already registered even if research or review fails, so other variants in the same batch continue to be filtered out. The report does not explain whether this behavior is intentional policy . It follows from the sequential processing in the example above. If processing continues after the first candidate fails, subsequent variants above the similarity threshold will not be retried in that batch. Moving registration to after successful research would still filter subsequent variants of successful candidates before research, while allowing variants of failed candidates to be retried. Whether the actual pipeline continues after a network exception was not confirmed, so this policy comparison must not be read as a verified result from the original implementation.
A minimal procedure for tuning the threshold. 0.8 is simply the value this project chose, not a validated value . To choose one for your own data, (1) manually label 10 “same topic” pairs and 10 “different topic” pairs from accumulated rejected and published titles, (2) print both groups’ scores using title_similarity above, then (3) choose a value above the highest score among different-topic pairs that still catches most same-topic pairs. If the two groups overlap, character-level comparison has reached its limits, so consider another approach, such as comparing sets of words. This case also included a check for false positives before implementation. The assistant first checked whether the fixture candidate titles would produce false positives against each other . It later reported that all 9 tests passed and the fixture end-to-end run also worked correctly .
What was not measured. A subsequent run with the actual provider (claude CLI) accepted 0 of 31 candidates. The reported breakdown was 27 source_conflict rejections, 4 model review failures, and 0 topic rejections . Since 27 + 4 = 31, that report table has no duplicate_topic row. This single run cannot establish how many calls the gate saved in practice. What this article confirms is limited to the gate’s placement and the testing approach used to verify that placement.
Bonus — the proposal and implementation used different comparison sets
The initial diagnosis proposed “checking similarity against previously rejected topics” . But in the completion report, known_titles starts with published article titles (ready_titles()) .
According to the report’s description, topics that reached research and were rejected in a previous run do not enter the next run’s comparison list. The original problem was repeated ps5-듀얼센스-연결 variants in the rejection list . This article’s approach therefore does not confirm that previously rejected topics were prevented from being researched again in the next run. Preventing that would require merging the rejection history store into the comparison list and a test that runs the same candidate list twice in succession, checking that the second run makes zero research calls. This is only a proposal; there is no record of it being implemented or verified in this case. The full final code was not available during reporting, so no definitive conclusion is drawn.
For the same reason, the record does not establish whether this catches variants with substantially different word order and phrasing. The test covered only one pair with a similarity of 0.919 .
Reporter's Notes — free rejections and expensive rejections
Park Doeun
Jung Bada
Jung HaneulPark Doeun
Park DohyunJung BadaJung HaneulPark Doeunps5-듀얼센스-연결 .Park DohyunJung BadaPark DohyunJung HaneulPark DoeunJung HaneulPark DoeunPark DohyunJung HaneulJung BadaJung HaneulPark DoeunPark DohyunJung Haneul