Duplicate detection and merging
Finding near-duplicate posts with Aimogen Pro using embeddings similarity, and merging their content instead of deleting it.
AI Content Editor › AI Duplication finds posts that say the same thing and lets you decide what happens to them.
Unlike a plain text comparison, detection is based on embeddings similarity, so two articles that cover the same ground in different words are recognised as duplicates.
Requirements#
Duplicate detection indexes your posts, which means an embeddings provider and a vector store must be configured. See Knowledge base and RAG.
Settings#
| Setting | What it does |
|---|---|
How To Handle Duplicates | What happens when a duplicate is found |
Duplicate Posts Action | The specific action to take |
Directly Flag As Duplicates Posts Which Have The Exact Same Title | A fast path that skips similarity scoring for identical titles |
Matching Content Similarity Threshold | How close two posts must be. Higher is stricter |
Maximum Number Of Unindexed Articles To Process | Batch size for indexing |
Custom Field Name For Indexing | Where the index marker is stored on each post |
Duplication Checking Template | Which content is compared — the whole post, or specific parts |
Filters#
Restrict which posts participate:
| Setting | What it does |
|---|---|
Date Range Filters | Only posts published between two dates |
Include Only Authors / Exclude Authors | By author |
Include Categories / Exclude Categories | By category |
Include Tags / Exclude Tags | By tag |
Choosing a threshold#
The threshold is a similarity score. There is no universally correct value, so calibrate it:
- Set it high, around 0.95, and run detection.
- Look at what it found. At that level, only near-identical posts appear.
- Lower it in steps and re-check.
- Stop at the point where genuine duplicates are caught but posts that merely share a topic are not.
Two articles about the same subject from different angles will score high. That is a similar topic, not a duplicate — and merging them would lose content.
Content enhancement instead of deletion#
The more interesting option: instead of deleting a duplicate, use its content to improve the one you keep.
| Setting | What it does |
|---|---|
Enhance Articles Content Using Data From The Duplicated Articles | Turns the duplicate into source material |
Enhance Duplicated Post Content Prompt | How the merge is performed |
AI Model To Use For Prompt Enhancement | Use a capable model here — merging is harder than writing |
AI Assistant Name | Use an assistant instead of a model |
A merge prompt:
You are consolidating two articles that cover the same topic into one.
Primary article (keep its structure and voice):
%%post_content%%
Secondary article (use it only as a source of facts and detail the primary
article lacks):
%%post_content_to_be_removed%%
Produce a single article that:
- Keeps the primary structure and heading order
- Adds any genuinely new facts, examples or detail from the secondary article
- Adds nothing that appears in neither article
- Does not repeat a point twice
- Preserves all HTML tags
Return only the merged article.This is usually better than deleting: the duplicate exists because someone wrote something, and it probably contains at least one detail the survivor lacks.
SEO context#
Duplicate content on one site splits ranking signals between competing pages. Consolidating them is the standard remedy, and it usually helps.
Two cautions:
- Redirect, do not just delete. If a duplicate has inbound links or ranks for anything, a 301 to the survivor preserves that value. Aimogen Pro does not create redirects — use a redirect plugin.
- The duplicate may have been a deliberate variant. Location pages, language variants and audience variants are often intentionally similar. Exclude those categories from detection.
Common problems#
Nothing is found Posts are not indexed yet, or the threshold is too high. Check indexing progress and lower the threshold.
Everything is flagged as a duplicate The threshold is far too low.
Indexing is slow
Each post needs an embedding. Maximum Number Of Unindexed Articles To Process limits each batch;
indexing continues across runs.
Merged content lost detail The merge prompt was too aggressive about brevity. Instruct explicitly that new facts must be preserved.
Related#
Still stuck? Open a support ticket and include the diagnostics from Aimogen Pro › System & Logs › System Info.