Atomic Claim Checking Makes LLM Knowledge Base Merges Easier to Audit

Merging duplicate support articles with a large language model can save writing time, but a polished result may quietly lose the detail that makes an article useful. Elastic’s approach tackles that risk by checking each source claim against the proposed merge, turning a broad question about fidelity into a reviewable list of exceptions.

The problem was practical: support teams had known about duplicate knowledge base articles for years, yet had completed very few merges. Doing one properly meant reconciling technical details across articles, retaining useful information and rewriting the result so it read as a single piece. The work could take an hour per pair. At that rate, a backlog was easy to understand—and hard to clear.

That backlog mattered because the knowledge base held diagnostic experience accumulated by hundreds of engineers. Two articles about the same issue might each contain information the other lacked: one a clearer explanation of the root cause, the other a command that fixed it. A customer finding only one could end up with an incomplete answer.

Using a model to combine the articles looked like an obvious shortcut. The difficulty was that fluent prose offered no reliable indication of whether the technical content had survived. The team’s response was to treat generation and verification as separate jobs, then give an engineer the final say.

Why a good-looking merge can still be wrong

A basic merge can be built quickly: give the model two articles and ask for one consolidated version. The result may have a consistent voice, clean structure and little repetition. But those qualities don’t show whether the model preserved every prerequisite, caveat or version boundary.

In the team’s experience, comparing a draft with its sources could reveal that a prerequisite step had disappeared, a version constraint had changed from 8.12+ to 8.x, or a configuration caveat had been dropped. One merge even introduced a parameter that did not exist in the source material.

Those errors are especially consequential in a knowledge base. A reader can challenge a bad answer in a chat. A faulty merge, by contrast, may become the canonical article while the originals are retired. Once that happens, an omission or invented detail can be difficult to spot because there is no longer an obvious source to compare with.

Fluency makes the review harder. A draft that retains most of its source details can read much like one that retains them all. A person skimming a polished article may miss the small, awkward detail that a customer needs. The team concluded that the workflow had to make omissions visible instead of relying on reviewers to notice them unaided.

Two passes, with different responsibilities

The workflow uses one model call to merge the articles, followed by a fresh audit against each source separately. The merge pass aims to produce a coherent article. The audit pass checks whether the source information can be found in that result. The merger’s own account of what it included can help guide review, but it is not treated as proof.

1. Merge: The model consolidates the two articles under explicit constraints. Code blocks, commands, configuration snippets, URLs and version numbers must carry over verbatim. Prose can be rewritten; technical payloads cannot.

2. Audit each source: In a separate context, the model extracts the source’s atomic claims and checks them against the merged article. It repeats this process once for each source so that details found in only one article are less likely to be lost in a combined audit.

   

An atomic claim is one independently checkable statement or instruction. Instead of asking whether an article generally explains a failure, the audit checks specific points: what a symptom indicates, whether a setting must be applied before restart, which version changed a behavior, or which deployment type a workaround doesn’t cover.

Each claim receives a disposition: present, partial or missing. The workflow also identifies conflicts between the sources. Together, the audit results form a coverage table that lists source claims and their status, plus a conflict log showing where the articles disagree and how the merged version handles each disagreement.

That structure changes the verification task. “Is this merge faithful?” asks for a sweeping judgment with no clear stopping point. “Does the merged article say this setting requires a restart?” is a bounded check against a specific claim. The approach breaks a broad assessment into many narrow comparisons that a model can perform without being asked to invent or improve the content.

There’s a second benefit: omission becomes harder to hide. A newly invented command is visible as new material. A dropped prerequisite has no such signal; a reviewer may not know to look for it. Listing the source claims first means each one needs an explicit disposition. An absent detail appears as a row for someone to assess.

Prompt design and technical details

The merge prompt frames both articles as material to process, not instructions to follow. That matters for technical documentation, which often includes imperative language such as “run this command” or “set this value.” Clear framing helps prevent the model from treating those sentences as directions for its own task.

The prompt also asks the model to return three things: the full merged article, a list of conflicts, and a list of omissions with reasons. Requiring the model to account for deletions can reduce careless dropping and draw attention to disagreements. But those lists still come from the model that did the merge. They are leads for review, not independent verification.

For the audit, the merged article and one source article are supplied in separate, clearly marked sections. The model is asked to extract every factual claim, step, value, warning or instruction from that source, splitting compound statements into individual claims. It then classifies each claim as present, partial or missing in the merged article.

Running this audit separately for each source is intentional. If both sources appear in the same audit, the model may blend their information. A detail present in only one article could then be mistakenly treated as covered because the other article contains something similar.

Once the two audit tables are combined, the engineer can focus on the exceptions. Present claims generally need no action. Missing claims are often intentionally excluded because they repeat the other source or don’t meet the quality bar, but the table makes that decision visible. Partial claims deserve close attention: one source may say a problem affects 8.x, while another gives the more precise range 8.12.0 through 8.12.3. A human can decide whether the narrower wording, a caveat or additional precision should be restored.

The engineer makes the final edits by hand. The goal is not to remove human judgment; it is to aim that judgment at specific decisions. Instead of rereading two full articles and hoping to notice what changed, the reviewer gets a short list of claims that need attention. The team describes the review as taking about five minutes rather than an hour.

Guardrails before and around the merge

Verification is only one part of the design. Before merging, the tool checks whether the articles are actually duplicates. Similar subjects can be adjacent without being interchangeable. Combining two distinct topics may produce plausible prose while destroying useful distinctions, so the tool can refuse to proceed when the pair doesn’t qualify.

Copy-editing intensity can also be adjusted. A mature article and a short stub don’t need the same level of rewriting. Engineers can choose a light edit or a fuller rewrite, while the atomic claims remain available for the audit even when the prose changes substantially.

The tool is delivered as a browser extension. The authors describe it as requiring no additional service or new system of record, and say it uses the existing data paths. Working inside the tools engineers already use is part of the design: an elaborate review process has little value if people avoid using it.

What the HTTP 429 example shows

The example merge combines two Elasticsearch articles about HTTP 429 responses. One source describes indexing rejections across platforms. It identifies overloaded nodes and full search or write queues as common causes, and gives diagnostic commands for checking thread pools, indexing pressure and circuit breaker rejections. It also recommends temporarily reducing ingest or search load, adding resources or instances, and, in some situations, adding primary indices.

The other source focuses on an Elasticsearch deployment in Elastic Cloud, with versions 8.17 and 9.x listed. It describes frequent 429 responses in internal proxy logs and points to slow queries, nested aggregations—particularly those around categories.id—and possible memory pressure as areas to investigate. Its suggested actions include adjusting slow log thresholds to capture queries taking longer than one second, reviewing query goals and efficiency, and collecting hot threads and heap dumps while errors are elevated.

The merged article brings those threads together. It explains that 429 responses can affect indexing and search, and may appear in client, Logstash, Beats or proxy logs. It preserves the indexing diagnostics and adds the query-related causes and investigation steps. It also distinguishes temporary load reduction from longer-term options such as scaling resources, distributing load across instances, reducing request complexity and reviewing expensive aggregations.

The resulting article gives readers a wider diagnostic path without implying that every 429 has the same cause. The exact error example and diagnostic commands are retained, while the prose is reorganized. The source article links cover HTTP status 429, Elasticsearch bulk rejections, slow logs and scalability guidance.

The team reports that its four-year backlog is now being worked through at about five minutes of human attention per merge. It also says that no merge has shipped with an undetected omission. That result reflects the team’s experience with this workflow; it doesn’t establish that a model’s audit is infallible. The practical safeguard is that the claim inventory and review step make missing information visible before publication.

Where this pattern applies

The same design can help wherever generated text must preserve details from source material and subtle errors carry a high cost. Keep the generating pass from certifying its own output. Break fidelity into claims that can be checked individually. Make omissions countable, preserve commands and other technical payloads verbatim, and give the reviewer a list of decisions rather than an undifferentiated document.

That architecture doesn’t make the model trustworthy on its own. It makes its work cheaper to inspect. For teams considering AI in a high-stakes content workflow, the useful question is whether they can define the source claims, check them independently and route exceptions to a person. If they can, model unreliability becomes a cost the process can manage—and the review can focus on the details most likely to matter.