How LLM Enterprise Search Works

Two employees typing the same question should sometimes get different answers.

If they do not, the system has either over-shared or under-served one of them. Compensation policy, deal rooms, and legal hold folders make that concrete.

We treat connectors, identity, permissions, ranking, freshness, citations, and abstention as part of answer quality. The model is only the last visible step.

Chat boxes hide a great deal of search engineering.

Two employees should not always see the same answer

Web search ranks public pages for anonymous users. One query has one best answer, and everybody who types it deserves the same result.

Enterprise search inverts that. Documents are private, the asker is a named identity, and the correct answer changes depending on who is asking.

Three other web-search intuitions fail here. There is no link graph, so authority cannot be inferred from what other pages cite. There is no web-scale click stream to learn relevance from, because a thousand-person company generates a rounding error of queries. And the corpus is mostly hostile: superseded drafts, half-finished decks, contradictory policies, and documents whose owner left two years ago.

Enterprise search is also more than retrieval over a document folder. Production systems own connectors to live sources, identity resolution, incremental freshness, revocation handling, analytics, and an administration surface for the people who have to explain a bad answer.

One boundary before we continue. We have not operated one of these systems at enterprise scale. What follows is an architecture guide assembled from vendor documentation, retrieval literature, and the failure patterns those two describe consistently, and it should be read as a design checklist rather than a report from a deployment.

Generation is stage six, and the earlier stages fail quietly

Every request passes through seven stages:

1. Connect: ingest documents and metadata from source systems.
2. Authorize: map source users, groups, sharing rules, and document permissions.
3. Index: extract text, split content, and build lexical and vector representations.
4. Retrieve: turn the question into searches and fetch candidate evidence.
5. Rank: order or rerank passages for the current user and intent.
6. Generate: ask a language model to answer from selected passages.
7. Verify: attach citations, apply policy, log the trace, and abstain when support is missing.

Errors compound. Strong models cannot cite a document the connector missed. Good retrievers cannot repair a permission filter applied after private text has already entered the prompt.

Notice which stage gets the budget and the demo time. Stage six is the one people evaluate, and it is the only one where a vendor swap is easy. Stages one through three decide whether the answer was ever possible, and they fail quietly, because a document that was never indexed produces no error anywhere.

A permission filter after the prompt is already a leak

Enterprise sources carry different access models: direct users, groups, nested groups, domains, public links, row-level rules, and application roles. The search system must translate all of those into one query-time decision.

Filter evidence before it reaches the language model. Post-generation redaction cannot work, because the model has already read the protected content and can paraphrase it without repeating the string a redaction rule was watching for.

Test with users whose access overlaps but is not identical. Revoke a group, delete a file, change a shared link, and move a document between spaces. Then measure how long the index and caches take to reflect each change, because the window between a revocation and its propagation is a live exposure with a duration you can measure and should publish internally.

Caches deserve separate scrutiny. An answer cached for one user and served to another is a leak that no permission filter will catch, since the filter already ran correctly the first time.

Clean demos are not permission evidence.

Hybrid retrieval is the default because the corpus is hostile

Keyword search is precise for product codes, names, exact phrases, and uncommon terms. Vector search helps when a question and a document express one idea in different words. Hybrid retrieval combines both, then a reranker inspects the top candidates more deeply.

Chunking affects every stage. Chunks that are too small lose context, chunks that are too large add irrelevant text and consume model input. Preserve titles, headings, dates, owners, source links, and access metadata beside each passage.

Freshness deserves a ranking signal of its own. Internal corpora accumulate superseded versions, and pure relevance scoring will happily surface a 2023 policy that reads more on-topic than the 2026 revision that replaced it. Recency, document status, and owner activity all belong in the ranking features rather than in a disclaimer under the answer.

Query rewriting can expand acronyms, add a product name, or split a broad question into searches. Log every rewritten query. Without that receipt, administrators cannot explain why a known document was missed, and "the AI didn't find it" becomes an unfalsifiable complaint.

Give the model the question, retrieved passages, source identifiers, and explicit instructions to distinguish supported facts from inference. Require citations at the claim or paragraph level. A list of sources at the end is weaker, because readers cannot tell which source supports which sentence, and neither can an auditor.

Abstention is a feature. When retrieval returns weak or conflicting evidence, the system should say what it found, what remains uncertain, and which source or person can resolve it. A fluent unsupported answer is worse than a plain list of search results, because it removes the reader's own judgment from the loop.

Models can also produce structured output: selected source IDs, confidence reasons, follow-up questions, and tool arguments. Validate that schema in the application rather than trusting well-formed text.

The retriever is the upgrade that matters

Compare models behind the same retriever, passages, system instructions, and output schema. Score citation support, unsupported claims, instruction following, tool arguments, latency, and price.

Context window and price are the two columns that bind hardest here, because retrieved passages arrive on every request and a search answer is mostly input tokens:

Table 1 Download CSV

Model	Creator	Input	Output	Context	Overall Score
Gemini 3.6 Flash	Google	$1.5	$7.5	1M	75
Grok 4.5	xAI	$2	$6	500K	75
GPT-5.4	OpenAI	$2.5	$15	1.05M	73
Kimi K3	Moonshot AI	$3	$15	1.05M	80
Claude Opus 5	Anthropic	$5	$25	1M	83
GPT-5.6 Sol	OpenAI	$5	$30	1.05M	82
Claude Opus 4.8	Anthropic	$5	$25	1M	76
GPT-5.5	OpenAI	$5	$30	1M	73
Claude Mythos 5	Anthropic	$10	$50	1M	83
Claude Fable 5	Anthropic	$10	$50	1M	83

Read that table as a candidate list, not a ranking. Faster and cheaper models are often the better search models when the evidence is already strong and the answer format is narrow. Larger models help when synthesizing conflicting policies or long technical material.

Neither compensates for low retrieval recall. That is the single most expensive mistake in this category: teams upgrade the model to fix answers that failed because the right document never entered the prompt, then pay more per request for the same miss.

SearchBlox's Mercury integration report illustrates the component boundary, with the search platform supplying hybrid retrieval and RAG while Mercury supplies generated text. That report documents a partnership and provider claims rather than a controlled cross-product benchmark, and we have not reproduced it.

A private question set is the only quality number that counts

Collect 100 to 300 questions from search logs, support tickets, onboarding, and subject-matter experts. For each, record expected sources, an acceptable answer, forbidden sources, and which test users may see the evidence.

Include:

• exact lookups and broad synthesis,
• acronyms, misspellings, and old product names,
• stale and conflicting policies,
• questions with no supported answer,
• recently edited, deleted, or shared documents,
• multilingual queries where the corpus supports them, and
• instructions inside documents that try to redirect the model.

Measure retrieval recall separately from answer quality. If the expected passage never reached the prompt, classify the failure as search. If the passage was present and the model ignored or distorted it, classify it as generation.

Keep a third class for access and freshness failures. A relevant passage can be wrong because the user was not allowed to see it, or because a newer document should have replaced it. Mixing those into answer accuracy makes the remediation invisible, and remediation is the entire point of measuring.

That split tells the team what to fix, and it usually says retrieval when everyone expected it to say model.

Re-run the set on a schedule rather than once at selection. Corpora drift, connectors break silently after a source-system update, and a question set that passed in March is the only way to notice.

Managed tools such as Glean package connectors, identity, indexing, retrieval, answers, and administration. Microsoft 365 Copilot fits organizations whose knowledge lives mainly inside Microsoft Graph. Google Agent Search and Elastic give builders more control. Coveo, Guru, and SearchBlox target particular search and deployment models.

Our enterprise AI search guide maps those operating models for a first purchase. The Glean alternatives comparison covers the harder case, where an index already exists and switching has a migration cost.

Whichever product wins, keep the evaluation set and the permission tests outside the vendor. Enterprise search becomes dependable when the organization can reproduce why an answer appeared, who was allowed to see it, and which evidence supported it, using artifacts that survive a contract renewal.