An internal AI knowledge search pilot should answer a narrow set of questions from approved documents and show where each answer came from. Retrieval-augmented generation, or RAG, retrieves relevant material for a model to use when composing a response. It does not make missing, outdated or restricted information safe to use.

Start with one question family

Choose a bounded use case such as finding current product specifications or explaining an approved internal process. Identify the intended users and the decisions they make. Avoid starting with every shared drive and an undefined promise to answer everything.

Collect real questions, paraphrases and cases where no answer should exist. For each, record the expected source, the important facts and the user’s permission level. Keep some examples separate from development so the final test is not just a replay of tuning data.

Give each document an owner

Record source location, document owner, version, effective date, review date and access rules. Decide how approved replacements and deletions reach the search index. An old procedure that remains searchable can be more damaging than a clear “no answer found.”

For an illustrative support handbook, distinguish a currently approved return procedure from an archived draft. The system should prefer the approved material and make the status visible in the source reference.

Enforce permissions before generation

A user should only retrieve content they are entitled to see. Apply authorization in the retrieval path, not merely as an instruction telling the model to keep secrets. Microsoft’s document-level access control guidance describes permission enforcement for search-based applications.

Test users with different access, revoked access and shared conversations. Check caches, logs and citations as well as the answer text. Retrieved documents must be treated as information, not as instructions that can override access rules or initiate actions.

Test retrieval and answers separately

  • Retrieval: did the system find the right current document?
  • Answer: are important claims supported by the retrieved passage?
  • Citation: does the link open the supporting source for that user?
  • Uncertainty: does the system decline or ask for clarification when evidence is missing?
  • Operation: are response time and cost acceptable under the expected usage?

Have business reviewers assess difficult answers, not only technically successful requests. A fluent answer with the wrong policy version is still a failed result.

Define the rollout decision

Agree acceptance criteria before running the pilot. Track failed questions, unauthorized retrieval attempts and stale-document cases. Name an owner for feedback and retest after changing the model, prompts, document parsing or permissions.

Explore our AI workflow automation service. Bring a small approved document set and example questions to the project discussion so the pilot has a clear boundary.