A safe assistant answers only from approved sources and only for people allowed to see them.
An internal knowledge assistant lets staff ask questions in plain language and get answers drawn from the company’s internal documents, such as procedures, policies, product information, and past cases. Most are built with retrieval: the system searches an index of approved content, passes the most relevant passages to a language model, and asks it to answer using only those passages.
Safety comes from three decisions made before any model is chosen. Which sources are approved for the assistant to use? How are each user’s existing access rights enforced when documents are retrieved? How does a reader verify an answer? A design that settles those questions is safer with an ordinary model than a design that skips them with the most capable model available.
The failure modes are predictable. An assistant can quote an outdated procedure, reveal a document the person asking should not see, or produce a fluent answer with no basis in any source. Each of these has a design response, and none of them is solved by writing a stricter instruction to the model and hoping it complies every time.
Choose and prepare the sources before choosing the model.
Start with a narrow, valuable body of content, such as the service procedures a field team uses or the policies that human resources answers questions about every week. Identify where the authoritative version lives, who is responsible for keeping it current, and which copies elsewhere should be excluded. An assistant that indexes three conflicting versions of a procedure will confidently repeat all three.
Clean the content before indexing it. Remove drafts and superseded versions, label documents with their effective dates and departments, and fix documents that exist only as scanned images. Long documents should be split along their natural headings so retrieval returns the relevant section rather than a random slice of a manual.
Set up an update path from the start. When a policy changes in the document system, the index should update on a schedule, and removed documents should disappear from it. Stale content is the quietest risk in these systems, because the answers still sound authoritative. A visible last-updated date on each cited source helps readers notice when something looks old.
Enforce permissions at retrieval, not in the prompt.
The most serious risk in an internal assistant is disclosure. If the index contains payroll records, legal matters, customer files, or board materials, a well-phrased question may surface them to someone without access. Instructions telling the model not to reveal sensitive information are not a control. The only dependable approach is to never retrieve content the person asking cannot already open.
In practice this means carrying access information into the index and filtering search results by the signed-in user’s identity and groups before anything reaches the model. The permissions should come from the source systems, such as the document platform or the directory, rather than being recreated by hand. Test the filtering with accounts from different roles as part of every release.
Some content should stay out entirely. Highly sensitive records, regulated information, and material under legal hold are often better excluded than filtered. Where an assistant must handle health, financial, or personal information, confirm the obligations with counsel or compliance advisers first, and choose providers and settings that support HIPAA requirements or other applicable rules instead of assuming a feature list settles the question.
Make every answer traceable to its source.
Every answer should cite the documents and sections it relied on, with links a reader can open. Citations let staff verify important answers, and they reveal when the assistant is drawing on the wrong material. An answer without a citation should be treated as a signal that retrieval failed, not as a finished response.
Instruct the model to decline when the retrieved passages do not contain the answer, and test that behavior with questions the sources cannot answer. A plain statement that the information was not found, with a pointer to the person or team responsible for the topic, is far more useful than a plausible guess that someone then acts on.
Be explicit about the assistant’s role. For procedures with safety, legal, financial, or customer consequences, the assistant should point to the governing document and the responsible person rather than paraphrase instructions. Clear labels in the interface, stating that answers come from internal documents and should be checked for important decisions, set the right expectation from the first question.
Run the assistant as an ongoing system, with review, logging, and a responsible team.
Name a person or team responsible for each content area and for the assistant itself. They review unanswered questions, answers marked unhelpful, and gaps in the source material. Many of the most useful improvements are not technical at all: they are missing procedures that need to be written, or conflicting documents that need to be reconciled.
Log enough to diagnose problems without creating a new store of sensitive information. Record which sources were retrieved and whether the answer was rated useful, and apply retention limits to the questions and answers themselves. Review the model provider’s terms on data retention and training, and restrict who inside the company can read assistant logs.
Expand deliberately. Add a new content area or user group only after the current one is performing well and its permissions are tested. Keep an evaluation set of real questions with known good answers, and run it whenever sources, prompts, or models change. Baybloom builds assistants on this pattern, treating retrieval, permissions, and evaluation as the core of the work.
Put it into practice
Talk through the project you are considering.
Bring the workflow, the systems around it, and the questions you need to resolve. We can help define a useful next step.
Discuss your project