THE BUSINESS AI PLAYBOOK
Build an AI support assistant that knows when the answer is missing
Create an AI support knowledge assistant with current sources, useful citations and reliable escalation so staff can answer customers with greater confidence.
Discuss with Oliver and his team 
A support knowledge assistant should retrieve the right approved source, explain the answer in context and admit when the evidence is missing or conflicting. Start with internal staff assistance before customer-facing automation. Assign owners to policies and product guidance, preserve permissions, and make escalation easy. Evaluate the assistant on real support cases, including questions it should refuse to answer confidently. A correct escalation is often better than an unsupported reply.
What to take away
- Knowledge quality and ownership matter as much as model capability.
- Test unanswered and conflicting-source cases, not just common FAQs.
- Measure customer resolution and factual accuracy alongside staff time.
Build a source inventory with ownership
List the documents staff actually consult: service policies, product guides, troubleshooting notes, contract-specific terms and known issues. Mark who owns each source, when it was reviewed and which customers or employees may access it. Remove or clearly label superseded versions.
Do not assume every file in a shared drive belongs in a customer answer. Internal exception notes, private account details and draft policies need different access rules. Microsoft’s documentation illustrates why existing permissions remain important when AI retrieves company information. The same practical principle applies to any knowledge system you build: access should follow the user’s legitimate role.
Require evidence and a useful uncertainty response
Define the answer format around the support job. Include the recommended action, the source used and any prerequisites. If the answer depends on product version or account tier, ask for that detail before offering instructions. Treat customer messages and retrieved documents as information, not permission to change the assistant’s operating rules.
When sources conflict, show the conflict to staff and identify the owner who can resolve it. Avoid a generic I cannot help dead end. A useful escalation includes the customer’s question, sources checked, missing fact and recommended destination. That creates a knowledge improvement opportunity rather than another abandoned ticket.
Illustrative support case: two versions of a policy
Imagine a customer asks whether a replacement is covered. The assistant finds an old general policy and a newer product-specific policy. It should not merge them into a convenient answer. It asks for the product version if needed, presents the applicable approved source to the support rep and flags any uncertainty.
The representative reviews the account context and decides the response. If the product-specific policy is incomplete, the ticket goes to its owner. After the owner publishes a correction, the assistant can use the new version. This creates a feedback loop that improves the source material, not just the next generated sentence.
Evaluate resolution rather than answer fluency
Build a test set from real cases with private data handled appropriately. Include obsolete documents, missing account information, ambiguous questions and requests outside support authority. Score factual correctness, source relevance, access control and escalation quality separately.
During rollout, track whether customers reopen tickets or require corrections. A shorter handling time can conceal incomplete answers. Let staff report bad sources directly from the workflow and assign those reports to document owners. Re-run the relevant cases whenever a major policy or retrieval setting changes.
FROM READING TO ACTION
Your first steps
- Choose one support topic with a clear document owner.
- Remove outdated sources and document access boundaries.
- Write an answer format with evidence and escalation requirements.
- Test common questions, missing answers and contradictory documents.
- Review reopened tickets and source corrections with the support team.
What to measure
| Measure | How to track it |
|---|---|
| Supported answer accuracy | Reviewed answers with correct, applicable source support divided by answers reviewed. |
| First-contact resolution | Eligible cases resolved without a repeat contact for the same issue within your chosen window. |
| Useful escalation rate | Escalations containing the required context and reaching the correct owner divided by escalations. |
Common questions
Should we upload every company document?
No. Start with a curated topic where ownership, freshness and permissions are clear. More documents can increase confusion if outdated or conflicting material is treated as equally authoritative.
Can customers use the assistant directly?
Potentially, after internal testing. Customer-facing answers need stricter information boundaries, clear limits and reliable access to a person when the issue exceeds the supported scope.
Sources & further reading
These sources provide background. The workflow designs are Palnau Systems implementation guidance; examples are illustrative, not reported client results. Product capabilities and requirements can change.
Prepared with AI assistance for business owners and managers. For a recommendation based on your actual systems and requirements, work directly with Oliver and his team.
