How to Build a Company AI Assistant People Actually Use

"Feed our documents to an AI and give the staff an assistant" — the sentence has been heard in a lot of meeting rooms over the past two years. Approving the project is easy. The hard part is the next sentence: who does this assistant serve, and what questions does it answer? If the answer is "everyone, about everything", the project usually ends as a chat box nobody opens.

Step one is narrowing, not expanding

The do-everything assistant fails in a predictable way: with no boundary on the document scope, preparation never finishes; when anything can be asked, anything can be answered badly, so trust never forms; and with no defined user, nobody owns "is it good", so there is nothing to accept the project against.

A workable starting point is one of three: internal knowledge Q&A (policies, processes, product material — what new hires and cross-team colleagues ask most), customer support (high-frequency customer questions with documented answers), or sales support (product specs, quoting rules, common competitor questions). Choose by two tests: where repeated questions burn the most staff time, and where authoritative source material already exists. The scenario that passes both is your entry point.

Think about it in three layers

Once the scope is narrow, you can talk about "building". A company AI assistant splits into three layers, each answering one question — and once the layers are clear, budget and division of labour become clear too.

The model layer answers "who generates the reply". This is an off-the-shelf large model, paid per use — a rented brain. The vast majority of companies should not train their own; there are fewer real choices at this layer than people expect, and smaller differences.

The knowledge layer answers "how it knows your company". Your product manuals, policy documents and past Q&A are organised and indexed, then retrieved before the assistant answers, to serve as the basis for the reply. Technically this is retrieval-augmented generation — the mechanics are covered in What Is RAG?.

The application layer answers "where employees actually meet it". Whether that is a bot inside WeCom or a side panel inside your business system decides whether it becomes a daily tool or a demo prop.

The most common misallocation is pouring all attention into the model layer — endlessly comparing providers — when in practice, most of the difference between a useful assistant and an ignored one is made in the knowledge and application layers.

The knowledge layer sets the ceiling on answers

The model's job is to organise whatever was retrieved into a fluent reply. If the material itself is outdated, contradictory or has no authoritative version, a stronger model just circles inside the wrong material — and packages the mess more convincingly. Half the real work of building an assistant is therefore document work: identify the authoritative version of each topic, retire expired files, fill the gaps behind high-frequency questions. It does not have to be perfect on day one — A Knowledge Base for Small Teams covers how to start light.

The application layer test: one step fewer

Every extra action between an employee's question and the answer costs adoption. An assistant that requires opening a separate site and logging in again fades to zero once the novelty wears off; one that can be @-mentioned in WeCom or DingTalk, or sits inside the system employees keep open all day, has a chance of becoming a habit. The same holds for customer-facing assistants: put them in the channels customers already use, rather than asking customers to come and find you.

Cold start: teach it to answer inside a small circle first

The document base will never reach "perfect", so instead of saving up for a grand launch, run a fixed loop:

1. Pick around twenty genuinely high-frequency questions from real ones people ask, and prepare the source material behind their answers.
2. Open the assistant to a small group — one department or one support team — for two to three weeks.
3. Every week, export the list of questions it failed to answer or answered wrongly.
4. Backfill material and adjust how documents are organised against that list, rather than adding files at random.
5. Once the hit rate stabilises, widen the audience or the question scope.

The value of the loop is that every hour spent on the knowledge layer serves a demonstrated need instead of a guess about what employees might ask. For deciding when the trial has passed, see writing pilot acceptance criteria — it turns "works well" into testable conditions.

Assemble it yourself, or bring in a vendor

Off-the-shelf parts already cover a lot: the bot interfaces of mainstream collaboration platforms plus a mature knowledge base tool, and one motivated person can assemble a usable internal Q&A assistant in a week or two. Doing it yourself is reasonable when three things hold: the question domain is narrow, the document volume is modest, and someone on the team is willing to look after it long-term.

Conversely — if the assistant must pull data from business systems, enforce permissions by department and role, cover multiple entry points, or carries real external cost when it answers wrongly — an external vendor earns its fee. What you are buying is not just development but implementation experience and post-launch operations support. The deciding question is not technical but operational: after launch, who reviews the unanswered questions, and who backfills the material? If nobody inside the company owns that, no vendor can save the project.

In the end, a company AI assistant is not built so much as raised. Narrow the entry point, get the three layers straight, move into the daily workflow, run the backfill loop — do those four things properly, and "people actually use it" stops being a matter of luck.