System design: an AI chat assistant for a website

Key points

• Answer only from the site’s own content, and let the server, not the model, turn cited chunk IDs into links.
• Re-index on publish through a queue, keyed by content hash, so unchanged text is never embedded twice.
• Use hybrid retrieval and a reranker, with a relevance threshold that refuses off-topic questions without calling the model.
• Stream with server-sent events: a chat answer is one-way traffic, and the format is plain HTTP.
• Layer the caches: whole answers, embeddings, and the provider’s prompt caching for the fixed prefix.
• Plan for the provider being down. A fallback answer built from search results beats an error message.

Requirements

Start the way you would in a system design interview: pin down what the assistant must and must not do before drawing boxes. A reasonable set for an assistant that answers visitors’ questions about a company’s website:

Non-goals matter as much. This assistant takes no actions: no account changes, no orders, no tools that write. That keeps it a retrieval system rather than an agent and removes a whole class of risk. If it later needs to act, the tool calling loop is the next layer.

High-level architecture

Two paths share one index. The write path runs when content changes; the read path runs for every question.

WRITE PATH (runs when content is published)
CMS publish webhook -> job queue -> ingestion worker:
  fetch page -> extract main text -> split by headings into chunks
  -> hash each chunk -> embed only new hashes -> upsert into the index
  -> delete chunks of removed pages -> bump INDEX_VERSION

READ PATH (runs per question)
browser -> chat API: validate -> rate limit -> response cache?
  -> rewrite follow-up into a standalone question
  -> retrieve: keyword (BM25) + vector -> fuse -> rerank -> threshold
  -> prompt: fixed instructions | passages with IDs | question
  -> model, streaming -> SSE to the browser
  -> check cited IDs -> server builds the links
  -> redacted log, metrics, feedback

Keyword and vector search can live in one store, such as Postgres with a vector extension or a search engine with vector support, or in two. Hybrid search covers how the two result lists are fused, what a vector database is covers the storage side, and RAG explained covers the retrieve-then-generate pattern the read path follows.

Ingestion and the data model

Trigger ingestion from the CMS publish event rather than a periodic crawl, so an edited page is answerable as soon as its job runs. Put the work on a queue: embedding calls are slow and rate-limited, and a queue gives you retries, backpressure and a dead-letter list. See background jobs and queues.

• Make jobs idempotent: key them by URL and content hash, so a duplicate webhook does nothing.
• Chunk along the page’s own structure, headings first and then paragraphs, and keep the heading path with every chunk. It improves retrieval and makes a readable citation label. Chunking strategies for RAG covers sizes and overlap.
• Embed only chunks whose hash is new. That is the embedding cache, and it makes re-publishing a page with one edited paragraph cheap.
• Either build a new index version and switch to it, or upsert in place and delete a page’s stale chunks. Either way, key the answer cache by index version so old answers expire with old content.
• Index only what is public. Drafts, admin screens and anything behind a login stay out, because whatever is indexed can be quoted.

The query path

1. Validate: cap the question length and the number of turns in a conversation.
2. Rate limit per IP and per session with a token bucket in a shared store. Over the limit, return status 429, which RFC 6585 defines, with a Retry-After header. The IETF draft on RateLimit header fields adds RateLimit-Policy and RateLimit so clients can see their quota; it is still an Internet-Draft, and it says Retry-After takes precedence when both are sent.
3. Normalize the question (case, whitespace, punctuation) and check the answer cache for the current index version.
4. Rewrite a follow-up into a standalone question from the last few turns, so "what about the mobile app?" retrieves the right pages.
5. Retrieve candidates with keyword and vector search, fuse the two lists, and rerank the top few dozen down to a handful.
6. If the best score is below a threshold, skip the model and send the out-of-scope message.
7. Build the prompt: fixed instructions first, then the passages with their IDs and titles, then the conversation and the question.
8. Stream the answer, then check its citations: keep only IDs that were in the prompt and map them to page links.
9. Log the redacted exchange with latency and token counts.

async function answer(q: Question, send: (event: string, data: object) => void) {
  await rateLimit.consume(q.ip, q.sessionId);    // over the limit: 429 + Retry-After
  const cacheable = q.history.length === 0;      // first questions only
  const key = cacheKey(normalize(q.text), INDEX_VERSION);
  const hit = cacheable && (await answerCache.get(key));
  if (hit) return send("done", hit);
  const query = cacheable ? q.text : await rewrite(q.history, q.text);
  const fused = rrf(await keywordSearch(query, 40), await vectorSearch(query, 40));
  const passages = (await rerank(query, fused)).slice(0, 6);
  if (!passages.length || passages[0].score < MIN_SCORE) {
    return send("done", { text: OUT_OF_SCOPE, citations: [] });  // no model call
  }
  const prompt = buildPrompt(STATIC_RULES, passages, q.history, q.text);
  let text = "";
  try {
    for await (const delta of model.stream(prompt, { maxOutputTokens: 600 })) {
      text += delta;
      send("delta", { text: delta });
    }
  } catch {
    return send("fallback", { text: FALLBACK, citations: toLinks(passages.slice(0, 3)) });
  }
  const done = { text, citations: toLinks(citedPassages(text, passages)) };
  send("done", done);
  if (cacheable) await answerCache.set(key, done);
}

Two details carry most of the quality. The threshold is what makes refusing off-topic questions cheap and dependable: a question about the weather finds no passage above it, and the model is never called. And citations are checked after generation: the model may only cite IDs it was given, and the server, not the model, writes the URLs, so an invented link never reaches the page. More on grounding is in how to reduce AI hallucinations.

Streaming: server-sent events or WebSockets

A chat answer is one request followed by a one-way stream of tokens. Server-sent events (SSE) fit that shape: plain HTTP, a text/event-stream response, and a line format that the WHATWG HTML standard defines with four fields: event, data, id and retry. When a browser’s EventSource reconnects, it sends the last ID it saw in a Last-Event-ID header, so the server can resume.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache

event: delta
data: {"text": "Orders ship from the nearest warehouse"}

event: delta
data: {"text": " and include a tracking link."}

: keep-alive comment, ignored by the client

event: done
data: {"citations": [{"title": "Shipping", "url": "/help/shipping"}]}

• EventSource only makes GET requests and cannot set custom headers, so many chat clients POST the question with fetch() and parse the same event-stream format from the response body.
• The standard suggests a comment line about every 15 seconds so that proxies do not drop an idle connection.
• Turn off response buffering in every proxy on the path, or tokens arrive in bursts.
• The standard also warns that browsers limit connections per server; HTTP/2 multiplexes streams over one connection, which removes most of that problem.

WebSockets earn their place when the server also pushes unprompted, as with a handoff to a person, typing indicators or voice, or when the client streams continuously. They need their own reconnection logic and connection-aware load balancing. The trade-offs are in WebSockets vs server-sent events.

Upstream is SSE too. With stream: true, OpenAI’s Responses API sends typed events such as response.output_text.delta and response.completed, and Anthropic’s Messages API sends message_start, content_block_delta events carrying text_delta payloads, and message_stop. Both can send an error event in the middle of a stream (Anthropic’s example is overloaded_error), so the server must handle failure after the first token, not only before it. OpenAI also notes that streaming makes moderation harder because partial output is harder to judge, so check input before the stream and the finished text after it.

Caching layers

Cache answers only to first questions with no conversation context, and never anything personal. Normalizing text catches trivial variants. Matching paraphrases by embedding similarity catches more, at the risk of serving an answer to a slightly different question, so set that threshold high and evaluate it.

Provider prompt caching rewards a stable prefix. OpenAI applies it automatically and reports hits in usage.input_tokens_details.cached_tokens; its docs say reuse needs the entire rendered prefix to match and recommend putting static content first. Anthropic caches up to a cache_control breakpoint, in the order tools, system, messages, with up to four breakpoints, one of which the API can place automatically; the default lifetime is five minutes, refreshed on every hit, with a one-hour option, and hits show in cache_read_input_tokens. Both have a minimum prompt length below which nothing is cached. So put the fixed instructions and examples first and the passages and question last: passages change per question, so the instruction prefix is the part that gets reused. More levers are in how to reduce LLM costs.

Guardrails

The OWASP Top 10 for LLM Applications (2025) makes a good checklist. The entries that apply to a website assistant:

Scope is enforced in three places, not one: the retrieval threshold, the system instructions, and an output check that the answer cites at least one passage. See prompt injection and AI agent guardrails for more.

Evaluation, monitoring and failure modes

Build a test set before launch: real questions with the pages that should answer them, plus off-topic and adversarial questions that should be refused. Score retrieval (did the right page come back?) separately from generation (is the answer supported by the passages, and are the citations right?), and rerun the set on every prompt, model or chunking change. How to evaluate RAG covers the metrics.

• Latency: time to first token and total time, at the median and the 95th percentile.
• Quality: refusal rate, answers without a citation, negative feedback, and questions with no good passage, which point to content gaps.
• Spend: tokens in and out per answer, and hit rates for each cache layer.
• Errors: provider errors by type, streams that break mid-answer, and rate-limit rejections.

A capacity estimate, with example numbers

Interviewers want to see the arithmetic, not exact figures. The numbers below are illustrative assumptions for a mid-sized content site, not measurements; replace them with your own.

EXAMPLE ASSUMPTIONS: illustrative only, not measurements

Index
  pages = 2,000, chunks per page = 15        -> 30,000 chunks
  1,024-dim float32 vectors = 4 KB each      -> about 120 MB of vectors

Traffic
  questions per day = 20,000; peak = 10x the average
  average = 20,000 / 86,400 s               -> about 0.23 questions/s
  peak                                      -> about 2.3 questions/s

Streams and tokens
  answer streams for about 8 s              -> about 18 open streams at peak
  prompt = 3,000 tokens in, 300 tokens out  -> about 7,600 tokens/s at peak
  response cache hit rate = H               -> model calls = (1 - H) x questions

Read it the way an interviewer would. With these assumptions the index fits in one machine’s memory, so a separate vector database is a choice, not a requirement. Request volume is modest; the binding constraint is the provider’s rate limit on tokens, which is why caching and a cap on prompt size matter more than more servers. And the chat API holds streams open, so size it by concurrent connections, not only by requests per second.

BYD Tech builds AI agents and assistants with the code in your GitHub from day one and AI-assisted work reviewed by an engineer.