Learn Chatbots in production

A toy chatbot is 30 lines: maintain a list of messages, send to the API, append the reply. A production chatbot is 5,000 — it streams tokens, handles thousands of concurrent users, persists sessions to a database, grounds answers in your private docs, and stays under budget. The core idea is the same; the engineering around it makes the product.

The core conversation loop

Each turn: (1) take the user's new message, (2) prepend conversation history + system prompt + retrieved context, (3) send to the LLM, (4) stream the response back to the user, (5) save the new exchange to the session. The LLM is stateless — you send the whole conversation every time. Memory lives in your database, not the model.

• System prompt — defines persona, tone, rules, format (set once at session start)
• User message — the current question or request
• Conversation history — last N turns (truncate or summarize when too long)
• Retrieved context — top-K chunks from your vector DB, if RAG (otherwise skip)
• Tools / function calls — for actions (search, DB query, API call)
• Assistant reply — streamed back, also appended to history

Production concerns

Things you'll need beyond the basic loop: (1) Streaming via Server-Sent Events or WebSockets so users see tokens live. (2) Token budget tracking to avoid runaway costs. (3) Conversation summarization when history grows too long for the context window. (4) Rate limiting per user. (5) Content filters for both inputs (jailbreak attempts) and outputs (PII leakage). (6) Observability: log every prompt + response for debugging.