Real AI products are 10% model and 90% engineering. By the time you've shipped, you've spent more time on prompt design, retrieval, caching, observability, latency, fallbacks for hallucinations, cost tracking, and evaluation than on the LLM itself. The model is the easy part — usually one HTTP call to a vendor.
The typical architecture
Most production AI apps follow this stack: (1) UI — web/mobile/desktop interface. (2) API — your backend that orchestrates LLM calls, adds context, validates output. (3) LLM — vendor (OpenAI/Anthropic) or self-hosted (Llama, Mistral). (4) Context plumbing — vector DB for RAG, structured DB for user data, tools for actions. (5) Observability — logs, metrics, evaluation pipelines.
- Code Copilot (GitHub Copilot, Cursor) — IDE plugin → streaming endpoint → code-tuned LLM → open file + project context
- Docs Chatbot (Intercom, Notion AI) — chat widget → REST endpoint → GPT-4o/Claude → RAG over your docs
- AI Image Studio (Midjourney, Krea) — web canvas → job queue → Stable Diffusion XL → prompt + style references
- Voice Assistant (ChatGPT voice, Pi) — mic → WebSocket → Whisper + GPT-4o + TTS → user history + tools
- Search w/ AI (Perplexity, Bing) — query → search index → multiple LLM calls (extract, rank, summarize) → cited answer
Common patterns to know
RAG (retrieval augmented generation) for grounding in private data. Function calling for letting the LLM take actions. Agentic loops for multi-step tasks (plan → execute → reflect). Streaming for responsiveness. Hybrid search (vector + keyword) for better retrieval. Caching of embeddings + LLM responses to cut cost. Re-ranking after retrieval for higher quality top-K.
- Pick a vendor LLM first — don't self-host until you've proven the product
- Start with prompt engineering before fine-tuning
- Add RAG when factual grounding matters (otherwise skip — it's complex)
- Cache aggressively — embeddings, common queries, prompt prefixes
- Measure quality with a real eval set, not vibes
- Set per-user cost caps and alert on outliers
- Log everything for debugging and iteration