The LLM Application Stack: From Model to Product

Understanding the LLM application stack

When you deploy an LLM in production, the raw model—no matter how powerful—is only one small piece. Between a model's weights and a user's answer lies an architecture with four distinct layers, each with its own failure modes, optimization levers, and design decisions.

Most teams shipping their first RAG system get surprised by this: the bug reports that come back don't say "your model is dumb." They say "we asked about our return policy, and the bot didn't mention the three-day window," or "the search results were completely irrelevant," or "the response took 45 seconds." These are not model bugs. They're bugs in the other three layers.

Understanding this stack matters because it changes how you debug, optimize, and reason about where to invest engineering effort. A 20% accuracy improvement in the retrieval layer (data layer) will move your product's quality more than switching from GPT-4 to Claude (model layer). An async retrieval strategy (orchestration layer) will cut your p99 latency more than a faster model. This course builds around the stack from bottom to top.

Layer 1: The model layer

The model layer is where the actual neural network lives—Claude, GPT-4, Llama, or whatever LLM you choose. Its job is single: given a prompt, produce tokens that form a coherent continuation. The model has a knowledge cutoff, a context window, a cost per token, and a latency profile (how long it takes to respond).

What the model layer does:

• Reads the prompt (system message, retrieved context, conversation history, user question)
• Generates tokens one at a time (or in parallel, depending on implementation)
• Outputs text (or structured data, or function calls)

What the model layer does NOT do:

• Fetch data
• Route queries to different models based on content
• Format answers for specific use cases
• Track what information sources were used
• Validate that the output is correct or grounded in retrieved facts

This last point is crucial. A common misconception is that upgrading the model will fix hallucinations, missing context, or incorrect citations. Often it won't. If the retrieval layer handed the model irrelevant chunks, the model can't fix that no matter how intelligent it is. If the orchestration layer didn't tell the model to cite sources, the model can't be blamed for not citing.

Model layer costs scale linearly with usage: The larger the model and the longer your context window, the more you pay per request. This is why smaller, cheaper models exist: they're good enough for many tasks, and the cost savings are real.

Layer 2: The data layer

The data layer includes everything that has to be prepared before the model sees it: documents, embeddings, vector indices, metadata stores, and structured knowledge. For RAG systems, this is where chunking happens, where semantic similarity is computed, and where retrieval is executed.

What the data layer does:

• Stores source documents (PDFs, web pages, internal wikis, code repositories)
• Chunks large documents into retrievable pieces (covered in depth in a later lesson)
• Computes embeddings for those chunks using an embedding model (different from the LLM)
• Builds indexes that enable fast nearest-neighbor search over millions of chunks
• Retrieves the most relevant chunks in response to a user's query
• Handles metadata (source URL, date, version, access control)

What the data layer does NOT do:

• Understand context or intent beyond what embeddings capture
• Know whether retrieved chunks actually answer the user's question
• Make routing decisions (e.g., "this question needs a different tool")

Quality bottleneck: The data layer is where most retrieval failures live. Bad chunking can split a fact away from the terms that would retrieve it. Stale documents mean the model is grounded in outdated information. Poor embedding models can fail on domain-specific language. And even perfect retrieval can fail if the query and documents use very different vocabulary.

Data layer costs scale with volume and storage: thousands of documents? Manageable. Millions? You need a proper vector database, not a basic text search. The embedding model is usually run once per document (at ingest time), so embedding cost is typically amortized across many queries.

Layer 3: The orchestration layer

The orchestration layer is the logic that ties everything together: routing, prompting, tool calling, retry logic, multi-step reasoning, and the RAG pipeline itself. If the model layer is "generate tokens" and the data layer is "store and retrieve facts," the orchestration layer is "decide what to ask, in what order, and what to do with the answer."

What the orchestration layer does:

• Builds prompts (system message, few-shot examples, retrieved context, user input)
• Decides which model to call for which task (routing)
• Implements tool use and function calling, interpreting the model's responses
• Implements the retrieval step in RAG (query embedding, database search, result insertion into prompt)
• Handles multi-turn conversations (appending history appropriately)
• Implements retry logic and error handling when the model fails
• Validates and structures the model's output

What the orchestration layer does NOT do:

• Understand the semantic meaning of data (that's the data layer)
• Generate novel text (that's the model)
• Serve the response to a user (that's the application layer)

Quality bottleneck: Orchestration bugs are subtle and often invisible. A prompt that drops context mid-way through a conversation. A RAG system that inserts retrieved chunks in the wrong order or doesn't explicitly tell the model to stay grounded in those chunks. A tool-calling loop that doesn't handle partial failures. A routing decision that sends the wrong query type to the wrong model. All of these are orchestration bugs, not model bugs.

Orchestration is where prompt engineering, context window management, and token economics come into play. How much of your context window should you reserve for the model's response versus retrieved context? How do you tell the model "answer ONLY using these chunks" without breaking its ability to reason? These are orchestration questions.

Layer 4: The application layer

The application layer is what users actually touch: a chat UI, a REST API, a Slack bot, an embedded widget. Its job is to receive a user's input, pass it through the stack, and return a result in a usable form.

What the application layer does:

• Receives user input (text, voice, structured data)
• Formats input for downstream layers
• Handles authentication and access control
• Calls the orchestration layer
• Formats the response for display (markdown, JSON, inline citations, etc.)
• Tracks metrics and logs for debugging
• Handles rate limiting and abuse prevention

What the application layer does NOT do:

• Decide whether an answer is correct (that's the orchestration layer's job via retrieval and grounding)
• Generate the answer (that's the model's job)
• Understand your domain or data (that's the data layer)

Quality bottleneck: Application-layer bugs are usually about user experience, not correctness. A slow UI due to poor loading states. A mobile app that crashes under poor network conditions. An API that returns results in the wrong format. A Slack bot that doesn't preserve conversation context correctly. These matter for shipping a product people actually use, but they won't help if retrieval is broken or the model is hallucinating.

How the layers interact: an example

A user asks a customer support chatbot: "Can I return this product if I opened the box?"

1. Application layer: Receives the question, authenticates the user, passes it to orchestration
2. Orchestration layer: Decides this is a RAG question (needs to retrieve policy, not just generate). Prepares the query.
3. Data layer: Embeds the query, searches the vector database for relevant policy documents, returns the top 3 chunks about returns
4. Orchestration layer: Inserts those chunks into a prompt, tells the model "Answer based ONLY on the provided context"
5. Model layer: Reads the prompt, generates a response: "Yes, opened items are returnable within 30 days per our policy..."
6. Application layer: Formats the response as markdown, adds a citation link to the policy document, displays it in the chat UI

Where can bugs happen?

• Data layer: The vector search returned chunks about "shipping" instead of "returns" (bad chunking, or the word "return" is used in different contexts and embeddings confused them). Now the model has the wrong context.
• Orchestration layer: The prompt was built wrong—maybe the retrieved chunks were included but the model wasn't told to stay grounded in them, so it hallucinated details from training data. Or the orchestration layer forgot to append the chunks at all.
• Model layer: Unlikely—if retrieval worked and the prompt was clear, the model will almost certainly give the right answer.
• Application layer: The chat UI crashed, or the response didn't display the citation.

A user reports "the chatbot gave me the wrong return policy." The bug is almost never "the model is stupid." It's usually "retrieval failed" or "we didn't tell the model to ground its answer in what we retrieved."

Why this stack matters for the rest of this course

This course is organized around the stack:

• Lessons 2-3 (Beginner): Token economics (orchestration layer) and embeddings (data layer). These are the foundations everything else builds on.
• Lessons 4-6 (Intermediate): Function calling (orchestration), model selection (model layer decision-making), and chunking strategies (data layer). This is where you start making concrete tradeoffs.
• Lessons 7-13 (Intermediate-Advanced): Deep dives into the RAG pipeline, vector databases, retrieval optimization, and structured data handling—all data layer.
• Lessons 14-20 (Advanced): Evaluation and monitoring, which cut across all layers—you'll measure what's broken and trace it to which layer needs the fix.

Each lesson is intentionally scoped to one layer, with one core concept, so you can master them independently and then compose them into a working system.

Worked example: Debugging a production RAG failure

You ship a customer-support RAG chatbot. For the first month, it's great: 80% of queries are answered without human escalation. Then you add 50 new product support docs. Performance drops to 60%. What do you debug?

First instinct: "The model isn't smart enough with more context."

Right approach: Check the stack layer by layer.

1. Data layer first: Is retrieval working? Pick 20 real customer queries and manually check whether the top-5 retrieved chunks actually contain answers.

   • You find: For 8 of the 20 queries, retrieval is returning chunks about "billing" when the question was about "product setup." The new docs used different terminology than your old docs.
   • Fix: Re-chunk the new docs more carefully, or add synonyms to metadata.

2. Orchestration layer: Did the prompt change? Did you accidentally remove the "answer ONLY using these chunks" instruction?

   • You find: The prompt is fine, but you're retrieving 20 chunks now instead of 5 (because you have 50 new docs and didn't tune retrieval). The model is getting confused by too much context.
   • Fix: Tune retrieval to return fewer, higher-confidence chunks.

3. Model layer: If 1 and 2 pass, only then test whether the model itself is the bottleneck.

   • You probably won't need to upgrade the model. The degradation was data and orchestration, not the model.

This is why understanding the stack matters: it tells you where to look first when something breaks.

Common mistake

Conflating "the model is bad" with "the application doesn't work." A state-of-the-art model will still fail if:

• Retrieval returns irrelevant chunks (data layer bug)
• The prompt doesn't ground the model in retrieved facts (orchestration layer bug)
• The application returns the answer in the wrong format (application layer bug)
• You don't actually measure whether answers are correct (no layer, just blindness)

The opposite mistake is just as common: investing all your engineering effort in the model layer (fine-tuning, prompt engineering, trying the latest model release) when 80% of your bugs are in data and orchestration. Measure across all four layers, and optimize where the data tells you the problem actually lives.