If you are moving from traditional software architecture toward AI architecture, one of the first questions is how an LLM should fit into the systems you design.
You can spend a significant amount of time learning transformer architectures, neural networks, and the mathematics behind how a model predicts the next token. Those subjects matter, particularly for people building and training models. But for most of us as architects and engineers, I would start somewhere else.
For this topic, my recommended target is mainly DESIGN, with some BUILD. You do not need to train a model. You need to understand how to design an application around one, and enough hands-on experience to trust that understanding.
Throughout this series, I’ll keep using four levels to calibrate how deep a topic deserves:
- KNOW: Concepts, terms, limits, and architecture implications.
- DESIGN: Deep enough to decide, compare alternatives, and guide implementation.
- BUILD: Implement a small working example yourself.
- DELEGATE: Know what it does and when it matters; leave the depth to specialists.
Start With the Right Mental Model
At a simplified level, an LLM generates output by predicting the next token, using patterns learned during training and the specific context it receives at that moment.
That sounds simple, but it has real architecture consequences. An LLM is not a database you query for stored facts, and it is not a rules engine executing predefined logic. Its responses depend heavily on the instructions and context it receives, and its output is probabilistic rather than fixed.
The LLM is a component of the solution, not the solution itself.
The application around it still carries responsibility. It assembles instructions and context, controls access to external systems, validates what it can, and decides how the model’s output gets used.
Context Matters
An LLM generates a response based on the information available to it in that moment: application instructions, the user’s request, conversation history, and any information retrieved from elsewhere.
This is an important shift for architects. If the model needs current enterprise information to answer a question, that information does not somehow exist inside the model. We need to design how the right information reaches it, and decide what belongs in that context and what doesn’t. One common approach is Retrieval-Augmented Generation (RAG), which deserves its own discussion later. For now, the point is simpler: the quality of an AI application’s response depends not only on the model, but on the context we choose to provide it.
LLM Output Becomes Part of an Application
When a response is intended only for a person to read, free-form text is often enough. But many AI applications need to pass the model’s output to another software component, store it, validate it, or trigger a workflow from it.
This is where structured outputs matter. Instead of returning a paragraph, the model can be required to produce information using a defined structure. Traditional application logic can then process those fields directly.
There is an important limit here worth remembering: structure can be validated, meaning still needs evaluation. A response that matches the schema perfectly does not guarantee the model’s underlying conclusion was correct.
Same Request, Different Response
One more thing is worth understanding before you design around an LLM: ask it the same question twice, and you may get two different, acceptable answers. That’s nondeterminism, and it’s normal, not a bug.
It’s a different problem from hallucination, where the model states something confidently that isn’t actually supported by the information it had.
Designing for one means accepting variation. Designing for the other means adding checks the model can’t provide for itself.
What’s New, What Transfers
None of this requires you to unlearn how you think about architecture. Assembling the right information for a component to do its job, deciding what a component’s output looks like, and validating what comes back before you trust it, these are integration problems you’ve solved before, just not with a component this variable.
What’s new is the variability itself. A traditional API returns the same response for the same request. An LLM doesn’t, and your architecture has to account for that rather than assume it away.
Resources
The following resources will help you go deeper into the concepts covered in this article.
AI Python for Beginners (deeplearning.ai): Solidifies Python fundamentals. Focus on data structures and the libraries essential for ML, NumPy and Pandas. Depending on your experience with Python, you may need to go through all of it or just a subset of the videos in this course.
Generative AI for Everyone (deeplearning.ai), Week 2: Covers how generative AI works, what it can and can’t do, and includes hands-on exercises in prompting and moving beyond basic prompting into more advanced uses.
ChatGPT Prompt Engineering for Developers (deeplearning.ai): In this course you will be writing and running real prompts against the OpenAI API in Jupyter notebooks. It covers how LLMs actually behave, core principles for writing effective prompts, and practical tasks like summarizing, inferring, and transforming text. If you want to move past reading about context and instructions and actually feel how changing them changes the output, this is the one to do.
Try It Yourself
Start Building Your Architecture Decision Assistant
Phase 1: Try it in a chat interface
Open ChatGPT, Claude, or another chat interface. Give it a short architecture decision from your own work and ask it to review the decision and return a structured response containing:
{
“decision”: “…”,
“rationale”: “…”,
“benefits”: [“…”],
“risks”: [“…”],
“tradeoffs”: [“…”],
“open_questions”: [“…”]
}
Use a real architecture decision if possible, but remove or anonymize any confidential information before sending it to an external AI service.
Start a new conversation and repeat the same request three times. Compare the results. Did the identified decision change? Were different trade-offs highlighted?
The objective is to experience both structured output and nondeterminism. The structure may remain consistent while the content inside it changes.
Phase 2: Turn it into a small application
Write a Python script that sends the same instruction through an LLM API. Request the same structured response and process at least one field in your code.
Run the request several times and compare the results again. This time, also inspect information available through the API or platform, such as token usage and response latency. You can also estimate the cost of each call based on the model’s pricing.
Save the application. In later exercises, you can extend the same assistant with new features.”