What is the recommended architecture for building an AI chatbot using .NET?

For a production AI chatbot in .NET, I’d recommend a modular monolith / Clean Architecture first, with an AI orchestration layer that can evolve into an agent architecture later. You generally don't need microservices just because the application uses an LLM.

Microsoft's current .NET AI guidance supports multiple model providers through Microsoft.Extensions.AI, while Semantic Kernel and the newer agent/tooling ecosystem provide orchestration capabilities. 

Recommended architecture

                    ┌──────────────────────┐
                    │   Web / Mobile UI    │
                    │ Blazor / React / etc │
                    └──────────┬───────────┘
                               │ HTTPS / SignalR
                               ▼
                    ┌──────────────────────┐
                    │    ASP.NET Core API  │
                    │ Auth / Rate limiting │
                    │ Streaming / Sessions │
                    └──────────┬───────────┘
                               │
                    ┌──────────▼───────────┐
                    │   Application Layer  │
                    │  Chat orchestration  │
                    │  Conversation logic  │
                    └──────────┬───────────┘
                               │
              ┌────────────────┼─────────────────┐
              ▼                ▼                 ▼
       ┌─────────────┐ ┌──────────────┐ ┌──────────────┐
       │ LLM Gateway │ │ RAG Service  │ │ Tool/Agent   │
       │             │ │              │ │ Orchestrator │
       └──────┬──────┘ └──────┬───────┘ └──────┬───────┘
              │               │                │
              ▼               ▼                ▼
       Azure OpenAI /   Azure AI Search    APIs / DB /
       OpenAI / other   Vector + Hybrid    Business funcs
              │
              └──────────────┬─────────────────┘
                             ▼
                    ┌──────────────────┐
                    │ Conversation DB  │
                    │ Redis / SQL etc. │
                    └──────────────────┘

1. ASP.NET Core as the application boundary

Use ASP.NET Core for the API rather than allowing your frontend to communicate directly with the LLM.

Responsibilities:

• Authentication/authorization
• Conversation/session management
• Rate limiting
• Input validation
• Streaming responses
• Tenant/user isolation
• Audit logging
• Calling the application layer

Microsoft's .NET architecture guidance recommends separating application concerns even when deploying them as a single application, and Clean Architecture is particularly appropriate for non-trivial ASP.NET Core applications.

A practical project structure:

src/
  Chatbot.Api/
      Controllers/
      Hubs/
      Middleware/

  Chatbot.Application/
      Chat/
      Agents/
      RAG/
      Tools/
      Interfaces/

  Chatbot.Domain/
      Conversations/
      Users/
      Documents/

  Chatbot.Infrastructure/
      AI/
      Search/
      Persistence/
      ExternalServices/

tests/
  Chatbot.UnitTests/
  Chatbot.IntegrationTests/

2. Put an LLM abstraction behind your application

Don't scatter calls to OpenAIClient or AzureOpenAI throughout your controllers.

Instead:

public interface IChatService
{
    IAsyncEnumerable<ChatChunk> StreamAsync(
        ChatRequest request,
        CancellationToken cancellationToken);
}

Then your infrastructure can implement it using the model/provider you choose.

This is particularly useful because the current .NET AI ecosystem supports providers including OpenAI, Azure OpenAI, Azure AI Foundry, Ollama, Gemini and others through common .NET abstractions.

For a Microsoft/Azure enterprise application, I'd typically start with:

ASP.NET Core
    ↓
Microsoft.Extensions.AI / orchestration layer
    ↓
Azure OpenAI / Azure AI Foundry

3. Add RAG when the chatbot needs your data

If the bot needs to answer questions about:

• company documents
• manuals
• policies
• product information
• SharePoint content
• internal knowledge bases

use RAG (Retrieval-Augmented Generation) rather than putting all of that information into the prompt.

User question
     ↓
Query understanding
     ↓
Search
 ┌───┴──────────────┐
 │                  │
Keyword          Vector
search           search
 │                  │
 └───────┬──────────┘
         ▼
   Semantic ranking
         ↓
   Relevant chunks
         ↓
       LLM
         ↓
 Answer + citations

Azure AI Search supports keyword, vector and hybrid search, as well as semantic ranking. (Microsoft Learn)

For an Azure-centric implementation, a very good starting stack is:

Blob Storage → ingestion/chunking → Azure AI Search → Azure OpenAI → ASP.NET Core

Microsoft also documents a .NET RAG architecture combining Azure OpenAI, Azure AI Search and storage, including automatic citations.

4. Treat tools as separate capabilities

If the chatbot needs to do things, don't give the LLM direct access to your database.

Expose controlled tools:

public interface IOrderTools
{
    Task<OrderStatus> GetOrderStatusAsync(string orderId);
    Task CancelOrderAsync(string orderId);
}

The architecture becomes:

                   LLM
                    │
              decides to call
                    │
                    ▼
             Tool interface
                    │
                    ▼
          Application service
                    │
                    ▼
       Authorization + validation
                    │
                    ▼
             External system

This gives you a crucial security boundary.

Microsoft's current agent architecture similarly separates agents, skills and tools, with tools representing discrete operations exposed through defined interfaces/schemas. 

5. Use Semantic Kernel selectively

Semantic Kernel is useful when your chatbot needs orchestration, plugins/tools, memory, structured AI workflows, or agent-like behavior.

Its kernel acts as a central dependency-injection/orchestration point for AI services and plugins. 

For example:

ChatService
    │
    └── Semantic Kernel
          ├── Chat completion
          ├── RAG plugin
          ├── Order plugin
          ├── Customer plugin
          └── Notification plugin

I wouldn't introduce Semantic Kernel merely because "it's an AI application." For a simple chatbot:

ASP.NET Core
    ↓
Microsoft.Extensions.AI
    ↓
LLM

may be enough.

Introduce an orchestration/agent framework once you actually have multiple tools, workflows, or agents.

6. Keep conversation state outside the LLM

Don't rely on the model to remember everything.

Use:

SQL/PostgreSQL/SQL Server

for durable conversation records:

Conversation
 ├── Id
 ├── UserId
 ├── CreatedAt
 └── Title

Message
 ├── Id
 ├── ConversationId
 ├── Role
 ├── Content
 ├── Tokens
 └── CreatedAt

And optionally Redis for:

• short-lived state
• caching
• rate limiting
• distributed locks
• frequently accessed data

For long conversations, don't blindly send the entire history to the model. Use:

Recent messages
       +
Conversation summary
       +
Relevant retrieved memories/documents
       ↓
      LLM

7. Security should be designed into the AI layer

This is especially important for enterprise chatbots.

Your pipeline should look more like:

User
 ↓
Authentication
 ↓
Authorization
 ↓
Prompt/input validation
 ↓
Conversation context
 ↓
RAG with permission filtering
 ↓
LLM
 ↓
Tool authorization
 ↓
Output validation
 ↓
User

For example, if Alice isn't allowed to access document X, the RAG layer must prevent document X from reaching the model. Don't rely on the LLM to decide whether Alice is allowed to see it.

For tools that modify data, add explicit authorization and preferably confirmation for consequential operations.

8. Observability is part of the architecture

Track at least:

Request
 ├── latency
 ├── model
 ├── input/output tokens
 ├── estimated cost
 ├── retrieved documents
 ├── tool calls
 ├── errors
 └── user feedback

You want to be able to answer:

"Why did the chatbot give this answer?"

That means retaining enough trace information to reconstruct:

User question
     ↓
Retrieved context
     ↓
Prompt/model configuration
     ↓
Tool calls
     ↓
Model response

Also build an evaluation set of representative questions and measure things such as answer correctness, retrieval quality, citation accuracy, latency and cost.

What I'd actually deploy

For a typical enterprise chatbot, my starting architecture would be:

Microsoft's current .NET AI documentation specifically covers chat applications, RAG, vector search, agents, tool execution and evaluation, so this stack aligns well with the current .NET ecosystem.  

The important architectural principle

Don't start with "microservices + agents + vector database."

Start with:

ASP.NET Core
      ↓
Application/Chat service
      ↓
LLM
      ↓
SQL

Then add capabilities as requirements appear:

Need company knowledge?   → RAG / AI Search
Need actions?              → Tools
Need complex orchestration? → Agent framework / Semantic Kernel
Need scale?                → Redis + queues + horizontal scaling
Need multiple services?    → Extract services only when justified

That gives you a system that's straightforward to develop and test initially, while leaving room for a sophisticated agent/RAG architecture later. Microsoft's ASP.NET Core architecture guidance likewise notes that a single deployed application is often easier to build, deploy and debug than prematurely splitting it into services.

If you're building a specific chatbot (e.g. customer support, internal enterprise knowledge bot, SQL/database assistant, or an agent that performs actions), the optimal architecture changes substantially.