--- title: "Kafka Implementation in .NET Core" description: "Learn how to implement Apache Kafka in .NET Core step by step. Understand Kafka producers, consumers, topics, partitions, consumer groups, and BackgroundService" author: "Anubhav Sharma" published: 2026-08-18 updated: 2026-08-18 canonical: https://answers.mindstick.com/blog/556/kafka-implementation-in-net-core category: "programming" tags: ["Kafka", ".net programming"] reading_time: 17 minutes --- # Kafka Implementation in .NET Core Apache Kafka can look complicated when you first hear terms like **producer, consumer, broker, topic, partition, and offset**. But the basic idea is actually quite simple: > **One application sends messages to Kafka, and another application reads those messages from Kafka.** ## What Is Kafka? Apache Kafka is a distributed event-streaming platform. In simple terms, Kafka helps different applications communicate with each other by sending and receiving messages. Imagine you have an e-commerce application. When a customer places an order, several things may need to happen: - Save the order. - Send an email. - Update inventory. - Generate an invoice. - Notify another system. Instead of making the order API call every system directly, we can send an **OrderCreated** event to Kafka. ```plaintext Order API | | OrderCreated v Kafka | +----> Email Service | +----> Inventory Service | +----> Invoice Service ``` This makes the system more loosely coupled and easier to scale. ## Important Kafka Concepts Before writing code, let's understand a few basic Kafka terms. ### 1. Producer A **producer** is an application that sends messages to Kafka. For example: ```plaintext Order Service ---> Kafka ``` The Order Service is the producer. ### 2. Consumer A **consumer** reads messages from Kafka. ```plaintext Kafka ---> Email Service ``` The Email Service is the consumer. ### 3. Topic A **topic** is like a named channel where messages are stored. For example: ```plaintext orders payments notifications ``` An order event could be sent to the `orders` topic. ### 4. Broker - A **Kafka broker** is a Kafka server that stores and serves messages. - A **Kafka cluster** can contain multiple **brokers**. - For a beginner's local setup, you can start with a single broker. ### 5. Partition A topic can be divided into multiple **partitions**. For example: ```plaintext orders ├── Partition 0 ├── Partition 1 └── Partition 2 ``` Partitions allow Kafka to process large amounts of data in parallel. You don't need to understand every detail of partitions to build your first Kafka application. Just remember: > A topic can contain multiple partitions, and Kafka uses partitions to scale message processing. ### 6. Offset An **offset** is the position of a message inside a partition. For example: ```plaintext Offset 0 -> Order 101 Offset 1 -> Order 102 Offset 2 -> Order 103 ``` Kafka uses offsets to keep track of which messages a consumer has processed. ### 7. Consumer Group A **consumer group** is a group of consumers working together. For example: ```plaintext orders topic | v Consumer Group | | v v Consumer Consumer 1 2 ``` Kafka can distribute partitions among consumers in the same group. ![Kafka Implementation in .NET Core](https://answers.mindstick.com/blogs/880e016f-0209-428d-9c54-2896afc1735e/images/167bd026-f34c-4ffb-8fe3-3a4847b4825f.png) ## Setting Up Kafka There are several ways to run Kafka locally. You can use [**Docker**](https://www.mindstick.com/forum/161003/what-is-docker), a local **Kafka installation**, or a managed Kafka service. For learning purposes, Docker is a convenient option. A simple Docker Compose setup can look like this: ```plaintext services: kafka: image: apache/kafka:latest ports: - "9092:9092" ``` Start Kafka with: ```plaintext docker compose up -d ``` The exact Kafka Docker configuration can vary depending on the Kafka version and environment, so for production or a more complete local setup, follow the Kafka documentation for the version you are using. For our .NET application, we will assume Kafka is available at: ```plaintext localhost:9092 ``` ## Creating the .NET Core Project Let's create a simple ASP.NET Core Web API. ```plaintext dotnet new webapi -n KafkaDemo cd KafkaDemo ``` Now install the popular .NET Kafka client: ```plaintext dotnet add package Confluent.Kafka ``` The [Confluent.Kafka package](https://developer.confluent.io/) provides .NET APIs for producing and consuming Kafka messages. ## Creating Our First Kafka Producer Let's say our application needs to publish an order event. Create a model: ```cs // Represents an order in our application. // This class will be used to create the data that we want to send to Kafka. public class Order { // Unique ID of the order. public int Id { get; set; } // Name of the product included in the order. // string.Empty prevents the property from being null by default. public string ProductName { get; set; } = string.Empty; // Price/amount of the order. // decimal is commonly used for monetary values. public decimal Amount { get; set; } } ``` Now let's create a Kafka producer. ```cs // Import the Confluent.Kafka namespace. // This package provides the classes required to work with Kafka in .NET. using Confluent.Kafka; public class KafkaProducer { // Address of the Kafka broker. // localhost means Kafka is running on our local machine. // 9092 is the default Kafka port in our local setup. private readonly string _bootstrapServers = "localhost:9092"; // This method sends a message to a Kafka topic. // 'async' allows us to send the message without blocking the application. public async Task SendMessageAsync(string message) { // Configure the Kafka producer. var config = new ProducerConfig { // Tell the producer where the Kafka broker is running. BootstrapServers = _bootstrapServers }; // Create a Kafka producer using our configuration. // Null means we are not using a message key. // string means the message value will be a string. using var producer = new ProducerBuilder(config).Build(); // Send the message to the "orders" Kafka topic. var result = await producer.ProduceAsync( "orders", new Message { // The actual message that we want to send to Kafka. Value = message }); // Display information about where Kafka stored the message. // TopicPartitionOffset tells us the topic, partition, and offset. Console.WriteLine( $"Message sent to {result.TopicPartitionOffset}"); } } ``` Let's understand what is happening. ### BootstrapServers ```plaintext BootstrapServers = "localhost:9092" ``` This tells the producer where Kafka is running. ### Topic ```plaintext "orders" ``` This is the Kafka topic where we want to send our message. ### Message ```plaintext new Message { Value = message } ``` We are sending a string message. ### ProduceAsync ```plaintext await producer.ProduceAsync(...) ``` This sends the message to Kafka. ## Calling the Producer from an API We can create a controller that sends an order event. ```cs // Marks this class as an API controller. // It enables features such as automatic model validation and API-specific behavior. [ApiController] // Defines the URL route for this controller. // All endpoints in this controller will start with: /api/orders [Route("api/orders")] public class OrdersController : ControllerBase { // Handles HTTP POST requests to: /api/orders // We use POST because we are creating a new order. [HttpPost] public async Task CreateOrder(Order order) { // Create an instance of our Kafka producer. // The producer will be responsible for sending the order to Kafka. var producer = new KafkaProducer(); // Convert the Order C# object into a JSON string. // Kafka will receive this JSON message. // // Example: // { // "id": 101, // "productName": "Laptop", // "amount": 75000 // } var message = JsonSerializer.Serialize(order); // Send the JSON message to the "orders" Kafka topic. await producer.SendMessageAsync(message); // Return a successful HTTP 200 response to the client. return Ok(new { // Message returned to the API caller. message = "Order event sent to Kafka" }); } } ``` Now we can call: ```plaintext POST /api/orders ``` with: ```plaintext { "id": 101, "productName": "Laptop", "amount": 75000 } ``` Our application converts the object into JSON and sends it to the `orders` Kafka topic. ## Creating a Kafka Consumer Now let's build the other side. A consumer reads messages from Kafka. ```cs // Import the Confluent.Kafka namespace. // This provides the classes we need to create and use a Kafka consumer. using Confluent.Kafka; public class KafkaConsumer { // Address of the Kafka broker. // localhost means Kafka is running on our local machine. // 9092 is the default Kafka port in our local setup. private readonly string _bootstrapServers = "localhost:9092"; // Starts the Kafka consumer and listens for messages. public void Start() { // Configure the Kafka consumer. var config = new ConsumerConfig { // Tell the consumer where the Kafka broker is running. BootstrapServers = _bootstrapServers, // Name of the consumer group. // Kafka uses the group ID to keep track of consumed messages. GroupId = "order-consumer-group", // If the consumer doesn't have a previously stored offset, // start reading messages from the beginning of the topic. AutoOffsetReset = AutoOffsetReset.Earliest }; // Create the Kafka consumer using the configuration above. // // Ignore means we are not using a message key. // string means the message value will be received as a string. using var consumer = new ConsumerBuilder(config).Build(); // Subscribe to the "orders" topic. // The consumer will now listen for messages published to this topic. consumer.Subscribe("orders"); // Continuously listen for new Kafka messages. // The loop keeps running until the application is stopped. while (true) { // Wait for a message from Kafka. // This call blocks until a message is available. var result = consumer.Consume(); // Display the message received from Kafka. Console.WriteLine( $"Received: {result.Message.Value}"); } } } ``` There are a few important settings here. ## GroupId ```plaintext GroupId = "order-consumer-group" ``` - This identifies the consumer group. - Kafka uses consumer groups to coordinate which consumers receive which partitions. ## AutoOffsetReset ```plaintext AutoOffsetReset = AutoOffsetReset.Earliest ``` This tells Kafka: > If this consumer group doesn't have a previously stored offset, start reading from the earliest available message. ## Subscribe ```plaintext consumer.Subscribe("orders"); ``` The consumer subscribes to the `orders` topic. ## Consume ```plaintext var result = consumer.Consume(); ``` This waits for a Kafka message. Once a message arrives, we can access it using: ```plaintext result.Message.Value ``` ## Running the Producer and Consumer Now imagine we have two applications: ```plaintext Order API | | Produce v Kafka | | Consume v Order Consumer ``` When we call: ```plaintext POST /api/orders ``` the API sends: ```plaintext { "id": 101, "productName": "Laptop", "amount": 75000 } ``` to Kafka. The consumer receives the same message: ```plaintext Received: {"id":101,"productName":"Laptop","amount":75000} ``` That's our first Kafka-based communication flow. ## A Better ASP.NET Core Approach: BackgroundService The previous consumer example contains an infinite loop: ```plaintext while (true) ``` We normally don't want to start that directly inside an API controller. ASP.NET Core provides `BackgroundService`, which is a much better place for a long-running Kafka consumer. For example: ```cs // BackgroundService is provided by ASP.NET Core. // It allows us to run a long-running task in the background // while our web application is running. public class KafkaConsumerService : BackgroundService { // IConfiguration allows us to read values from // appsettings.json, environment variables, etc. private readonly IConfiguration _configuration; // The configuration object is injected through Dependency Injection. public KafkaConsumerService(IConfiguration configuration) { _configuration = configuration; } // ExecuteAsync is called automatically by ASP.NET Core // when the BackgroundService starts. protected override Task ExecuteAsync( CancellationToken stoppingToken) { // Configure the Kafka consumer. var config = new ConsumerConfig { // Read the Kafka server address from appsettings.json. // Example: localhost:9092 BootstrapServers = _configuration["Kafka:BootstrapServers"], // Read the consumer group name from configuration. // Kafka uses the group ID to manage message consumption. GroupId = _configuration["Kafka:GroupId"], // If this consumer group does not have a saved offset, // start reading messages from the beginning. AutoOffsetReset = AutoOffsetReset.Earliest }; // Create the Kafka consumer using the configuration. // // Ignore means we are not using a Kafka message key. // string means the message value will be received as a string. using var consumer = new ConsumerBuilder(config).Build(); // Subscribe to the "orders" Kafka topic. // The consumer will listen for messages published to this topic. consumer.Subscribe("orders"); try { // Keep consuming messages while the application is running. // // stoppingToken.IsCancellationRequested becomes true // when ASP.NET Core asks the background service to stop. while (!stoppingToken.IsCancellationRequested) { // Wait for a new message from Kafka. // // Passing stoppingToken allows Consume() to stop // gracefully when the application is shutting down. var result = consumer.Consume(stoppingToken); // Display the message received from Kafka. Console.WriteLine( $"Received order: {result.Message.Value}"); } } catch (OperationCanceledException) { // This exception is expected when the application // is shutting down and the cancellation token is triggered. // // Close the consumer gracefully and commit any // required consumer state before shutting down. consumer.Close(); } // The background service has finished. return Task.CompletedTask; } } ``` Then register it in `Program.cs`: ```cs builder.Services.AddHostedService(); ``` Now ASP.NET Core starts the Kafka consumer as a background service. ## Moving Kafka Configuration to appsettings.json Instead of hardcoding: ```plaintext localhost:9092 ``` we can put the configuration in `appsettings.json`. ```plaintext { "Kafka": { "BootstrapServers": "localhost:9092", "GroupId": "order-consumer-group", "OrderTopic": "orders" } } ``` - Then access the values through `IConfiguration`. - This is better because the Kafka server can be different in development, testing, and production. For example: ```plaintext Development -> localhost:9092 Testing -> test-kafka:9092 Production -> production-kafka:9092 ``` ## Sending Objects Instead of Plain Strings In real applications, we usually send structured events rather than random strings. For example: ```cs // Create an anonymous object that represents an order. // This object contains the data that we want to send to Kafka. var order = new { // Unique ID of the order. Id = 101, // Name of the product that was ordered. ProductName = "Laptop", // Total amount of the order. // The value represents 75,000. Amount = 75000, // Store the date and time when the order was created. // UtcNow uses UTC time, which is recommended for // storing timestamps consistently across different servers. CreatedAt = DateTime.UtcNow }; // Convert the C# order object into a JSON string. // Kafka will receive this JSON string as the message. // // Example result: // { // "Id": 101, // "ProductName": "Laptop", // "Amount": 75000, // "CreatedAt": "2026-08-18T10:30:00Z" // } var message = JsonSerializer.Serialize(order); ``` Then send the JSON to Kafka: ```cs // Send the message to the Kafka topic named "orders". await producer.ProduceAsync( "orders", // Create a Kafka message. new Message { // We are not using a message key, so the key is Null. // The actual JSON message is stored in the Value property. Value = message }); ``` The consumer can deserialize it: ```cs // Convert the JSON message received from Kafka // back into our C# Order object. // // result.Message.Value contains the JSON string // that was sent by the Kafka producer. var order = JsonSerializer.Deserialize( result.Message.Value); ``` Now the consumer can work with a normal C# object. ## Using a Kafka Key Kafka messages can also have a key. For example: ```cs // Create a Kafka message with both a key and a value. // // The key helps Kafka determine which partition // should receive the message. var message = new Message { // Convert the order ID to a string and use it as the Kafka key. // // For example: // Order ID 101 -> Key "101" Key = order.Id.ToString(), // Convert the Order object into JSON. // This JSON becomes the actual message value sent to Kafka. // // For example: // { // "Id": 101, // "ProductName": "Laptop", // "Amount": 75000 // } Value = JsonSerializer.Serialize(order) }; ``` The key can be useful when you want messages with the same key to consistently go to the same partition. For example: ```plaintext Order 101 -> Partition 1 Order 102 -> Partition 2 Order 101 -> Partition 1 ``` The exact partition assignment depends on Kafka's partitioning configuration. ## Why Use Kafka in .NET Applications? Kafka becomes particularly useful when your application has multiple services or needs to process a large number of events. For example: ```plaintext +--> Email Service | Order Service --> Kafka --> Inventory Service | +--> Payment Service | +--> Reporting Service ``` - Without Kafka, the Order Service might need to directly call all four services. - With Kafka, the Order Service can publish an event once. - Other services can independently consume that event. - This provides several benefits. - **Loose Coupling** - The producer doesn't need to know all the consumers. - **Scalability** - Consumers can be scaled horizontally. - **Asynchronous Processing** - The producer doesn't have to wait for every downstream operation to finish. - **Durability** - Kafka stores messages according to its retention configuration, allowing consumers to process or replay events according to the application's design. ## Kafka vs RabbitMQ: Beginner's View If you have heard of RabbitMQ, you may wonder how Kafka is different. A simplified way to think about it is: | Kafka | RabbitMQ | | --- | --- | | Event streaming platform | Message broker | | Designed for high-throughput event streams | Strong messaging/routing capabilities | | Messages are stored according to retention policies | Messages are commonly consumed and removed/acknowledged | | Consumers track offsets | Consumers commonly use acknowledgements | | Excellent for event streaming and analytics | Excellent for traditional messaging and routing | The choice depends on your application's requirements. > Kafka isn't automatically better than RabbitMQ, and RabbitMQ isn't automatically better than Kafka. ## Common Beginner Mistakes ## 1. Hardcoding Kafka Configuration Avoid: ```plaintext BootstrapServers = "localhost:9092"; ``` - everywhere in your application. - Use configuration instead. ## 2. Creating a Producer for Every Message This pattern: ```plaintext using var producer = new ProducerBuilder(config).Build(); ``` - for every request isn't a good production design. - A producer is generally intended to be reused. - In an ASP.NET Core application, consider registering a producer as a singleton or wrapping it in a suitable application service. ## 3. Running Consumers in Controllers Don't put: ```plaintext while (true) { consumer.Consume(); } ``` - inside an API action. - Use `BackgroundService` or another appropriate hosted-service architecture. ## 4. Ignoring Error Handling - Kafka operations can fail because of network problems, broker availability, serialization errors, configuration problems, and other issues. - Production applications should have appropriate logging, error handling, retry strategies, and monitoring. ## 5. Not Thinking About Message Contracts If one service sends: ```plaintext { "id": 101, "amount": 1000 } ``` and another service expects: ```plaintext { "orderId": 101, "totalAmount": 1000 } ``` you have a contract problem. Define and manage your event schemas carefully. ## A Simple Real-World Architecture A beginner-friendly Kafka architecture for an e-commerce application could look like this: ```plaintext +----------------+ | Order API | +-------+--------+ | | Publish v +-------------+ | Kafka | | orders | +------+------+ | +------------+------------+ | | | v v v +----------+ +----------+ +----------+ | Email | | Inventory| | Reporting| | Service | | Service | | Service | +----------+ +----------+ +----------+ ``` - The Order API publishes an `OrderCreated` event. - The Email Service consumes it and sends an email. - The Inventory Service consumes it and updates inventory. - The Reporting Service consumes it and updates reporting data. - Each service can evolve independently. ## Conclusion Kafka can seem complicated at first, but the core idea is simple: ```plaintext Producer ---> Topic ---> Consumer ``` In a .NET Core application, the `Confluent.Kafka` library provides the tools needed to communicate with Kafka. A typical application might look like: ```plaintext ASP.NET Core API | | Produce event v Kafka | | Consume event v BackgroundService ``` Once you understand **producer, consumer, topic, partition, offset, and consumer group**, the rest of Kafka becomes much easier to learn. The most important thing for a beginner is not to memorize every Kafka configuration option. Start with one simple flow—**send an order event from a .NET API to Kafka and consume it with a background service**—and then gradually add partitions, consumer groups, retries, serialization, and production-level features. --- Original Source: https://answers.mindstick.com/blog/556/kafka-implementation-in-net-core Copyright © MindStick Software Pvt. Ltd. This Markdown version is provided for developers, AI systems, and offline reading.