Apache Kafka is one of the most popular distributed event streaming platforms. Thousands of well-known companies use it, including LinkedIn, Airbnb, Goldman Sachs, Oracle, PayPal, Salesforce, Shopify, Spotify, and many others. It's no surprise that it gets very exciting when someone claims to have built a better Kafka cloud experience, making it up to 10 times faster!
At the end of 2020, Vectorized released a new modern open-source event streaming platform called Redpanda. It's faster than Kafka, ZooKeeper-free, JVM-free, and, most importantly, Kafka-compatible. You can find a complete list of differences here.
Having the same APIs as Kafka allows reaping all Redpanda benefits with no code or tooling changes. It just works! This simplicity is a considerable cost-saver, making the transition to Redpanda a very attractive option. No wonder why more and more companies are starting to use it.
Trying It Out
It only takes a few minutes to set up Redpanda for testing purposes.
Running Redpanda on Docker Compose
Let's create a docker-compose.yml file with the following content:
version: '3.4'
services:
redpanda:
command:
- redpanda
- start
- --smp
- '1'
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- '0'
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
image: docker.vectorized.io/vectorized/redpanda:v21.9.5
container_name: redpanda-1
ports:
- 9092:9092
- 29092:29092To run Redpanda, we need to execute the following console command in the directory:
docker-compose up -dStreaming messages
First, we need to create a topic:
docker exec -it redpanda-1 rpk topic create messages-topic --brokers=localhost:9092Let's produce a message by executing the following console command, typing text into the console, and pressing CTRL + D:
docker exec -it redpanda-1 rpk topic produce messages-topic --brokers=localhost:9092Let's consume the message by running the following console command:
docker exec -it redpanda-1 rpk topic consume messages-topic --brokers=localhost:9092Trying Kafka client libraries
Let's create a .NET worker service using the most popular Kafka client library for .NET - Confluent.Kafka. This service will produce a new message to the previously created topic every second while simultaneously consuming and printing messages from the same topic.
using Confluent.Kafka;
namespace WorkerTest;
public class Worker : BackgroundService
{
private const string Topic = "messages-topic";
private const string BootstrapServers = "localhost:9092";
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var consumer = CreateAndStartConsumer(stoppingToken);
using var producer = CreateProducer();
while (!stoppingToken.IsCancellationRequested)
{
var message = new Message<string, string>
{
Key = Guid.NewGuid().ToString("N"),
Value = $">>> {DateTimeOffset.Now}"
};
await producer.ProduceAsync(Topic, message, stoppingToken);
await Task.Delay(1000, stoppingToken);
}
}
private static IConsumer<string, string> CreateAndStartConsumer(CancellationToken token)
{
var config = new ConsumerConfig
{
GroupId = "test-consumer",
BootstrapServers = BootstrapServers,
AutoOffsetReset = AutoOffsetReset.Earliest
};
var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe(Topic);
Task.Run(() =>
{
while (!token.IsCancellationRequested)
{
var cr = consumer.Consume(token);
Console.WriteLine(cr.Message.Value);
consumer.Commit(cr);
}
}, token);
return consumer;
}
private static IProducer<string, string> CreateProducer()
{
var config = new ProducerConfig
{
BootstrapServers = BootstrapServers
};
return new ProducerBuilder<string, string>(config).Build();
}
}Let's run the service by executing the following console command:
dotnet runThis is what our service console looks like:
We didn't write any Redpanda-specific code and could use Kafka client libraries without issues.
Thank you for reading. Maybe you are already using Redpanda? I'd love to hear about your experiences with Redpanda in the comments.
