Redpanda - A New Cool Kid on the Event Streaming Block

Apache Kafka is one of the most popular distributed event streaming platforms. Thousands of well-known companies use it. Among these are LinkedIn, Airbnb, Goldman Sachs, Oracle, PayPal, Salesforce, Shopify, Spotify, and many others. It's no surprise that it gets very exciting when someone claims to have built a better Kafka cloud experience making it up to 10 times faster!

At the end of 2020, Vectorized released a new modern open-source event streaming platform called Redpanda. It's faster than Kafka, ZooKeeper-free, JVM-free, and, most importantly, Kafka-compatible. You can find a complete list of differences here.

Having the same APIs as Kafka allows reaping all Redpanda benefits with no code or tooling changes. It just works! This simplicity is a considerable cost-saver making the transition to Redpanda a very attractive option. No wonder why more and more companies are starting to use it.

Trying it out

It only takes a few minutes to set up Redpanda for testing purposes.

Running Redpanda on Docker Compose

Let's create a docker-compose.yml file with the following content:

version: '3.4'

services:
  redpanda:
    command:
    - redpanda
    - start
    - --smp
    - '1'
    - --reserve-memory
    - 0M
    - --overprovisioned
    - --node-id
    - '0'
    - --kafka-addr
    - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
    - --advertise-kafka-addr
    - PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
    image: docker.vectorized.io/vectorized/redpanda:v21.9.5
    container_name: redpanda-1
    ports:
    - 9092:9092
    - 29092:29092

To run Redpanda, we need to execute the following console command in the directory:

docker-compose up -d

Streaming messages

First, we need to create a topic:

docker exec -it redpanda-1 rpk topic create messages-topic --brokers=localhost:9092

Let's produce a message by executing the following console command, typing text into the console, and pressing CTRL + D:

docker exec -it redpanda-1 rpk topic produce messages-topic --brokers=localhost:9092

Let's consume the message by running the following console command:

docker exec -it redpanda-1 rpk topic consume messages-topic --brokers=localhost:9092

Trying Kafka client libraries

Let's create a .NET worker service using the most popular Kafka client library for .NET - Confluent.Kafka. This service will produce a new message to the previously created topic every second while simultaneously consuming and printing messages from the same topic.

using Confluent.Kafka;

namespace WorkerTest;

public class Worker : BackgroundService
{
    private const string Topic = "messages-topic";
    private const string BootstrapServers = "localhost:9092";

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var consumer = CreateAndStartConsumer(stoppingToken);
        using var producer = CreateProducer();

        while (!stoppingToken.IsCancellationRequested)
        {
            var message = new Message<string, string>
            {
                Key = Guid.NewGuid().ToString("N"),
                Value = $">>> {DateTimeOffset.Now}"
            };
            await producer.ProduceAsync(Topic, message, stoppingToken);
            await Task.Delay(1000, stoppingToken);
        }
    }

    private static IConsumer<string, string> CreateAndStartConsumer(CancellationToken token)
    {
        var config = new ConsumerConfig
        {
            GroupId = "test-consumer",
            BootstrapServers = BootstrapServers,
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
        var consumer = new ConsumerBuilder<string, string>(config).Build();
        consumer.Subscribe(Topic);

        Task.Run(() =>
        {
            while (!token.IsCancellationRequested)
            {
                var cr = consumer.Consume(token);
                Console.WriteLine(cr.Message.Value);
                consumer.Commit(cr);
            }

        }, token);

        return consumer;
    }

    private static IProducer<string, string> CreateProducer()
    {
        var config = new ProducerConfig
        {
            BootstrapServers = BootstrapServers
        };
        return new ProducerBuilder<string, string>(config).Build();
    }
}

Let's run the service by executing the following console command:

dotnet run

This is what our service console looks like:

We didn't write any Redpanda specific code and could use Kafka client libraries without issues.

Thank you for reading. Maybe you are already using Redpanda? I'd love to hear about your experiences with Redpanda in the comments.