How Kafka ETL Simplifies Data Processing from Extraction to Loading

Real-time data is a baseline expectation right now. Apache Kafka has emerged as the backbone of modern data engineering, processing more than one trillion messages per day at companies like LinkedIn, where it was originally built. Organizations now rely on Kafka-based ETL pipelines to move data from dozens of sources into warehouses, lakes, and analytics platforms with sub-second latency.

Yet for all its power, Kafka comes with a steep learning curve. You need to manage brokers, configure topics and partitions, handle schema evolution, and wire up producers and consumers all before a single byte of business data flows anywhere useful. 

Teams often spend weeks on infrastructure before writing transformation logic.

This guide walks you through what Kafka ETL actually looks like in practice: its architecture, the methods you can use to build pipelines, and the best practices that separate fragile proof-of-concepts from production-grade workflows. 

Whether you are building from scratch with Kafka Streams APIs or looking for a faster path with a managed platform, this article gives you the full picture.

Understanding the Kafka Architecture

Before building a Kafka ETL pipeline, it helps to understand the moving parts. Kafka’s distributed architecture is what gives it the ability to handle millions of events per second without data loss. Below is a quick-reference overview of the eight core components, followed by a deeper look at each.

Component	Role	Key Characteristic
Kafka Brokers	Receive, store, and serve messages	Horizontally scalable; cluster of 3+ brokers is standard
Kafka Topics	Named channels that organize message streams	Logical grouping; each topic holds an ordered log of events
Kafka Partitions	Subdivisions of a topic for parallel processing	Enable concurrent reads/writes; each partition is an ordered queue
Kafka Producers	Publish messages to topics	Push-based; can choose partition routing via keys or custom logic
Kafka Consumers	Read and process messages from topics	Pull-based; track offset to resume from last read position
Consumer Groups	Coordinate multiple consumers for load balancing	Each partition is assigned to one consumer per group
Kafka Connect	Pre-built connectors to ingest from/to external systems	Source and sink connectors; no custom code needed
Kafka Streams	Stream processing library built on top of Kafka	Stateful transformations, joins, and aggregations in-flight

Kafka Brokers

A Kafka broker is a server that stores messages and handles requests from producers and consumers. Brokers form a cluster — a group of machines that act as one logical system. When a producer writes a message, it lands on a specific broker responsible for that partition. Consumers pull messages from the same broker.

Kafka clusters typically run a minimum of three brokers to provide fault tolerance. If one broker fails, the others continue serving data from replicated copies. As data volume grows, you add more brokers without taking the cluster offline — this horizontal scalability is one of Kafka’s core design goals.

Kafka Topics

A topic is a named, append-only log where producers write and consumers read. Think of it as a category or feed name — “user-clickstream,” “order-events,” or “sensor-readings.” Producers write to a topic; consumers subscribe to it.

Unlike a traditional message queue, Kafka retains messages for a configurable retention period (default: 7 days) even after they are consumed. This means multiple consumer groups can independently read the same topic at different offsets — an event emitted for real-time alerting can also be consumed later by a batch analytics job from the same topic.

Kafka Partitions

Partitions are how Kafka achieves parallelism. Each topic is split into one or more partitions, each of which is an independent ordered log stored on a broker. Producers assign messages to partitions — either by a message key (consistent hashing), round-robin, or custom logic.

The partition count is the primary lever for throughput: more partitions allow more consumers to read in parallel. A common rule of thumb is to set the number of partitions equal to the expected peak throughput divided by the throughput of a single consumer. Partition count can be increased but not decreased after a topic is created, so it is worth planning ahead.

Kafka Producers

Producers are the clients that publish messages to Kafka topics. A producer serializes data (typically to JSON, Avro, or Protobuf), selects a partition, and sends the message to the leader broker for that partition. Producers can be configured for different reliability guarantees:

• acks=0: Fire-and-forget; fastest but no delivery guarantee.
• acks=1: Leader confirms write; single-broker failure can still cause loss.
• acks=all: All in-sync replicas confirm; slowest but fully durable.

For ETL pipelines where data loss is unacceptable, acks=all with retries enabled is the standard configuration.

Kafka Consumers

Consumers pull messages from Kafka topics. Each consumer maintains an offset — a pointer to the last message it processed. Unlike traditional queues that push messages and delete them after delivery, Kafka consumers control their own pace and can replay messages by resetting their offset.

This pull-based model is particularly useful in ETL contexts: if a downstream system is temporarily unavailable, the consumer simply stops pulling. When the system recovers, it resumes from its last committed offset with no data loss and no need to re-publish messages.

Consumer Group

A consumer group is a set of consumers that work together to process messages from a topic. Kafka assigns each partition to exactly one consumer within the group at a time. This means:

• If you have 6 partitions and 3 consumers in a group, each consumer handles 2 partitions.
• Adding a fourth consumer does nothing until you also add more partitions.
• If one consumer fails, Kafka automatically reassigns its partitions to the remaining active consumers (rebalancing).

Multiple independent consumer groups can subscribe to the same topic — each maintains its own offset and processes messages at its own rate. This is how Kafka supports one event stream feeding several downstream systems simultaneously.

Kafka Connect

Kafka Connect is a framework for building and running connectors that move data between Kafka and external systems without writing custom code. There are two types:

• Source connectors: Pull data from external systems (databases, REST APIs, file systems) and publish it to a Kafka topic.
• Sink connectors: Consume from Kafka topics and push data to external systems (data warehouses, object stores, Elasticsearch).

Confluent Hub alone hosts over 200 ready-made connectors. For most ETL use cases, Kafka Connect eliminates the need to write producer or consumer code from scratch — you configure a connector, define the topic mapping, and it handles the rest.

Kafka Streams

Kafka Streams is a client library for building real-time stream processing applications on top of Kafka. Unlike separate processing frameworks (Flink, Spark Streaming), Kafka Streams runs as a standard Java application — no dedicated cluster required. It reads from input topics, applies transformations, and writes results to output topics.

Key operations include:

• Stateless operations: filter(), map(), flatMap() — transform each record independently.
• Stateful operations: count(), aggregate(), join() — maintain state across a window of records.
• KStream vs KTable: KStream represents an unbounded event stream; KTable represents a changelog (latest value per key), useful for lookups and enrichment.

For Kafka ETL pipelines, Kafka Streams is most useful in the Transform step, filtering bad records, enriching events with reference data, or aggregating metrics before loading to a warehouse.

Prerequisites

• Working knowledge of Kafka.
• A general idea of ETL.
• A general idea of APIs.
• A general idea of Java/Scala.

Methods to Set Up Kafka ETL

There are multiple ways in which you can set up Kafka ETL:

Method 1: Using Kafka Stream APIs to Set Up Kafka ETL

This can be implemented using the following steps:

Step 1: Loading Data into Kafka

To start the export process, you first need to have a Kafka source connector, that will help you bring in data from a variety of data sources. Click here and choose your desired Kafka source connector from a large variety of available connectors such as Kafka Connect Elasticsearch, Kafka Connect Couchbase, etc.

Once you’ve selected your desired connector, you now need to make use of JDBC connector to load data into Kafka in the form of key-value pair messages. 

To do this, you will need to make use of Single Message Transforms (SMTs) and use the following configuration for your Kafka topics:

Open the source connector properties file for your JDBC connector known as source-quickstart-sqlite.properties and append the following lines of code, that will help transform the fetched data by making use of the Single Message Transform functions namely, ValueToKey and Extract Field. This ensures that the data is converted into the correct format before it’s stored as Kafka topics.

# Add the `id` field as the key using Simple Message Transformations
transforms=InsertKey, ExtractId

# `ValueToKey`: push an object of one of the column fields (`id`) into the key
transforms.InsertKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.InsertKey.fields=id

# `ExtractField`: convert key from an object to a plain field
transforms.ExtractId.type=org.apache.kafka.connect.transforms.ExtractField$Key
transforms.ExtractId.field=id

Once you’ve modified the configurations to bring in data as key-value pairs, you need to set your key & value converter to the string and Avro format respectively. You can do this by adding the following lines of code to the JDBC source connector properties files:

# key converter: String (just a bare field)
key.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=false

# value converter: schema’d Avro with pointer to Schema Registry
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schemas.enable=true
value.converter.schema.registry.url=http://schemaregistry1:8081

Kafka Connect will now automatically fetch data from your desired data source such as SQLite3 once the source data changes. It will then store it in the form of key-value pairs in the Kafka topic you’ve created. You can now access data in the as a KSream object directly from your application, with each row, now stored as an Avro record.

# Lookup the schema of the message value for the topic “retail-locations”
$ curl -X GET http://schemaregistry1:8081/subjects/retail-locations-value/versions/1 | jq
{
  "subject": "retail-locations-value",
  "version": 1,
  "id": 2,
  "schema": "{"type":"record","name":"locations","fields":[{"name":"id","type":["null","long"],"default":null},{"name":"name","type":["null","string"],"default":null},{"name":"sale","type":["null","long"],"default":null}],"connect.name":"locations"}"
}

Step 2: Using Kafka Stream APIs to Transform Data

Once you’ve successfully loaded your data into Kafka as a topic, you can now use a variety of applications to access this data and use the schema registry to deserialize the Avro Record. Transform your data for Kafka ETL as follows:

Using Kafka Topics to create Kafka Streams

To create Kafka Stream Objects, you need to convert the byte records of your Kafka topic into a Java object. To do this, create a schema file for your Avro records called “location.avsc”, that will help outline the data structure that matches the clients’ expectations, having three records id, name and sale (example).

Once you’ve created the Avro schema file, you now need a Java class that matches the schema and helps deserialise the Avro records. You can directly import the Java class(ex: Location) using the maven plug-in avro-maven-plugin, found in the pom.xml file. You can import the class using the following lines of code: 

import io.confluent.examples.connectandstreams.avro.Location;

Configure your streams so that the correct serialization/deserialization class is used. To do this, use the following lines of code:

#Define the Serdes for the key
streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
 
#Define the Serdes for the value
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, GenericAvroSerde.class);        
 
#Point to the Schema Registry
streamsConfiguration.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, http://schemaregistry1:8081);

Once you’ve made all the configurations correctly, create the KStream object using the following line of code:

#Build a KStream<String, GenericRecord> from the Kafka topic
KStream<String, GenericRecord> locationsGeneric = builder.stream(“retail-locations”);
 
#Build a KStream<Long, Location> from locationsGeneric
KStream<Long, Location> locationsStream = locationsGeneric.map((k, v) -> new KeyValue<Long, Location>(Long.parseLong(k), new Location ((Long) v.get("id"), (String) v.get("name").toString(), (Long) v.get("sale")) ));

This is how you can create KStream objects to set up streaming Kafka ETL.

Using Kafka Stream Processing Operations to transform the data

Once you’ve created your KStream object (locationsStream), you can now start transforming your data stream, containing key-value pair-based messages.

The Kafka Stream APIs provide various streams processors, which take data records as the input, going one at a time, perform numerous transformations on them and then produces single or multiple outputs. Kafka stream processors can be either stateless and hence transform a particular message at a time or be stateful and perform aggregation operations on various data messages.

Some of the common transformations you can perform using Kafka Stream APIs are as follows:

Sum Values for a Key: This makes use of the reduce method to sum the values of messages that are grouped based on a key. You can do this by using the following command to find the sum of all sales for a given key:

KStream<Long,Long> salesAgg =sales.groupByKey(Serdes.Long(), Serdes.Long()).reduce((aggValue, newValue) -> aggValue + newValue, SALES_STORE).toStream();

Mapping Data Stream to Key-Value pair: You can use the map method and convert a KStream<Long, Location> to KStream<Long, Long> using the following line of code:

KStream<Long,Long> sales = locationsStream.map((k, v) -> new KeyValue<Long, Long>(k, v.getSale()));

This is how you can transform your data using the Kafka Stream APIs stream processors.

Step 3: Loading Data into your desired destination such as S3

Once you’ve transformed & enriched your data, you can now transfer it to a destination of your choice such as S3 using a Kafka sink connector. You can also run multiple sink connectors in parallel and load the same data into various locations. To do this, use a KStream object to write out the information of a particular topic using the following line of code: 

#Write KStream to a topic
salesAgg.to(Serdes.Long(), Serdes.Long(), OUTPUT_TOPIC);

The Kafka sink connector will now transfer your data to the destination, using the downstream system. For example, if you want to load the data into your S3 bucket, you will need to provide your Kafka topic name, S3 region and bucket to configure the sink connector:

$ bin/connect-standalone connect-standalone.properties s3-sink-connector.properties

This is how you can set up streaming Kafka ETL using the Stream APIs.

Limitation of Using Kafka Stream APIs

• Setting up streaming ETL can be a challenging task and requires having a strong knowledge of Java. To make the process easier, here are a few tips to overcome some common ETL challenges. 

Method 2: Using Hevo Data to Set Up Kafka ETL

Step 1: Configure Kafka as a Source

Step 2: Configure your Destination Settings

After configuring Kafka as your source, select the objects you want to ingest, and in the next step, select your Destination. Fill in the required details and click on Save & Continue.

With these simple steps, you have successfully created your pipeline from Kafka to your desired destination.

Method 3: Using Kafka Connect to Set up Kafka ETL

You can build an ETL pipeline with Kafka Connect using the following steps:

Step 1: Preparing data at your desired data source such as MySQL

To begin setting up Kafka ETL, you first need to prepare the data you want to transfer using the Kafka ETL pipeline. For example, if you’re using MySQL as your data source, you can create a table, “customers” and add data to it using the “Insert Into” command:

$ mysql -u root --password="mypassword"
mysql> CREATE DATABASE demodata;
mysql> USE demodata;
mysql> CREATE TABLE customers (
->   id serial NOT NULL PRIMARY KEY,
->   name varchar(100),
->   email varchar(200),
->   department varchar(200),
->   modified timestamp default CURRENT_TIMESTAMP NOT NULL,
->   INDEX `modified_index` (`modified`)
-> );
mysql> INSERT INTO customers (name, email, department) VALUES ('aleesha', 'aleesha@abc.com', 'engineering');
mysql> INSERT INTO customers (name, email, department) VALUES ('bharat', 'bharat@abc.com', 'sales');
mysql> exit;

Once you’ve prepared your data, you now need to start the services required for Kafka Connect and the HDFS cluster using the following command:

$ ./start.sh

This is how you can prepare your data source for setting up Kafka ETL.

Step 2: Ingesting Data into Kafka using Kafka Connect

To ingest data from your desired source, such as MySQL, Kafka makes use of the JDBC connector to bring in the data from the data source and the HDFS connector to load data from Kafka. Kafka ensures that the data remains in Kafka, allowing users to transfer the data to multiple destinations in parallel, such as Elasticsearch.

To start the process, enable Kafka Connect with both connectors, using the following command:

$ connect-standalone /mnt/etc/connect-avro-standalone.properties 
    /mnt/etc/mysql.properties /mnt/etc/hdfs.properties &

The properties, mysql.properties and hdfs.properties, are responsible for controlling how the JDBC and HDFS connectors work, respectively. To know more about them, click on the links given below:

• JDBC Configurations
• HDFS Configurations

Once you’ve enabled Kafka Connect, the JDBC connector will create a new topic called test_jdbc_name. The HDFS connector will now start reading the data from this topic and will transfer it to HDFS, by storing the data in the directory, topics/test_jdbc_customers/. It will also create a new external table in Hive, known as test_jdbc_customers.

You can now use Hive to retrieve the data stored in HDFS, using the following command:

Step 3: Setting up Change Data Capture for your Data Source

With the JDBC connector in place, you can set up the change data capture by configuring the following fields:

• incrementing.column.name:  This field represents the column that will increment.
• timestamp.column.name: This field represents the timestamp column that the JDBC driver will refer to while performing the update.
• Mode: This field represents the mode of change data capture.

The best mode of carrying out change data capture is by using a combination of the timestamp column and incrementing column, as each tuple, containing the timestamp & incrementing (id & timestamp) columns, can be easily identified and updated. Even if the updates happen partially, they can be completed at a later stage.

Once you’ve set up change data capture, whenever the data at your source gets modified, the JDBC connector will automatically apply the same changes to your destination. For example, if you modify the customers’ table in MySQL using the following command:

mysql>UPDATE users SET email = 'aleesha@def.com', modified = CURRENT_TIMESTAMP WHERE name='aleesha';
mysql>UPDATE users SET email = 'bharat@ghi.com', modified = CURRENT_TIMESTAMP WHERE name='bharat';

The JDBC connector will immediately identify the updates and will copy them to Kafka, from where they are then applied to the destination. You can now use a select statement to retrieve the data stored in the customers’ table and see if it was updated or not:

This is how you can set up Kafka ETL and transfer data from a source database such as MySQL to a data warehouse or any other destination.

Step 4: Configuring Schema Migration for HDFS Connector

The HDFS connector allows users to configure the schema they want to use for loading the data into a destination of their choice, with the help of the “schema.compatibility” parameter. You can set this to “None”, “Backward”, “Forward” or “Full” as per your requirement.

When the schema configuration is “Backward”, the HDFS connector keeps track of the most recent schema and only when a record with a larger schema comes, it automatically takes up the new schema and maps all data records to the new schema. Even when a user inserts a new record with the old schema, it automatically maps it to the new schema and then loads the data into Hive.

To implement this, remove the name column from the customers’ table using the following command:

Once the schema change comes into place, Kafka Connect will use the Avro converter to register this schema in the schema registry, and the HDFS connector will automatically refer to the migration mode and make the update before loading the data. 

This is how you can configure schema migration for the HDFS connector.

Limitations of using Kafka Connect

• Deploying custom plugins using Kafka Connect can be challenging, especially for a beginner.
• It requires having an in-depth knowledge of Java /Scala. 
• It can be tough to distinguish between commercial and open-source features.

Best Practices for Kafka ETL Pipeline

A Kafka ETL pipeline that works in a test environment often fails in production under real load, schema changes, or network partitions. These five practices address the most common failure modes and help you build pipelines that are reliable, maintainable, and easy to scale.

1. Design Partitions for Your Target Throughput

Partition count is the single biggest lever on Kafka throughput — and it is nearly impossible to reduce after the fact. Before creating a topic, estimate your peak message rate and divide by the maximum throughput a single consumer can sustain. For example, if your pipeline needs to process 600,000 messages per minute and each consumer handles 60,000 messages per minute, you need at least 10 partitions.

Use message keys consistently. When messages share a key (such as a customer ID or order ID), Kafka routes them to the same partition, guaranteeing ordering within that key. This matters for ETL pipelines that perform stateful joins or aggregations — scrambled ordering produces incorrect results.

2. Enable Replication and Plan for Broker Failure

Set the replication factor to at least 3 for all production topics. With replication factor 3, your pipeline survives a single broker failure without data loss or downtime. Combine this with min.insync.replicas=2 and acks=all on producers: Kafka will only acknowledge a write once two replicas have it, which eliminates the silent data loss that occurs with weaker settings.

Also configure unclean.leader.election.enable=false. Allowing an out-of-sync replica to become the leader after a failure can result in duplicate or missing messages — a risk that is rarely worth accepting in an ETL context.

3. Use Schema Management to Prevent Pipeline Breaks

Schema drift is one of the most common causes of silent data quality failures in Kafka pipelines. A producer adds a new field or renames a column, and downstream consumers start throwing deserialization errors or silently dropping records. Prevent this by enforcing schemas at the topic level.

The most widely adopted approach is Apache Avro with a Schema Registry (such as Confluent Schema Registry). Avro schemas are compact and fast; the registry enforces compatibility rules (backward, forward, or full) so producers cannot publish schema changes that would break existing consumers. For teams already on JSON, JSON Schema validation through the registry provides similar protection without the Avro migration overhead.

4. Apply Message Compression to Reduce Costs and Latency

Kafka supports producer-level compression: lz4, snappy, gzip, and zstd. For most ETL workloads, lz4 delivers the best balance of CPU overhead and compression ratio. Enabling compression reduces both network bandwidth and broker storage — at high message volumes, this translates directly into infrastructure cost savings.

Compression works best when messages are batched together. Set linger.ms to a small value (5–20 ms) on producers to allow a batch to fill before sending. This slightly increases latency at the producer but significantly improves throughput and compression efficiency. For strictly real-time pipelines where every millisecond matters, skip linger.ms and accept lower compression ratios.

5. Monitor Consumer Lag as Your Primary Health Signal

Consumer lag — the difference between the latest offset in a partition and the consumer’s committed offset — is the clearest signal that your ETL pipeline is falling behind. A lag of zero means consumers are keeping up. Rising lag means your consumers cannot process messages as fast as producers generate them.

Set up lag alerts that trigger before lag reaches a critical threshold. Tools like Kafka’s built-in JMX metrics, Confluent Control Center, or open-source alternatives like Burrow and Kafka Lag Exporter (for Prometheus/Grafana) can expose consumer group lag in real time. Beyond lag, track producer throughput, broker disk utilization, and request error rates to catch problems before they become outages.

Conclusion

This article introduces you to the various methods that can be used to set up Kafka ETL. It also provides in-depth knowledge about the concepts behind every step to help you understand and implement them efficiently. These methods, however, can be challenging as they require a deep understanding of Java programming language and other backend tools. This is where Hevo saves the day! To learn more, you can schedule a personalized demo with us.

FAQ on Kafka ETL