Working with Kafka Elasticsearch Connector Simplified: 4 Easy Steps

In partnership with Elastic, we have made it easier than ever to connect always-on, always-moving data flowing through Confluent to Elastic’s real-time search and analytics database. With our seamless integrations, organizations can set their data in motion for a new generation of use cases that were not possible before.

Table of Contents

– Jay Kreps, co-founder, and CEO, Confluent

Confluent and Elastic are thrilled to collaborate to make Apache Kafka Elasticsearch integration easier than ever. This allows enterprises to stream data from Kafka into Elasticsearch in real-time, enabling log analysis, full-text search, and more. The Elasticsearch Service Sink Connector in Confluent Cloud eliminates the need for clients to manage their own Kafka clusters.  This allows businesses to stream data from Kafka to Elasticsearch across all major cloud providers, including Amazon Web Services (AWS), Microsoft Azure, and Google Cloud.

This article will walk you through the key features of Elasticsearch and Kafka. You will learn more about Kafka Elasticsearch Service Sink Connector and discuss its various features. However, this article focuses on helping you to understand the key steps involved to stream your data from Kafka to Elasticsearch. So, read along to gain insights into Kafka Elasticsearch integration and how you can leverage it to streamline your Kafka Elasticsearch workflows.

Prerequisites

To successfully set up your Kafka Elasticsearch Service Sink Connector ensure that you meet the following requirements:

• Confluent Platform 3.3.0 or above.
• A Kafka cluster on the Confluent Cloud. If you haven’t set it up, then refer to this Quick Start Guide.
• Authorized access to a Confluent Cloud cluster.
• Java version 1.8.
• Elasticsearch version 7.x.
• The Elasticsearch Service and Confluent Cloud should be set to the same region.

Introduction to Elasticsearch

Elasticsearch is an Apache Lucene-based distributed Search and Analytics engine. It has swiftly become the most popular search engine since its introduction in 2010. It is frequently used for Log Analytics, full-text Search, Security Intelligence, Business Analytics, and a lot of other applications.

Elasticsearch is a NoSQL database, open-source, and written in Java. It is available for both On-Premises and Cloud. You have the option of running it yourself or using a hosted Elasticsearch service like AWS Elasticsearch. You can also use Elasticsearch to aggregate info as well as get insight into trends and patterns in your data. You can also integrate it with BI tools such as Power BI, Looker, and Tableau. Elasticsearch makes use of an Inverted Index data structure, which allows it to execute extremely quick real-time full-text searches. 

Although Elasticsearch is primarily a search engine, customers began to leverage it for logs and wished to conveniently ingest and view them. The primary components of the Elastic stack, known as ELK, are Elasticsearch, Logstash, and Kibana is widely used in the industry.

Key Features of Elasticsearch

Elasticsearch’s distributed nature, speed, scalability, and ability to index any document make it suitable for practically any application. Let’s explore these features a bit more:

• Seamless and Fast: Elasticsearch employs schema-free JSON documents, supports simple REST-based APIs, and uses a basic HTTP interface, making it simple to get started and quickly create applications for a range of use-cases.
• High Performance: Elasticsearch’s distributed nature allows it to analyze massive amounts of data in parallel, allowing it to quickly identify the best matches for your searches.
• Automatic Node Recovery: If a Node departs the Elasticsearch Cluster for any reason, such as Node failure, the Master Node takes the required steps to manage the load by replacing the Node with its replica and rebalancing all Shards.
• High Scalability: Elasticsearch’s distributed architecture allows it to scale up to thousands of servers and handle petabytes of data without slowing down. It takes care of this distributed design automatically, allowing customers to concentrate on their core business.
• Extensive Tools and Plugins: Elasticsearch integrates seamlessly with Kibana, a prominent visualization, and reporting tool. It also integrates with Beats and Logstash, allowing you to quickly convert and load source data into your Elasticsearch cluster. To add additional functionality to your applications, you can leverage a variety of open-source Elasticsearch plugins including language analyzers and suggesters.
• Easy Application Development: Elasticsearch supports a wide range of languages, including Java, Python, PHP, JavaScript, Node.js, Ruby, and several others.

Interested in reading more about Elasticsearch, visit Elasticsearch HomePage.

Introduction to Kafka

Apache Kafka is a distributed Event Streaming platform that enables applications to handle massive volumes of data in a short amount of time. Its fault-tolerant, highly scalable architecture is capable of managing billions of events. The Apache Kafka framework is a distributed Publish-Subscribe Messaging system that receives Data Streams from many sources and is implemented in Java and Scala. It also allows you to analyze real-time Big Data streams.

Kafka’s ability to withstand peak data input volumes is a distinct and strong advantage. It can simply and quickly scale up and down with minimal downtime. Kafka’s popularity among other Data Streaming technologies has grown as a result of its minimal data redundancy and fault tolerance.

Key Features of Kafka

Apache Kafka is incredibly popular because of its features, which include ensuring uptime, making scaling simple, and allowing it to manage massive volumes. Let’s have a look at some of the powerful features it provides:

• High Scalability: The partitioned log model used by Kafka distributes data over several servers, allowing it to extend beyond the capacity of a single server.
• Low Latency: Kafka separates data streams, resulting in extremely low latency and great throughput.
• Fault-Tolerant & Durable: Data is written to disc, and partitions are distributed and replicated across several servers. This protects data from server failure and makes it fault-tolerant and long-lasting. The Kafka cluster can handle failures in the master and database. It’s capable of restarting the server on its own. 
• Extensibility: Since Kafka’s surge in popularity in recent years, many other applications have built connectors. This enables the installation of extra features, such as integration with other systems, in a matter of seconds. Check out how you can integrate Kafka with Amazon Redshift and Salesforce.
• Metrics and Monitoring: Kafka is a popular tool for tracking operational data. This requires gathering data from several apps and consolidating it into centralized feeds with metrics. To read more about how you can analyze your data in Kafka, refer to Real-time Reporting with Kafka Analytics. 

Key Features of the Elasticsearch Kafka Connector

The Kafka Elasticsearch Sink Connector makes it easy to link Apache Kafka and Elasticsearch. You can stream data from Kafka into Elasticsearch to perform Log Analysis or full-text Search. You can also benefit from being able to do real-time Analytics or integrate it with other platforms like Kibana. So, let’s take a look at some of the robust features offered by Kafka Elasticsearch Sink Connector.

• Easy Mapping Management
• Seamless Schema Management
• Effective Multitasking
• Supports Exactly Once Delivery
• Prevents Thundering Herd
• Secure Connection

1) Easy Mapping Management

Connect schemas allow the Kafka Elasticsearch Connector to infer mappings. When the connection is activated, it generates mappings based on Kafka message schemas. The inference is confined to field types and default values if a field is absent. If you require further adjustments, you need to manually construct mappings.

2) Seamless Schema Management

The Kafka Elasticsearch Connector can handle backward, forward, and other compatible schema updates in Connect. It also facilitates schema evolution. When Elasticsearch discovers a previously unknown field in a document, it implements dynamic mapping to identify the data type for the field and adds the new field to the type mapping automatically.

3) Effective Multitasking

To increase throughput, the Kafka Elasticsearch Connector enables batching and pipelined writes to Elasticsearch. It collects messages in batches and allows many batches to be processed simultaneously. The number of tasks in the tasks can be specified. The maximum configuration option. Hence, when numerous files need to be processed, this can result in significant performance benefits.

4) Supports Exactly Once Delivery

Elasticsearch’s idempotent write semantics ensures that the Kafka Elasticsearch Connector performs exactly-once delivery to Elasticsearch. The connector guarantees precise delivery by specifying IDs in Elasticsearch documents. If keys are provided in Kafka messages, they are immediately converted to Elasticsearch document IDs. However, if the keys aren’t specified or are explicitly ignored, the connector uses topic+partition+offset as the key, guaranteeing that each message in Kafka corresponds to precisely one document in Elasticsearch.

5) Prevents Thundering Herd

If the Elasticsearch service is inadvertently overloaded, the Kafka Elasticsearch connector may be unable to write to the Elasticsearch endpoint. The connection will retry the request several times before failing and may employ an exponential backoff mechanism to give the Elasticsearch service time to recover, thus, preventing additional overloading. This approach adds jitter to the computed backoff timings to prevent a thundering herd, which occurs when a high number of requests from several jobs are made at the same time, thereby, overloading the service.

6) Secure Connection

By configuring the Kafka Elasticsearch connection, the Elasticsearch connector can write data to a secure Elasticsearch cluster that supports basic authentication. By following the steps in the Kafka Connect Security guidelines, you can securely streamline your data from Kafka to Elasticsearch.

Steps to Set Up ElasticSearch Kafka Connection

The Kafka Elasticsearch Service Sink Connector for Confluent Cloud helps you to seamlessly move your data from Kafka to Elasticsearch. It supports many data outputs from the Kafka topics such as Avro, JSON Schema, Protobuf, or JSON-schemaless. It publishes data to an Elasticsearch index from a Kafka topic. 

In this section, you will learn the key steps to configure your Kafka Elasticsearch Service Sink Connector using Confluent Cloud Console. So, get started with adding a connector and configuring it to stream events to an Elasticsearch deployment by following the steps listed below.

• Step 1: Add the Kafka Elasticsearch Service Sink Connector
• Step 2: Set Up Kafka Elasticsearch Connection
• Step 3: Launch the Kafka Elasticsearch Service Sink Connector
• Step 4: Check Kafka Elasticsearch Connection Status

Step 1: Add the Kafka Elasticsearch Service Sink Connector

The first step is to add the connector to the Confluent Cloud. Follow the steps below:

• Launch your Kafka cluster you created on Confluent Cloud.
• Now navigate to “Data Integration → Connectors” and click on “+ Add Connector”.
• Next, click on the “Elasticsearch Service Sink” connector as shown below.

Step 2: Set Up Kafka Elasticsearch Connection

After adding the Kafka Elasticsearch Service Sink Connector, follow the steps below to set up the connection.

• Firstly choose the Kafka Topics and enter a name for the connector.
• Next, choose one of the options provided to enter your Kafka Cluster credentials. You can provide these either through service account resource ID or enter the API key and secret.
• Now, select the input message format and provide the correct Elasticsearch connection details. You will be required to enter your connection URI, Elasticsearch deployment credentials, and other details. 
• After entering all the required information, enter the number of tasks you can create.

Step 3: Launch the Kafka Elasticsearch Service Sink Connector

Now that you have configured your Kafka Elasticsearch Connection, you can now launch the Kafka Elasticsearch Service Sink Connector. Before you launch it, make sure that to verify all your connection details carefully. Click “Launch” once you have verified the details. An example of connection details can be seen below.

Step 4: Check Kafka Elasticsearch Connection Status

Before you move your data from Kafka cluster to Elasticsearch, make sure you check that the Kafka Elasticsearch Connection is synced properly. The status of the successful connection will change from Provisioning to Running as seen below.

Good work! You have successfully configured your Kafka Elasticsearch Service Sink Connector in Confluent Cloud using the Confluent Cloud Console. Now you can move your Kafka data to Elasticsearch easily. If you wish to set up your Kafka Elasticsearch Service Sink Connector in Confluent Cloud using Confluent CLI, you can refer to the following Documentation.

Limitations of Elasticsearch Kafka Connection

Kafka Elasticsearch Service Sink Connector in Confluent Cloud was introduced in the previous year. Hence, the Kafka and Confluent teams are working to improve them. However, this connector does have a few limitations. These are listed below:

• The connector is only compatible with Elastic Cloud’s Elasticsearch Service.
• Elasticsearch version 7.1 and later are supported by the connector. It does not support Elasticsearch version 8.x.
• The target Elasticsearch deployment and the Confluent Cloud cluster must be in the same region.
• Single Message Transformations (SMTs) that change the topic name are presently not supported by the Kafka Elasticsearch Sink Connector. SMTs alter incoming messages after they’ve been produced by a source connection but before they’re published to Kafka. Outbound messages are transformed by SMTs before being transmitted to a sink connection. 
• The following transformations are also not allowed by the Kafka Elasticsearch Connector:
  • org.apache.kafka.connect.transforms.TimestampRouter
  • io.confluent.connect.transforms.MessageTimestampRouter
  • io.confluent.connect.transforms.ExtractTopic$Key
  • io.confluent.connect.transforms.ExtractTopic$Value

Conclusion

This article helped you gain knowledge about Elasticsearch and Kafka. You explored the key features of these services and also learned the capabilities of Kafka Elasticsearch Service Sink Connector. Moreover, you understood the key steps to configure your Kafka Elasticsearch Connector in Confluent Cloud. You can also use the Elasticsearch Connector for your Confluent On-Premise platform. You can refer to Elasticsearch Service Sink Connector for Confluent Platform documentation.

As a Developer, you might have faced challenges while moving your Kafka data to other destinations like Elasticsearch. You can check out an easier yet robust way – Hevo to move your data from Kafka, Elasticsearch to other sources. 

Hevo Data is a No-Code Data Pipeline that offers a faster way to move data from 100+ Data Sources including Apache Kafka, Kafka Confluent Cloud, Elasticsearch and other 40+ Free Sources, into your Data Warehouse to be visualized in a BI tool. You can use Hevo Pipelines to replicate the data from your Apache Kafka Source or Kafka Confluent Cloud to the Destination system. Hevo is fully automated and hence does not require you to code. 

VISIT OUR WEBSITE TO EXPLORE HEVO

Want to take Hevo for a spin?

SIGN UP and experience the feature-rich Hevo suite first hand. You can also have a look at the unbeatable pricing that will help you choose the right plan for your business needs.

Share your experience of setting up Kafka Elasticsearch Connector and moving your data from Kafka to Elasticsearch with us in the comments section below!