Top 15 Powerful Confluent Connectors in 2023

The Confluent Platform is a set of infrastructure services, tools, and standards that helps your company stream data easily.  It lets you focus on how to derive business value from your data rather than worrying about the underlying mechanics of how data is managed between various systems. Confluent’s 120+ pre-built connectors enable customers to easily migrate to new cloud-native systems and applications such as AWS, Azure, Google Cloud, Snowflake, and many more.

In this article, you will discover the 15 best Confluent Connectors. You will understand more about Confluent, its key features, and why you need Confluent Connectors. So, read along and gain insights about the powerful Confluent Connectors in the market and explore their key benefits.

Table of Contents

• What is Confluent?
  • Key Features of Confluent
• What are Confluent Connectors?
  • Features of Confluent Connectors
• Importance of Confluent Connectors
• Kafka Connect in Distributed Mode
  • Step 1: Starting Kafka Connect
  • Step 2: Adding Connectors Using the REST API
• Best Confluent Connectors in the Market
  • Confluent Open Source / Community / Partner Connectors
    • Debezium SQL Server CDC Source Connector
    • Elasticsearch Sink Connector
    • Google BigQuery Sink Connector
    • JDBC Connector (Source and Sink)
    • MongoDB Connector (Source and Sink)
    • Snowflake Sink Connector
  • Confluent Commercial Connectors
    • Amazon Redshift Sink Connector
    • Amazon S3 Source Connector
    • Databricks Delta Lake Sink Connector for AWS
    • Datadog Metrics Sink Connector
    • Google Firebase Realtime Database Connector (Source and Sink)
    • MQTT Connector (Source and Sink)
    • Salesforce (Bulk API) Connector (Source and Sink)
  • Confluent Premium Connectors
    • Oracle CDC Source Connector
    • Splunk S2S Source Connector
• Benefits of Confluent Connectors
• Conclusion

What is Confluent?

Confluent is a full-featured data streaming platform that lets you effortlessly access, store, and manage data as real-time streams. It extends the advantages of Apache Kafka with enterprise-grade functionality while minimizing the burden of Kafka maintenance and monitoring.  Over 80% of the Fortune 100 companies use data streaming technology today, and the bulk of them use Confluent.

Confluent makes it simple to develop a whole new category of event-driven applications, build a universal data pipeline, and unleash powerful use cases with complete scalability, productivity, and reliability by combining historical and real-time data into a single, central source of truth.

Instead of worrying about the underlying mechanics, such as how data is delivered or integrated between separate systems, Confluent Platform allows you to focus on how to create business value from your data. In particular, it makes it easier to connect data sources to Kafka, develop streaming applications, and secure, monitor, and manage your Kafka infrastructure.

Confluent’s software consists of 3 main services: a free, Open-Source streaming platform that makes it simple to get started with real-time data streams; an Enterprise-grade version with more administration, operations, and monitoring tools; and a Premium Cloud-based version.

Key Features of Confluent

Let’s explore some of the powerful features of Confluent that make it so popular among organizations.

• Multi-Region Clusters: The inclusion of multi-region clustering allows a Confluent client to extend a single Confluent Platform cluster over many regions. This enhances the deployed product’s Disaster Recovery ability.
• Schema Validation: Kafka employs schemas to organize data and guarantee that it is usable in the future, just like its database counterparts. Customers define their schemas in the Kafka Schema Registry, which allows them to change them over time. Customers can now enforce schemas within the platform directly at the Kafka broker, which is located in the Confluent Server on the Confluent Platform.
• Role-Based Access Control: RBAC allows operators to specify policies at a higher level rather than having to manage access control lists (ACLs) for each Kafka user for each Kafka function. To allow or prohibit user access to Kafka resources, RBAC uses current Active Directory and Lightweight Directory Access Protocol (LDAP) implementations.
• Tiered Storage: Tiered Storage enables Kafka to distinguish between 2 types of storage: local discs and cost-effective object stores like Amazon S3 and Google Cloud Storage. Brokers can now dump data to object storage as it matures on the local disk while maintaining performant access to the data. It allows for unlimited data storage and dynamic scalability.
• Self-Balancing Clusters: Self-Balancing Clusters immediately detect the existence of new brokers or the removal of existing brokers while scaling the platform up or down and trigger a subsequent partition reassignment. This allows you to add and delete brokers with ease, making your Kafka clusters even more flexible.
• Cluster Linking: It makes Kafka deployments in hybrid and multi-cloud environments easier. Cluster Linking, rather than siloing specific clusters, guarantees that Kafka data is available wherever it is needed by replicating data without the need for extra nodes in the design.

Visit the Confluent HomePage to explore other key features.

What are Confluent Connectors?

Confluent Kafka Connectors are the pre-built ready-to-use components of the Confluent that allow users to import data from external data source or systems into Kafka topics and also aids in exporting data from Kafka topics to external systems. There are many Confluent Connectors available that help users access data. 

The source connector collects data from a system, and these source systems include Databases, stream tables, or message brokers.

The Sink connector delivers data from Kafka topics to other systems such as Elastisearch.

Features of Confluent Connectors

Some of the main features of Confluent Connectors are listed below:

• Distributes and Standalone Modes: Kafka allows users to deploy clusters by using Kafka, as well as setups for development, testing, and small production deployments.
• Confluent Connectors offers a framework to connect external systems with Apache Kafka to simplify the deployment, development, and management of connectors.
• Confluent Connectors comes with REST Interface support that allows users to manage connectors using REST APIs.
• Kafka Connect uses default group management protocols and allows us to add more workers to scale up the Kafka Connect clusters.

Importance of Confluent Connectors

Confluent Cloud provides Apache Kafka Connectors that are pre-built and fully managed. These connectors make it simple to connect to popular data sources and sinks. Confluent Connectors make transferring data in and out of Kafka easy with a simple GUI-based configuration and elastic scalability with no infrastructure to manage, allowing you more time to focus on application development. Let’s see why you need Confluent Connectors:

• Build a Robust Tech Stack: Confluent Connectors support both traditional and new cloud-based systems. This allows businesses to upgrade their technological stack by allowing them to connect to popular data sources and sinks in real-time. Confluent Connectors offer excellent customer experiences through data-driven backend operations. This is crucial for businesses competing in a digital-first environment to innovate and succeed.
• Boost Team Productivity: Each Confluent Connector may save up to approximately 3–6 engineering months of development and maintenance efforts on average. This allows your Developers to focus on developing and maintaining business-critical applications rather than establishing and maintaining basic infrastructure tools and integrations.
• Accelerate & Secure Integrations: Confluent Connectors allow Developers to connect to common data sources and sinks quickly, consistently, and securely right out of the box. They ensure data interoperability and governance, as well as granular access to particular connectors through Role-Based Access Control (RBAC). Furthermore, Confluent Connectors are Kafka-native and include industry-leading, enterprise-grade support to help you prevent downtime and data loss.

If you’re a Kafka Developer or Architect, you have 3 basic choices for integrating useful data sources and sinks. These are:

• Personalized Connectors using Kafka Connect: The time, effort, and resources needed to design, create, test, and maintain each connector are the drawbacks of this strategy. Depending on the complexity of the connector, this process might take 3–6 engineering months, not to mention the technical debt incurred in terms of new bespoke tools for your team to manage and support.
• Open Source Connectors: The disadvantage of this strategy is that these connectors are not subject to enterprise-level vendor testing and maintenance, which is unsuitable for the vast majority of mission-critical applications. If you work for a company that uses unsupported connectors in production, you can’t afford to take this risk if you want to avoid downtime, data loss, and business interruption.
• Pre-Built Connectors: Confluent features 120+ ready-to-use connectors for the most commonly used data sources and sinks. These Confluent Connectors have been created & tested by Kafka experts and they come with industry-leading support.

Kafka Connect in Distributed Mode

To make full use of full use of distributed nature of Kafka, you need to launch Kafka Connect in distributed mode. Kafka topics store metadata and connector settings instead of file systems. The following steps to launch Kafka Connect in distributed mode are listed below:

Step 1: Starting Kafka Connect

A reference of the configuration for distributed mode is available at “$CONFLUENT_HOME/etc/kafka/connect-distributed.properties” directory. 

For this, mostly all the parameters are the same as in Standalone mode, the few differences are listed below:

• The group.id defines the name of the Cluster group and it must be different from any consumer group.
• The offset.storage.topic, config.storage.topic and status.storage.topic defines the topics for settings. 

You can start Kafka Connect in distributed mode by using the following command given below: 

$CONFLUENT_HOME/bin/connect-distributed $CONFLUENT_HOME/etc/kafka/connect-distributed.properties

Step 2: Adding Connectors Using the REST API

You need to send 2 POST requests to http://localhost:8083/connectors containing the following JSON structs.

For this, first, you need to create the body for the source connector POST as a JSON file. You can name it connect-file-source.json. 

{
    "name": "local-file-source",
    "config": {
        "connector.class": "FileStreamSource",
        "tasks.max": 1,
        "file": "test-distributed.txt",
        "topic": "connect-distributed"
    }
}

POST the file using the command given below:

curl -d @"$CONFLUENT_HOME/connect-file-source.json" 
  -H "Content-Type: application/json" 
  -X POST http://localhost:8083/connectors

Let’s do the same for the sink connector, calling the file connect-file-sink.json. 

{
    "name": "local-file-sink",
    "config": {
        "connector.class": "FileStreamSink",
        "tasks.max": 1,
        "file": "test-distributed.sink.txt",
        "topics": "connect-distributed"
    }
}

Again, POST this file.

curl -d @$CONFLUENT_HOME/connect-file-sink.json 
  -H "Content-Type: application/json" 
  -X POST http://localhost:8083/connectors

You can also verify if the setup is working correctly by using the following command given below.

$CONFLUENT_HOME/bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-distributed --from-beginning
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}

Now, navigate to the $CONFLUENT_HOME folder and you will see the file test-distributed.sink.txt. Open it and you can see the output as shown below.

cat $CONFLUENT_HOME/test-distributed.sink.txt
foo
bar

After we tested the distributed setup, let’s clean up, by removing the two connectors:

curl -X DELETE http://localhost:8083/connectors/local-file-source
curl -X DELETE http://localhost:8083/connectors/local-file-sink

Best Confluent Connectors in the Market

Confluent Hub is an online marketplace that allows you to quickly explore, search, and filter for Confluent Connectors and other Kafka plugins that best suit your data mobility needs. Confluent offers 120+ Pre-Built Connectors. All the Confluent Connectors can be categorized into the following 3 categories:

• Confluent Open Source / Community / Partner Connectors
  • Debezium SQL Server CDC Source Connector
  • Elasticsearch Sink Connector
  • Google BigQuery Sink Connector
  • JDBC Connector (Source and Sink)
  • MongoDB Connector (Source and Sink)
  • Snowflake Sink Connector
• Confluent Commercial Connectors
  • Amazon Redshift Sink Connector
  • Amazon S3 Source Connector
  • Databricks Delta Lake Sink Connector for AWS
  • Datadog Metrics Sink Connector
  • Google Firebase Realtime Database Connector (Source and Sink)
  • MQTT Connector (Source and Sink)
  • Salesforce (Bulk API) Connector (Source and Sink)
• Confluent Premium Connectors
  • Oracle CDC Source Connector
  • Splunk S2S Source Connector

1) Confluent Open Source / Community / Partner Connectors

Debezium SQL Server CDC Source Connector

The Debezium SQL Server connector captures row-level changes in a SQL Server database’s schemas. When the Debezium SQL Server connector connects to a SQL Server database or cluster for the first time, it takes a consistent snapshot of the database’s schemas. Following the initial snapshot, the connector constantly records row-level changes for INSERT, UPDATE, and DELETE operations committed to SQL Server databases that have been enabled for CDC. For each data change action, the connector generates events and sends them to Kafka topics. The connector sends all of a table’s events to a separate Kafka topic. Data change event records from that topic can then be consumed by applications and services.

Refer to Confluent Connectors: Debezium SQL Server CDC Source Connector to know more.

Elasticsearch Sink Connector

The Elasticsearch Sink Connector Data moves data from Apache Kafka to Elasticsearch.  It publishes data from an Apache Kafka topic to an Elasticsearch index. In Elasticsearch, every data for a topic has the same type, this enables the development of schemas for data from various topics independently. Elasticsearch can be leveraged for text searches, analytics, and a key-value store. This connector supports both the Analytics and Key-Value Store use cases.

Want to learn how can load integrate Elasticsearch with Kafka Confluent, read the Working with Kafka Elasticsearch Connector Simplified Guide to know more.

Google BigQuery Sink Connector

Data can be easily streamed into BigQuery tables using the Kafka Connect Google BigQuery Sink connector. This connector can build BigQuery tables with the proper BigQuery table schema when streaming data from Kafka topics with registered schemas. The BigQuery table structure is based on information from the topic’s Kafka schema.

Refer to Confluent Connectors: Google BigQuery Sink Connector to know more.

JDBC Connector (Source and Sink)

You can use the  JDBC Source connector to import data into Kafka topics from any relational database that has a JDBC driver. On the other hand, the JDBC Sink connector allows you to export data from Kafka topics to any relational database that supports the JDBC driver. The JDBC connector can connect to a broad range of databases without requiring any special code. Data is loaded by running a SQL query and producing an output record for each row in the result set on a regular basis. All tables in a database are transferred to their own output topic by default. The database is constantly checked for new or deleted tables.

Refer to Confluent Connectors: JDBC Connector to know more.

MongoDB Connector (Source and Sink)

The MongoDB Kafka connector is a Confluent-verified connector that publishes updates from MongoDB into Kafka topics as a data source and retains data from Kafka topics as a data sink into MongoDB.

Refer to Confluent Connectors: MongoDB Connector to know more.

Snowflake Sink Connector

Snowflake is a Data Warehouse designed specifically for the Cloud. The Snowflake Kafka connector allows you to rapidly and efficiently transport messages from Kafka topics into Snowflake tables in formats such as Avro, JSON, and Protobuf. In the Kafka setup, topics may be mapped to existing Snowflake tables. If the topics aren’t mapped, the Confluent Connector generates a new table for each one, naming it after the topic.

Refer to Confluent Connectors: Snowflake Sink Connector to know more.

2) Confluent Commercial Connectors

Amazon Redshift Sink Connector

You can export data from Apache Kafka topics to Amazon Redshift using the Kafka Connect Amazon Redshift Sink Connector. Data from Kafka is polled by the connector, which then uploads it to an Amazon Redshift database. The polling data is based on topics that have been subscribed to. Table auto-creation and limited auto-evolution are both supported by this connector.

Refer to Confluent Connectors: Amazon Redshift Sink Connector to know more.

Amazon S3 Source Connector

The Amazon S3 Source connector can read data from any sort of file naming convention specified under an S3 bucket. The filenames don’t have to be in a specific format. The connector will be able to read files that are in any of the supported formats (for example, JSON, Avro, and Byte Array).

Refer to Confluent Connectors: Amazon S3 Source Connector to know more.

Databricks Delta Lake Sink Connector for AWS

The Databricks Delta Lake Sink connector polls data from Kafka periodically. It then transfers the data to an Amazon S3 staging bucket and then commits the records to a Databricks Delta Lake instance.

Refer to Confluent Connectors: Databricks Delta Lake Sink Connector for AWS to know more.

Datadog Metrics Sink Connector

Datadog Metrics Connector using the Post time series API, exports data from Kafka topics to Datadog. The connector receives a Struct as the value of a Kafka record, which includes fields for name, timestamp, and values. The metrics value is contained in the values field. This connector can start with just one job to handle all data exports and scale horizontally as more tasks are added.

Refer to Confluent Connectors: Datadog Metrics Sink Connector to know more.

Google Firebase Realtime Database Connector (Source and Sink)

Users can read data from a Google Firebase Realtime Database and store it in Kafka topics using the Google Firebase Source connector. If users want to read data from numerous Kafka topics and publish it to Google Firebase Realtime Database then they can use the Google Firebase Sink connector.

Refer to Confluent Connectors: Google Firebase Realtime Database Connector to know more.

MQTT Connector (Source and Sink)

The MQTT connector is used to connect to existing MQTT servers. The Kafka Connect MQTT Source connector establishes a connection to a MQTT broker and subscribes to the topics specified. On the other hand, the Kafka Connect MQTT Sink connector establishes a connection with a MQTT broker and sends data to a MQTT topic. Both MQTT source and sink connector support SSL.

Refer to Confluent Connectors: MQTT Connector to know more.

Salesforce (Bulk API) Connector (Source and Sink)

Salesforce Bulk API Connector connects Salesforce.com and Apache Kafka. The Salesforce Bulk API Source connector allows you to use the Salesforce Bulk Query API to extract records and updates from Salesforce.com. This connector works with both standalone and distributed Connect workers.

The Salesforce Bulk API Sink connector uses records from Kafka topics to conduct CRUD actions (insert, update, delete) on Salesforce SObjects and publishes them to Salesforce. The Salesforce PushTopic Source connector can then write these records to a Kafka topic.

Refer to Confluent Connectors: Salesforce (Bulk API) Connector to know more.

3) Confluent Premium Connectors

Oracle CDC Source Connector

The Confluent Oracle CDC Source Connector is a Premium Confluent connector that requires a specific subscription. Changes in an Oracle database are captured via the Oracle CDC Source Connector and sent to Kafka topics as change event records.

The connector reads the database’s redo log with Oracle LogMiner and requires extra logging with “ALL” columns. Oracle 11g, 12c, 18c, and 19c are supported via the connector. It works with both container and non-container databases, as well as databases on-premises and in the cloud. 

Refer to Confluent Connectors: Oracle CDC Source Connector to know more.

Splunk S2S Source Connector

Splunk S2S Source connector allows you to connect Splunk to Kafka. The Splunk universal forwarder sends data to the connector. Only 1 job can be launched at a time using this connector  This pre-built, well-designed premium connector allows you to deliver log data to Splunk and other downstream applications and data systems in a cost-effective and efficient manner.

Refer to Confluent Connectors: Splunk S2S Source Connector to know more.

These are just a few of the Confluent Connectors that are widely used in the market. Do you want to explore other connectors? Visit the Confluent Connectors Portfolio to discover them.

Benefits of Confluent Connectors

Organizations can effortlessly integrate and analyze data from their business sources with Confluent Connectors allowing more tailored customer experiences and greater operational efficiencies. Let’s have a look at the different benefits that Confluent Connectors provide.

• 120+ Pre-Built Connectors: You can leverage out-of-the-box, expert-certified Confluent Connectors to quickly connect to vital organizational applications, databases, and other data management systems.
• Fully Managed & Hosted: You can utilize Confluent’s completely managed service on Confluent Cloud to relieve your resources of the operational burdens of managing your connectors.
• Data Compatibility & Governance: Maintaining the right data schemas ensures data consistency. Confluent Connectors ensure that all of your integrated data is in the same standardized format so that you can scale across numerous business lines with confidence.
• Expert-Built & Tested: Rest confident that every connector, whether built or approved by Confluent, has been rigorously reviewed and tested by Kafka specialists. You’ll save 6 engineering months by not having to design, create, test, and manage a single custom connector.
• Enterprise-Level Support: You have access to professional support 24/7 for faster issue resolution.  Confluent’s professionals are available to help with not only your Confluent connector needs, but any data in motion needs throughout your whole platform.
• Industry-Leading Security: Role-Based Access Control simplifies enterprise-scale security by approving access to particular connectors for individuals or teams, thereby lowering the risk of security incidents and breaches.

Conclusion

In a nutshell, you gained a basic understanding of Confluent and its key features. You understood the importance of Confluent Connectors and their key benefits. Moreover, you discovered the 10 robust Confluent Connectors available on Confluent Hub. If you are searching for Kafka compatible Connectors, check out the best Kafka Connectors here.

However, when streaming data from various sources to Confluent or vice versa, you might face various challenges. If you are facing these challenges and are looking for some solutions, then check out a simpler alternative like Hevo.

Hevo Data is a No-Code Data Pipeline that offers a faster way to move data from 100+ Data Sources including Apache Kafka, Kafka Confluent Cloud, and other 40+ Free Sources, into your Data Warehouse to be visualized in a BI tool. You can use Hevo Pipelines to replicate the data from your Apache Kafka Source or Kafka Confluent Cloud to the Destination system. Hevo is fully automated and hence does not require you to code. 

VISIT OUR WEBSITE TO EXPLORE HEVO

Want to take Hevo for a spin?

SIGN UP and experience the feature-rich Hevo suite first hand. You can also have a look at the unbeatable pricing that will help you choose the right plan for your business needs.

Feel free to share your experience with Confluent Connectors with us in the comments section below!