Setting up Salesforce Change Data Capture: 7 Easy Steps

With the rise in popularity of cloud-based CRM systems, many organizations now prefer these rather than implementing a custom CRM for their use case. Salesforce is one of the most popular cloud-based CRM systems at this point. A challenge that comes with using these cloud-based CRMs is in integrating them with the internal data systems and reporting modules that rely upon getting up to date data from the CRM.
The implementation architecture which allows real-time synchronization of data between two such systems is called Change Data Capture. Salesforce provides support for Change Data Capture through its streaming events API, and in this post, we will learn about setting up Salesforce CDC in details.

What is Salesforce

Salesforce is a cloud-based customer relationship management software that is offered as a service. It helps organizations manage their leads, process orders and account orders, process invoices, and even support customers without the need for infrastructure maintenance.
Salesforce also provides reporting and Analytics capabilities in its application suite. It also provides an app store for third-party dashboards and tools for custom dashboards. The Salesforce Sales Analytics App uses all the available information to gather valuable insights from the customers.

What is Salesforce Change Data Capture

• Instant Event Notifications: Salesforce Change Data Capture provides instant record change notifications.
• Event Bus: The events are published onto an event bus, on which the clients can listen to a specific channel.
• Data Processing: Customers can process such events to add records to other databases.
• Data Replication: It supports data replication from data changes with the help of external databases.
• Transaction Metadata: The change events are endow with metadata, which are helpful to identify the grouping and ordering of transactions.

Steps to Set up & Subscribe to Salesforce Change Data Capture

Salesforce event messages come in JSON format that contains a schema identifier and a payload. The payload key generally contains the type of operation and value of the fields that have changed. You can set up Salesforce Change Data Capture for your customer data using the following steps:

Step 1: Setting up Change Data Capture

• Step 1.1: Go to the ‘Change Data Capture’ section in the setup tab and search for the object for which Change Data Capture has to be enabled.

• Step 1.2: All the objects including custom objects will be available in this section. In this section let’s work with a custom object sale_item. The object has two fields – name and description. 
• Step 1.3: From the list, select the required table and add it to the list of selected entities for Change Data Capture.

Step 2: Connecting to Java Tool

• Step 2.1: Using a Java tool called emp connector, subscribe to the event and print out the event messages. 
• Step 2.2:To get this tool, first clone the GitHub repository here – git clone  https://github.com/forcedotcom/EMP-Connector.git

Step 3: Building the Jar

java -jar target/emp-connector-0.0.1-SNAPSHOT-phat.jar <username> <password> /data/Employee__ChangeEvent

Step 4: To Access Specific Default Objects

NOTE: If the developer needs access to all the objects, they can use the /data/ChangeEvents URL. 

• Step 4.1: For accessing specific default objects, the URL will be in the below format.

/data/<Object_Name>ChangeEvent

In our case, since our object sale_item is a custom object, we will use the below URL

/data/sale_item__ChangeEvent

Step 5: Subscribe To Events

• Step 5.1: Open the shell and use the below command using the jar file to subscribe to events in the above URL.

java -jar target/emp-connector-0.0.1-SNAPSHOT-phat.jar <username> <password> /data/sale_item__ChangeEvent

NOTE: Here the username and password are the signed-in user credentials. There is also the option to use OAuth based token for authentication. Keep the shell running the above command open for further steps.

Step 6: Creating a New Object 

Now that we have subscribed to the event, let’s create a new sale_item object and see what the event is like. 

• Step 6.1: Go to the App launcher and select sale_item. 
• Step 6.2:Click New and populate the name and description fields. Click Save. 

Step 7: Final Set Up

Go to the terminal where the emp connector is running. You will find messages in the below format.

{   "schema": "-pszPCNGM45tUPU1ftkjxEA",   "payload": {     "LastModifiedDate": "2020-01-25T20:36:12.000Z",     "OwnerId": "005RM000001vI54mYAE",     "CreatedById": "005RM000001vI54mYAE",     "ChangeEventHeader": {       "commitNumber": 65842604581,       "commitUser": "005RM000001vI54mYAE",       "sequenceNumber": 1,       "entityName": "sale_item__c",       "changeType": "CREATE",       "changedFields": [],       "changeOrigin": "com/salesforce/api/soap/47.0;client=SfdcInternalAPI/",       "transactionKey": "00051c2e-a75a-3f97-03fc-cdf4e16d9d3c",       "commitTimestamp": 1569443783000,       "recordIds": [         "a00RM0000114ICTYA2"       ]     },     "CreatedDate": "2020-01-25T20:36:12.000Z",     "name": "IPhone",     "description": "IPhone black with triple camera",     "LastModifiedById": "005RM000001vI54mYAE",   },   "event": {     "replayId": 15053   } }

In the above format changeType field is important since it designates what type of change happened. It can take one among the four values – CREATE, UPDATE, DELETE, UNDELETE.
The transaction key and sequence number signify the transaction which contained the specific commit and order of the commit in that transaction.
The replayId can be used to replay the same event for up to 3 days.

This is how you can set up Salesforce Change Data Capture.

Use Case of Salesforce Change Data Capture

Here is an implementation scenario of Change Data Capture Events where you need to keep your User Interface updated with the latest round of changes:

• User “John Krasinski” is looking at an Opportunity Record which is being simultaneously viewed by the user “Emily Blunt” as well.
• Emily Blunt has modified the record data and amended the Opportunity Amount.
• John Krasinski who is unaware of the change and still looking at the old record has to make a crucial decision based on the Opportunity Amount and send the same to the higher management.
• Now in this situation, John Krasinski may take an incorrect decision and send incorrect data to higher management.

To avoid the aforementioned situation, you can leverage the Change Data Capture events and lightning:empApi (Winter’19 release) element to keep a track of changes in the Opportunity Record and update the data on John Krasinski’s screen as soon as the data is modified by Emily Blunt.

Limitations of the Custom-Code Approach for Salesforce Change Data Capture

• The next step in this process is writing the insert queries to load records in the target database, which usually requires an expert knowledgeable in both Salesforce and the target database. To avoid such complexities, tools like Hevo Data simplify Salesforce Data Integration with databases or data warehouses within a few clicks.
• The client application needs to manage transaction IDs and sequence numbers to obtain a reliable copy. It should be designed to collect messages using the same transaction ID and store only when the ID changes.
• Also, it must check for consistency by sequence numbers and should take into account possible gaps that are missing because of a lack of permissions.
• Additional code is also necessary when data transformation is required before loading into the target database.

Conclusion

This article teaches you how to set up Salesforce Change Data Capture with ease. It provides in-depth knowledge about the concepts behind every step to help you understand and implement them efficiently. These methods, however, can be challenging especially for a beginner & this is where Hevo saves the day. Hevo Data, a No-code Data Pipeline helps you transfer data from a source of your choice such as Salesforce in a fully automated and secure manner without having to write the code repeatedly. Hevo, with its strong integration with 150+ sources & BI tools, allows you to not only export & load data but also transform & enrich your data & make it analysis-ready in a jiffy.

Want to take Hevo for a spin? Get a 14-day free trial and experience the feature-rich Hevo suite firsthand.

What are your thoughts about setting up Salesforce Change Data Capture? Let us know in the comments?

FAQs