Setting up Salesforce Change Data Capture: 7 Easy Steps

With the rise in popularity of cloud-based CRM systems, many organizations now prefer these rather than implementing a custom CRM for their use case. Salesforce is one of the most popular cloud-based CRM systems at this point. A challenge that comes with using these cloud-based CRMs is in integrating them with the internal data systems and reporting modules that rely upon getting up to date data from the CRM. The implementation architecture which allows real-time synchronization of data between two such systems is called Change Data Capture. Salesforce provides support for Change Data Capture through its streaming events API, and in this post, we will learn about setting up Salesforce CDC in details.

Table of Contents

• What is Salesforce
• What is Salesforce CDC
• Steps to Set up & Subscribe to Salesforce Change Data Capture
• Salesforce Change Data Capture Implementation Example
• Limitations of the Custom-Code Approach for Salesforce Change Data Capture
• Conclusion

What is Salesforce

Salesforce is a cloud-based Customer Relationship Management software that is offered as a service. It helps organizations to manage their leads, process orders, account orders, process invoices, and even in supporting the customers. What makes Salesforce popular is the fact it completely absolves the organizations of doing anything to maintain the infrastructure and software. Salesforce also provides reporting and Analytics capabilities in its application suite.

Beyond the default dashboards and reports, Salesforce also provides an application store from where customers can download dashboards and reports created by third-party developers. If that still does not meet your requirements, it also has facilities for implementing custom reports and dashboards. The Salesforce Sales Analytics App uses all the available information to gather valuable insights from the customers.

Check out the official website of Salesforce for more insights.

What is Salesforce CDC

Salesforce Change Data Capture allows one to receive instant event notifications of changes happening to Salesforce records. These messages are sent to the event bus to which clients can subscribe by specifying the required channel. The client application can then subscribe to these events and perform the required processing to insert the records into an external database. 

Data replication from Salesforce to another database typically involves creating a copy of the complete database and subscribing to these events to frequently sync the changing data. Salesforce change events understand the concept of transactions and have enough metadata for the client to identify the grouping and order of the changes that are part of a single transaction. 

In case you are looking to specifically load data to cloud data warehouses like Redshift, Snowflake or BigQuery or to databases such as PostgreSQL or MySQL, click on the links to explore detailed guides to move data to the respective targets.

Steps to Set up & Subscribe to Salesforce Change Data Capture

Salesforce event messages come in JSON format that contains a schema identifier and a payload. The payload key generally contains the type of operation and value of the fields that have changed. You can set up Salesforce Change Data Capture for your customer data using the following steps:

Step 1: To set up Change Data Capture on an object, the developer has to go to the ‘Change Data Capture’ section in the setup tab. Once in the Change Data Capture section, search for the object for which Change Data Capture has to be enabled. All the objects including custom objects will be available in this section. In this section let’s work with a custom object sale_item. The object has two fields – name and description. From the list select the required table and add it to the list of selected entities for Change Data Capture.

Step 2: Once this is done, we need to subscribe to the event channel. For this, Salesforce provides an easy to use Java tool called emp connector that can subscribe to the event and print out the event messages. To get this tool first clone the github repository here – git clone  https://github.com/forcedotcom/EMP-Connector.git

Step 3: Use the below command to build the jar:

java -jar target/emp-connector-0.0.1-SNAPSHOT-phat.jar <username> <password> /data/Employee__ChangeEvent

Step 4: Salesforce events can be accessed by subscribing to a specific URL. In case the developer needs access to all the objects, they can use the /data/ChangeEvents URL. For accessing specific default objects URL will be in the below format.

/data/<Object_Name>ChangeEvent
 
For accessing change data events on custom objects the below format is to be used.

 
/data/<Object_Name>__ChangeEvent
 
In our case, since our object sale_item is a custom object, we will use the below URL

 
/data/sale_item__ChangeEvent

Step 5: Open the shell and use the below command using the jar file to subscribe to events in the above URL.

java -jar target/emp-connector-0.0.1-SNAPSHOT-phat.jar <username> <password> /data/sale_item__ChangeEvent

Here the username and password are the signed-in user credentials. There is also the option to use OAuth based token for authentication. Keep the shell running the above command open for further steps.

Step 6: Now that we have subscribed to the event, let’s create a new sale_item object and see what the event is like. Go to the App launcher and select sale_item. Click New and populate the name and description fields. Click Save. 

Step 7: Go to the terminal where the emp connector is running. You will find messages in the below format.

{   "schema": "-pszPCNGM45tUPU1ftkjxEA",   "payload": {     "LastModifiedDate": "2020-01-25T20:36:12.000Z",     "OwnerId": "005RM000001vI54mYAE",     "CreatedById": "005RM000001vI54mYAE",     "ChangeEventHeader": {       "commitNumber": 65842604581,       "commitUser": "005RM000001vI54mYAE",       "sequenceNumber": 1,       "entityName": "sale_item__c",       "changeType": "CREATE",       "changedFields": [],       "changeOrigin": "com/salesforce/api/soap/47.0;client=SfdcInternalAPI/",       "transactionKey": "00051c2e-a75a-3f97-03fc-cdf4e16d9d3c",       "commitTimestamp": 1569443783000,       "recordIds": [         "a00RM0000114ICTYA2"       ]     },     "CreatedDate": "2020-01-25T20:36:12.000Z",     "name": "IPhone",     "description": "IPhone black with triple camera",     "LastModifiedById": "005RM000001vI54mYAE",   },   "event": {     "replayId": 15053   } }

In the above format changeType field is important since it designates what type of change happened. It can take one among the four values – CREATE, UPDATE, DELETE, UNDELETE. The transaction key and sequence number signify the transaction which contained the specific commit and order of the commit in that transaction. The replayId can be used to replay the same event for up to 3 days.

This is how you can set up Salesforce Change Data Capture.

Let’s now look at some of the challenges that a typical developer will face while setting up Salesforce Change Data Capture manually.

Salesforce Change Data Capture Implementation Example

Here is an implementation scenario of Change Data Capture Events where you need to keep your User Interface updated with the latest round of changes:

• User “John Krasinski” is looking at an Opportunity Record which is being simultaneously viewed by the user “Emily Blunt” as well.
• Emily Blunt has modified the record data and amended the Opportunity Amount.
• John Krasinski who is unaware of the change and still looking at the old record has to make a crucial decision based on the Opportunity Amount and send the same to the higher management.
• Now in this situation, John Krasinski may take an incorrect decision and send incorrect data to higher management.

To avoid the aforementioned situation, you can leverage the Change Data Capture events and lightning:empApi (Winter’19 release) element to keep a track of changes in the Opportunity Record and update the data on John Krasinski’s screen as soon as the data is modified by Emily Blunt.

Limitations of the Custom-Code Approach for Salesforce Change Data Capture

• The logical next step in this flow is to use the above feed and write the required insert queries to insert the records into the target database. And, in most cases that is easier said than done, because it needs an expert developer who knows the intricacies of Salesforce as well as the target database. A way to avoid such a learning curve is to use an easy-to-implement solution like Hevo Data which can help set up Salesforce Data Integration with numerous databases or data warehouses in just a few clicks.
• To create a proper replica that has guaranteed consistency, the client application code must contain logic to handle the transaction id and sequence number. The logic should monitor the transaction identifier of each message and add all the messages in the same transaction to a container that is saved to the target database once the transaction identifier changes. The sequence number must be verified to ensure the order. The sequence number may have gaps in them in case there are objects that the user does not have permission to access.
• In case you need to transform the data that is extracted from Salesforce before moving to the target database or data warehouse – you will need to write additional code to achieve this.

Conclusion

This article teaches you how to set up Salesforce Change Data Capture with ease. It provides in-depth knowledge about the concepts behind every step to help you understand and implement them efficiently. These methods, however, can be challenging especially for a beginner & this is where Hevo saves the day. Hevo Data, a No-code Data Pipeline helps you transfer data from a source of your choice such as Salesforce in a fully automated and secure manner without having to write the code repeatedly. Hevo, with its strong integration with 150+ sources & BI tools, allows you to not only export & load data but also transform & enrich your data & make it analysis-ready in a jiffy.

Visit our Website to Explore Hevo

Want to take Hevo for a spin? Sign Up for a 14-day free trial and experience the feature-rich Hevo suite first hand.

What are your thoughts about setting up Salesforce Change Data Capture? Let us know in the comments?