Troubleshooting Fivetran Sync Delays: Step-by-Step Guide

Running into Fivetran sync delays? Whether your syncs are failing, stalling, or lagging, you’re not alone. These issues usually occur when data volumes spike, destination limits are hit, or dependent pipelines break upstream.

To resolve them, you’ll need to address API limit restrictions, reduce transformation dependencies, and address latency issues. In this article, we’ll cover these fixes and walk through a step-by-step guide to troubleshooting Fivetran sync delays.

Understanding How Fivetran Syncs Work

Fivetran connectors are pre-built components that connect sources and destinations to sync data. They identify core tables in the source that must be synced to the destination. Once the core tables are identified, Fivetran runs a historical sync, also called the initial sync.

 During the historical sync, all historical data from the core tables in the source is extracted and used to create corresponding schemas in the destination. It also performs transformations. That means the destination doesn’t support certain data types from the source, Fivetran converts them into a supported format. Once schemas and data types are mapped, the data is moved from the source to the destination.

After a successful historical sync, the connector switches to incremental sync. This means only new or modified data since the last sync is updated in the destination at regular intervals. The connector checks for changes such as schema updates, metadata changes, partition shifts, and more, and reflects those changes in the destination.

How frequently Fivetran runs incremental syncs depends on several factors. Some connectors have default intervals. Others may require a higher subscription tier to allow more frequent syncing. You can also configure custom sync intervals. 

There’s also something called adaptive syncing. If your data source hasn’t changed much, Fivetran may reduce sync frequency to save compute. But if changes spike, it syncs more aggressively.  That said, sync delays aren’t always about sync intervals. Network latency, API response times, and destination load times can also contribute.

Common Causes of Fivetran Sync Delays

When you come across delays in your Fivetran data sync pipelines, these are the potential root causes:

1. Huge source changes 

If the source contains numerous data updates and schema changes, sync time increases because there is more data and changes to process and move to the destination.

You can check the Fivetran platform connector logs to identify significant changes in the source. You can also query the destination to see if there were many recent updates made. That helps explain why a sync took longer.

If the amount of data to move increases, sync time naturally increases too. 

2. Latency 

Since Fivetran runs in the cloud, network issues can affect sync speed. So, check the network delays for potential latency issues. Latency can also come from regional differences. For example, if your source is in the US and the destination is in the EU, syncing between regions may introduce delays.

3. API rate limits

Some sources have API rate limits. That means the number of times we can call the API in a given time is restricted to a certain limit. When that limit is reached, Fivetran waits for the API quotas to reset. So the sync process is delayed until the source API allows more requests.

4. Transformation dependencies

Transformation steps run during the sync can introduce delays between the source and destination. Even though Fivetran follows ELT (transformations are applied after the data lands in the destination), the data becomes available downstream only after those dbt models finish running. So complex transformations can cause data access delays for downstream applications.

How to Diagnose Sync Delay Issues?

Continuously monitoring the Fivetran platform connector and logs helps you identify data sync delays. Here’s how you can do it:

Using Fivetran’s sync dashboard

The Fivetran dashboard tracks the status of your data pipelines, configurations, and performance. Navigate to the specific connector in the dashboard and check for suspicious activity, such as increased data volume or schema changes.

You can also check the sync status in the connector tab. If the status shows “Delayed,” check the alerts page for error messages.

Fivetran logs

Fivetran includes a platform connector that delivers logs and account metadata for debugging. It automatically creates a schema in your destination/warehouse and stores sync event details. You can analyze this information to spot errors. 

Fivetran usually retains logs for one week. If you want to access logs for longer, connect to an external log service like AWS CloudWatch, Azure Monitor, Datadog, or Google Cloud Monitoring. You can analyze these logs to identify delay patterns.

Connector runtime

For syncs to run on time, two things need to happen: they must start at the correct interval and complete within the expected runtime. If either fails, sync cycles could result in delays. 

If you’re using the Fivetran Platform Connector, query the `sync_history` table to extract start time, end time, and status details, and calculate the runtime. If it’s increasing, it may indicate a growing data volume. 

Source issues

The issue isn’t always with Fivetran, source issues like slow queries, exceeded API quotas, or network latency can also cause delays. To catch these, use a cloud monitoring tool that tracks and logs key metrics, so you can analyze them and identity the root cause.

Fixing the Delay: Step-by-Step Troubleshooting Guide

Troubleshooting Fivetran sync delays is easier and more effective when you follow this step-by-step practical guide.

Check source health 

No matter how well Fivetran is set up, it can’t pull data if the source system is down. Check if the source is under heavy load, undergoing maintenance, or has lost network access. 

To track the source database health issues like memory usage, network failures, or timeouts, use cloud monitoring tools. These tools continuously monitor system performance and alert you to outages.

Addressing API limit issues

When a connector hits API rate limits, syncs pause and cause delays. In that case, upgrade to a premium tier if available. Sources like Salesforce or Google Ads offer higher API limits on premium plans. Or reduce the sync frequency to stay within limits. 

Also, check the connection with the source API and reauthenticate if access tokens have expired.

Adjusting sync intervals

If sync intervals are too often (like every 15 mins), change it to less aggressive intervals (like every 30 minutes). This is especially useful when the runtime increases due to increased data volumes because longer runtimes can overlap with the next scheduled sync and cause failures.

If you’re running a historical (initial) sync, reduce the volume of data by shortening the sync window. For example, pull data from the last 2 years instead of 5.

Managing large tables and transformations

Massive source tables with poor maintenance can slow down Fivetran syncs. Though Fivetran uses incremental updates, it depends on partitioning, primary keys, or update timestamps. Without these, incremental syncs become difficult.

In that case, break down large tables into small, manageable pieces. Smaller batches run faster and reduce timeout issues. 

Sometimes, data isn’t available downstream even after Fivetran sync. It could be due to dbt models in the middleware. When there are many dependencies, heavy joins, or complex aggregations, it can delay data availability.

To improve performance, use materialized views, simplify joins, and reduce unnecessary model dependencies.

Fivetran support

If you’ve tried all of the above and still face delays, contact the Fivetran support team. Share relevant logs, your findings, and any recent changes made to your setup. Providing clear context helps them understand the issue quickly and work with you to resolve it efficiently.

A Better Solution: Real-time Data Integration with Hevo 

Unlike Fivetran, Hevo supports real-time data streaming for continuous data flow from source to destination. Its event-driven architecture collects data as it’s generated, instead of processing it in batches.

Hevo’s real-time engine is powered by Apache Kafka. When data volumes spike, Kafka buffers and balances the load without dropping events. This setup handles changing data volumes efficiently, eliminating sync delays.

Moreover, Hevo’s stream processing uses in-memory storage to minimize latency and provide fast data access. Fivetran pricing gets expensive with time. Customers also compare different ETL tools and appreciate Hevo’s affordability and real-time data access, so they suggest migrating from Fivetran to Hevo.

FAQs on Troubleshooting Fivetran Sync Delays