AWS Data Migration Service (DMS) is a database migration service provided by Amazon. Using DMS, you can migrate your data from one database to another database. It supports both, homogeneous and heterogeneous database migration. DMS also supports migrating data from the on-prem database to AWS database services.
Using DMS you can migrate your tables from Aurora to Redshift. You need to provide source and target database endpoint details along with schema name. DMS uses replication instance to process the migration task. In DMS, you need to set up a replication instance and provide the source and target endpoint details. Replication instance reads the data from the source and loads the data into the target. This entire processing happens in the memory of replication instance. For migrating high volume of data, it is recommended to use replication instance of higher instance class.
Steps to move data from Aurora to Redshift using AWS DMS:
1. Let us create a table in Aurora (Table name redshift.employee). We will move the data from this table to Redshift using DMS.
2. We will insert some rows in the Aurora table redshift.employee before we move the data from this table to Redshift.
3. Go to DMS service and create a replication instance.
4. Create source and target endpoint and test the connection from the replication instance.
Once both the endpoints are created, it will look as shown below:
5. Once replication instance and endpoints are created, create a replication task. Replication task will take care of your migration of data.
6. Select the table name and schema, which you want to migrate. You can use % as wildcards for multiple tables/schema.
7. Once setup is done, start the replication task.
8. Once the replication task is completed, you can see entire details along with the assessment report.
9. Now, since replication task has completed its activity, let us check the data in Redshift to know whether the data has been migrated.
As shown in the steps above, DMS is pretty handy when it comes to replicating data from Aurora to Redshift but it requires performing a few manual activities.
Pros of moving data from Aurora to Redshift using AWS DMS:
- Data movement is secure as data security is fully managed internally by AWS.
- No database downtime is needed during the migration.
- Replication task setup requires just a few seconds.
- Depending upon the volume of data migration, users can select replication instance type and the replication task will take care of migrating the data.
- You can migrate your data either in Full mode or in CDC mode. In case your replication task is running, change in the data in the source database will automatically reflect in the target database.
- DMS migration steps can be easily monitored and troubleshot using Cloudwatch logs and metrics. You can even generate notification emails depending upon your rules.
- Migrating data to Redshift using DMS is free for 6 months.
Cons of moving data from Aurora to Redshift using AWS DMS:
- While copying data from Aurora to Redshift using AWS DMS, it does not support SCT (Schema Conversion Tool) for your automatically schema conversion which is one of the biggest demerits of this setup. For more details on SCT, kindly refer AWS documentation link.
- Due to differences in features of Aurora database and Redshift database, you need to perform a lot of manual activities for the setup i.e. DMS does not support moving Stored Procedures since in Redshift there is no concept of Stored Procedures, etc.
- Replication instance has a limitation on storage limit. It supports up to 6 TB of data.
- You cannot migrate data from Aurora from one region to another region meaning both Aurora database and Redshift database should be in the same region.
Easier Way to move data from Aurora to Redshift
Using Hevo Data Integration Platform, you can seamlessly transfer data from Aurora to Redshift using 3 simple steps.
- Connect and configure your Aurora database.
- Select the replication mode: (a) load selected Aurora tables (b) load data via Custom Query (c) load data through Binlog.
- For each table in Aurora choose a table name in Redshift where it should be copied.
While you unwind, Hevo will take care of retrieving the data and transferring it to your destination warehouse. Unlike AWS DMS, Hevo provides you with an error-free, fully managed set up to move data in minutes. You can check a detailed article to compare Hevo Vs AWS DMS.
Overall the DMS approach of replicating data from Aurora to Redshift is satisfactory, however, you need to perform a lot of manual activities before the data movement. Few features that are not supported in Redshift has to be handled manually as SCT does not support Aurora to Redshift data movement. In a nutshell, if your manual setup is ready and taken-care you can leverage DMS to move data from Aurora to Redshift. You can also refer to our other blogs where we have discussed Aurora to Redshift replication using AWS Glue and AWS Data Pipeline.