How to Migrate Data from Kinesis Stream to S3?

Do you want to stream your data to Amazon S3? Are you finding it challenging to load your data into your Amazon S3 buckets? If yes, then you’ve landed at the right place! This article will answer all your queries & relieve you of the stress of finding a truly efficient solution. Follow our easy step-by-step guide to help you master the skill of seamlessly setting up the Kinesis Stream to S3 to bring in your data from a source of your choice in real-time!

It will help you take charge in a hassle-free way without compromising efficiency. This article aims at making the data streaming process as smooth as possible.

Upon a complete walkthrough of the content, you will be able to seamlessly transfer your data to Amazon S3 for a fruitful analysis in real-time. It will further help you build a customized ETL pipeline for your organization. Through this article, you will get a deep understanding of the tools & techniques, and thus, it will help you hone your skills further.

Introduction to Amazon S3

Amazon S3 is a cloud storage service offered by AWS that allows users to store and retrieve any amount of data at any time from anywhere on the web. Whether you are a business looking for secure file storage or an individual who needs reliable data backup, Amazon S3 does everything.

Key Features of S3

• Versioning: Maintain multiple versions of an object. Suppose you accidentally delete or modify a file; the versioning feature of S3 allows you to retrieve the older version.
• Cost-Efficient:  Flexible pricing allows you to hold huge amounts of data without ever having to break the bank. In addition, you can opt for multiple storage classes and hold it according to your budget.
• High Durability and Availability: S3 offers high durability(up to 99.999999999%) and availability. Your data is automatically replicated across multiple sources to avoid getting lost.
• Security: With built-in encryption options, access control policies, and integration with AWS Identity and Access Management (IAM) for precise security management, your data remains safe in Amazon S3.
• For further information on Amazon S3, you can check the official website here.

Introduction to Amazon Kinesis

Amazon Kinesis is the powerful, real-time data streaming service offered by AWS. It aggregates, processes, and analyzes large volumes of streaming data in real time. Therefore, whether you are dealing with log files, event data, IoT streams, or social media updates, Amazon Kinesis will help you get insights as the data flows without batch processing.

It houses the support for streaming data to the following storage destinations:

• Amazon S3
• Amazon Redshift
• Amazon ElasticSearch
• Splunk

Key Features of Kinesis

1. Real-Time Data Processing: With Kinesis, you can process and analyze data as it’s produced. Whether you’re monitoring application logs or tracking user activity, Kinesis lets you react to data in the moment.
2. Multiple Data Ingestion Options: It provides multiple data ingestion options like Kinesis Data Streams, Kinesis Data Firehose, etc.
3. Low Latency: Data flows through Kinesis with ultra-low latency, allowing you to act on information the moment it arrives.

For further information on Amazon Kinesis, you can check the official website here.

Understanding Streaming Data

Streaming data refers to the data an application or a source generates in real-time. The volume of this data depends solely upon the data source that helps produce it. It thus varies largely; for example, it can be as high as 20000+ records per second or as low as one record per second. A wide variety of tools available in the market, such as Amazon Kinesis, Apache Kafka, Apache Spark, etc. help capture and stream this data in real-time.

Some examples of streaming data are as follows:

• Web clicks/stream generated by a web application.
• Log files generated by an application.
• Stock market data.
• IoT device data (sensors, performance monitors etc.).

Prerequisites

• Working knowledge of Amazon S3.
• Working knowledge of Amazon Kinesis.
• An Amazon S3 account.

Steps to Set Up the Kinesis Stream to S3

Amazon Kinesis allows users to leverage its Firehose functionality to start streaming data to Amazon S3 from a data producer of their choice in real-time.

You can set up the Kinesis Stream to S3 to start streaming your data to Amazon S3 buckets using the following steps:

• Step 1: Signing in to the AWS Console for Amazon Kinesis
• Step 2: Configuring the Delivery Stream
• Step 3: Transforming Records using a Lambda Function
• Step 4: Configuring Amazon S3 Destination to Enable the Kinesis Stream to S3

Step 1: Signing in to the AWS Console for Amazon Kinesis

• Visit the official AWS console website.
• Sign in using your AWS credentials (username and password).
• If it’s your first time using AWS Kinesis, you’ll see the AWS Kinesis starting page. Click Get Started to begin the setup process.
• On the setup page, you will see four different options for creating an AWS Kinesis stream. Select Deliver Streaming Data with Kinesis Firehose Delivery Streams.

Step 2: Configuring the Delivery Stream

• Configure your delivery stream by choosing the necessary options in the console.
• Provide a unique name for the delivery stream.
• Select Direct PUT or Source as the data source option.

Step 3: Transforming Records using a Lambda Function

• Enable record transformations under the Transform Source Records section.
• Choose or create a Lambda function for processing data.
• Select the General Firehose Processing Lambda blueprint.
• Provide a unique name for the Lambda function and assign an IAM role that grants access to AWS S3 and Firehose.

• Edit the policy by adding the following code:

{
        	"Effect": "Allow",
        	"Action": [
            	"firehose:PutRecordBatch"
        	],
        	"Resource": [
            	"arn:aws:firehose:your-region:your-aws-account-id:deliverystream/your-stream-name"
        	]
}

You will now be able to see the streaming data in the following format:

{"TICKER_SYMBOL":"ABC","SECTOR":"AUTOMOBILE","CHANGE":-0.15,"PRICE":44.89}

• Use the following Lambda function code to transform incoming records:

use strict';
console.log('Loading function');
exports.handler = (event, context, callback) => {
   /* Process the list of records and transform them */
   const output = event.records.map((record) => {
       console.log(record.recordId);
       const payload =JSON.parse((Buffer.from(record.data, 'base64').toString()))
       const resultPayLoad = {
               ticker_symbol : payload.ticker_symbol,
               sector : payload.sector,
               price : payload.price,
           };
          
       return{
           recordId: record.recordId,
           result: 'Ok',
           data: (Buffer.from(JSON.stringify(resultPayLoad))).toString('base64'),
       };
   });
   console.log(`Processing completed.  Successful records ${output.length}.`);
   callback(null, { records: output });
};

Once you’ve made the necessary changes and saved it, you will now be able to see a new lambda function in the delivery stream page.

Step 4: Configuring Amazon S3 Destination to the Enable Kinesis Stream to S3

• Choose Amazon S3 as the destination for your delivery stream.

• Provide the name of your desired S3 bucket.
• Optionally, enable the backup option and provide a backup path.

• Configure the buffer size, interval, encryption, S3 compression, and error logging. Use default values if applicable.
• Assign an IAM role that allows AWS Kinesis to access the Amazon S3 bucket.

Step 5: Testing the Stream

• Verify the streaming process by checking if the data (without the change attribute) appears in your Amazon S3 bucket.
• Open the test demo data node in your delivery stream.
• Click the Start button to initiate the data transfer.

Once you’ve opened the data node, click on the start button to start transferring data to your kinesis delivery stream. In case you want to stop the process, you can click on the stop button. You can now verify the data streaming process by opening a file in your Amazon S3 bucket and checking if the streaming data contains the change attribute.

This is how you can stream your data in real-time by setting up the Kinesis Stream to S3.

Limitations of Manually Setting up Kinesis to S3 Connection

1. Complex Setup: Configuring streams, Lambda, and IAM roles is tedious and prone to errors.
2. Time-Consuming: Setting up each component manually takes a lot of time.
3. Limited Monitoring: Real-time monitoring and debugging require manual effort.
4. Error Handling: Handling failed deliveries or retries needs custom setups.
5. Maintenance Overhead: Regular updates to configurations and permissions are needed.

To overcome these limitations, try Hevo for a simple, reliable, and cost-effective solution. Sign up for Hevo’s free trial and explore seamless data migration.

Conclusion

This article teaches you how to set up the Kinesis Stream to S3 with ease. It provides in-depth knowledge about the concepts behind every step to help you understand and implement them efficiently. These methods, however, can be challenging especially for a beginner & this is where Hevo saves the day.

Hevo Data, a No-code Data Pipeline, helps you transfer data from a source of your choice in a fully-automated and secure manner without having to write the code repeatedly. Hevo, with its strong integration with 150+ Data sources & BI tools, allows you to not only export & load data but also transform & enrich your data & make it analysis-ready in a jiff.

You can also have a look at the unbeatable Hevo Pricing that will help you choose the right plan for your business needs.

FAQ