Amazon S3 REST API Integration: 7 Easy Steps

Using RESTful services has become a great way of incorporating flexibility into the system. With REST APIs, the data remains independent of methods and resources. Using REST API to transfer data to Amazon S3 can help implement different types of calls, relay back multiple data formats, and facilitate easy adaptation to structural requirements. This article explores the concept of a RESTful system and the benefits of the same when used for robust data transfer. In this article, you will read the steps to implement your Amazon S3 REST API Integration.

Pre-Requisites

• Working knowledge of S3 REST APIs.
• Working knowledge of Amazon S3.
• An Amazon S3 account & bucket.

What is REST API?

REST API stands for Representational State Transfer, which presents some constraints of the REST architecture style. An HTTP request is used to access and use the data with commands such as PUT, GET, DELETE, POST, etc. 

A REST API, basically defines how different applications communicate over HTTP.  These different components remain loosely coupled and the transfer of information is fast and carried out efficiently. Since data formats aren’t defined, it serves a more diverse purpose and is easier to develop. Thus, REST APIs are a popular choice among web developers and have emerged as the standard protocol for web-based APIs.

More information regarding the REST API can be found here.

The REST architecture can be understood as follows:

What is Amazon S3?

Amazon S3 (Simple Storage Service) offers public cloud storage available within Amazon Web Services S3. S3 buckets, which are like file folders, can be used to store objects by storing data and its metadata. S3 makes web-scale computing really easy, which is why it is a popular choice among developers. 

When used in tandem with REST API, Amazon S3 transfers can be made more flexible and format independent, which is a requirement for many businesses dealing with huge amounts of data.  This duo can help derive a robust system for data access and management along with the possibility of some elite features like object versioning implemented flexibly with S3 transfers.

More information regarding the Amazon S3 can be found here.

Steps to Set Up Amazon S3 REST API Integration

Using REST API to transfer data to Amazon S3 has its benefits such as; Incorporates flexibility into the system, the data will remain independent of methods and resources used, helps implement different call types and relays back multiple data formats, and facilitates easy structural requirements.

Hevo Data supports both Amazon S3 as a source and REST API as a connector to enable seamless data integration. You can leverage all these benefits by trying Hevo platform with a 14 day free-trail.

REST API can be used in an API gateway as an Amazon S3 proxy to facilitate a RESTful transfer. The API needs to be created and methods need to be exposed to carry out data manipulation. For getting started with the S3 REST API integration process you need to access the API Gateway console in your AWS account.

• Step 1: Create an AWS Account
• Step 2: Declare IAM Permissions for the API
• Step 3: Create and Establish API Resources
• Step 4: Create and Initialize API Method
• Step 5: Expose API Method’s Access to S3 Bucket
• Step 6: Render API Methods to Access an Object in S3 Bucket
• Step 7: Call API via REST API Client

Step 1: Create an AWS Account

Head over to https://portal.aws.amazon.com/billing/signup to create an AWS account and verify your account to get it ready for API integration.

Step 2: Declare IAM Permissions for the API

The next step involves setting up IAM [Identity Access Management] permissions. For the API to use necessary Amazon S3 actions, appropriate IAM role creation and IAM policy attachment need to be carried out and verified.

You can use arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess in the IAM role to declare permissions. This can be implemented as shown:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "*"
        }
    ]
}

The above-represented policy document indicates that all Get* and List*  actions can be invoked from S3 resources.

Similarly, you can include Put* and Post* actions within your policy as implemented below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:Put*",
            "Resource": "*"
        }
    ]
}  

Implement the IAM “Amazon S3 Full Access Policy” within the IAM role to allow the API to create, update, view, and delete data buckets for the manipulation of objects in S3.

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
} 

After declaring the required permissions and policies, the policy should be attached to the IAM role for the API to access the same and “Assume Role” as shown below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "apigateway.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
} 

Step 3: Create and Establish API Resources 

Establishing API resources is essential for your Amazon S3 REST API Integration. You can use the API Gateway console to create an API and specify its resources for a specific Amazon S3 bucket and S3 object. In the API Gateway console, create an API, a child resource “Folder” and define its resource path. Consequently, create an “Item” child resource and define its resource path as shown:

Step 4: Create and Initialize API method 

The next step in setting up your Amazon S3 REST API Integration involves the creation and initialization of an API method for the invoking of API actions. Proceed to configure the GET method to integrate with Amazon S3.

• You can do this by opening the “Actions” menu from the “Resources” panel and choosing “Create method”. Choose GET from the list and start creating the method. Define the resource path s3-host-name/bucket/key and “Use path override” as an Action type. 
• Copy the prior IAM role ARN [Amazon Resource Name] from the console and include it in the “Execution role”. Click “Save” to finish this setup. The GET https://your-api-host/stage/  is now integrated with https://your-s3-host/.
• Enable the IAM to control access to the GET method and declare necessary response types using the “Method Response” box.
• Set up response header mappings for the GET method and integrate the response using integration.response.header.Content-Type.
• Test the GET method on the API root source and ensure that the target S3 bucket is in a region different from the API to avoid potential errors.

Step 5: Expose API Method’s Access to S3 Bucket

Expose GET, PUT and DELETE methods, and map folders to the bucket. Use Method Request and Integration Request to carry out this setup. To test this method before moving on to the last step of your S3 REST API integration, provide the location constraint within the request payload as shown:

<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> 
   <LocationConstraint>{region}</LocationConstraint> 
</CreateBucketConfiguration>

Step 6: Render API Methods to Access an Object in S3 Bucket

To perform the final step of your S3 REST API integration, proceed to expose PUT Object, GET Object, DELETE Object, and HEAD Object operations to allow the API to access objects with the S3 bucket. This can be observed with the additional “item” parameter and implemented as shown:

Step 7: Call API via REST API Client

This is the final step of setting up your Amazon S3 REST API integration. Proceed to call API via a REST API client such as Postman. You need to launch the client and carry out authorization to create a pair of security credentials (keys). 

• Add a bucket,  for instance,  apig-demo-5 to the Amazon S3 account in a specified region. 
• Select PUT method and type the following method URL: https://api-id.execute-api.aws-region.amazonaws.com/stage/folder-name.
• In the Content-type header, set the value as application/xml.
• After choosing the “Body” menu item, implement the following XML request:

<CreateBucketConfiguration> 
  <LocationConstraint>{region}</LocationConstraint> 
</CreateBucketConfiguration>

Click “Send” to submit the request and you will receive a 200 OK response if successful. Implement a PUT request to add a text file to the bucket. For instance, the bucket name (apig-demo-5), filename(Readme.txt), and text string(Hello, World!) are specified to generate a request as shown:

PUT /S3/apig-demo-5/Readme.txt HTTP/1.1
Host: 9gn28ca086.execute-api.{region}.amazonaws.com
Content-Type: application/xml
X-Amz-Date: 20161015T062647Z
Authorization: AWS4-HMAC-SHA256 Credential=access-key-id/20161015/{region}/execute-api/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=ccadb877bdb0d395ca38cc47e18a0d76bb5eaf17007d11e40bf6fb63d28c705b
Cache-Control: no-cache
Postman-Token: 6135d315-9cc4-8af8-1757-90871d00847e
 
Hello, World!

A successful 200 OK response should be received. Extend a GET request to fetch the content of the file added to the apig-demo-5 bucket as shown:

GET /S3/apig-demo-5/Readme.txt HTTP/1.1
Host: 9gn28ca086.execute-api.{region}.amazonaws.com
Content-Type: application/xml
X-Amz-Date: 20161015T063759Z
Authorization: AWS4-HMAC-SHA256 Credential=access-key-id/20161015/{region}/execute-api/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=ba09b72b585acf0e578e6ad02555c00e24b420b59025bc7bb8d3f7aed1471339
Cache-Control: no-cache
Postman-Token: d60fcb59-d335-52f7-0025-5bd96928098a

A successful 200 OK response should be received. 

Make the following request to list items in the specified bucket:

GET /S3/apig-demo-5 HTTP/1.1
Host: 9gn28ca086.execute-api.{region}.amazonaws.com
Content-Type: application/xml
X-Amz-Date: 20161015T064324Z
Authorization: AWS4-HMAC-SHA256 Credential=access-key-id/20161015/{region}/execute-api/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=4ac9bd4574a14e01568134fd16814534d9951649d3a22b3b0db9f1f5cd4dd0ac
Cache-Control: no-cache
Postman-Token: 9c43020a-966f-61e1-81af-4c49ad8d1392

After successful implementation, a 200 OK response will be received with the XML payload indicating a single item in the bucket as shown:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>apig-demo-5</Name>
    <Prefix></Prefix>
    <Marker></Marker>
    <MaxKeys>1000</MaxKeys>
    <IsTruncated>false</IsTruncated>
    <Contents>
        <Key>Readme.txt</Key>
        <LastModified>2016-10-15T06:26:48.000Z</LastModified>
        <ETag>"65a8e27d8879283831b664bd8b7f0ad4"</ETag>
        <Size>13</Size>
        <Owner>
            <ID>06e4b09e9d...603addd12ee</ID>
            <DisplayName>user-name</DisplayName>
        </Owner>
        <StorageClass>STANDARD</StorageClass>
    </Contents>
</ListBucketResult>

Thus, you can successfully transfer linked items to the S3 bucket using the REST API. 

More information regarding this API can be found here.

Data paths and JSON values need to be used accurately for the proper implementation of these steps as shown. Here are some code snippets from an example that implements the above-described method to sample REST API as an Amazon S3 proxy:

{
  "swagger": "2.0",
  "info": {
    "version": "2016-10-13T23:04:43Z",
    "title": "MyS3"
  },
  "host": "9gn28ca086.execute-api.{region}.amazonaws.com",
  "basePath": "/S3",
  "schemes": [
    "https"
  ],
  "paths": {
    "/": {
      "get": {
        "produces": [
          "application/json"
        ],
        "responses": {
          "200": {
            "description": "200 response",
            "schema": {
              "$ref": "#/definitions/Empty"
            },
            "headers": {
              "Content-Length": {
                "type": "string"
              },
              "Timestamp": {
                "type": "string"
              },
              "Content-Type": {
                "type": "string"
              }
            }
          },
 
"security": [
          {
            "sigv4": []
          }
        ],
        "x-amazon-apigateway-integration": {
          "credentials": "arn:aws:iam::<replaceable>123456789012</replaceable>:role/apigAwsProxyRole",
          "responses": {
            "4d{2}": {
              "statusCode": "400"
            },
            "default": {
              "statusCode": "200",
              "responseParameters": {
                "method.response.header.Content-Type": "integration.response.header.Content-Type",
                "method.response.header.Content-Length": "integration.response.header.Content-Length",
                "method.response.header.Timestamp": "integration.response.header.Date"
              }
            },
            "5d{2}": {
              "statusCode": "500"
            }
          },
          "uri": "arn:aws:apigateway:us-west-2:s3:path//",
          "passthroughBehavior": "when_no_match",
          "httpMethod": "GET",
          "type": "aws"
        }
      }
    },

With this step, your Amazon S3 REST API Integration is complete!

Learn about AWS API Gateway Integration: A Comprehensive Guide.

Limitations of Amazon S3 REST API Integration using Manual Method

The above method of setting up Amazon S3 Rest API Integration although feasible, comes with certain limitations which are as follow:

• You will have to add/remove/change the new properties associated with your connections when any change occurs, to ensure smooth data transfers and high efficiencies for your S3 RESt API integration.
• Working with APIs to load data requires you to have the strong technical knowledge to handle operations such as connection pooling, query optimization, compression, validation, etc. Then only you will be able to set up the S3 REST API integration.

Conclusion

Thus, the much-needed flexibility and efficiency can be added to your Amazon S3 data transfer mechanism using REST APIs. This step-wise guide can be implemented to use set up the Amazon S3 REST API Integration, where the REST APIs act as an Amazon S3 proxy and facilitate data transfer. The configured system can enhance various functionalities within the REST API configuration. 

The manual approach of setting up the S3 REST API integration will add complex overheads in terms of time and resources. Such a solution will require skilled engineers and regular data updates. Furthermore, you will have to build an in-house solution from scratch if you wish to transfer your data from S3 or REST APIs to a Data Warehouse for analysis.

Hevo Data provides an Automated No-code Data Pipeline that empowers you to overcome the above-mentioned limitations. Hevo has connectors to 150+ data sources (including 40+ free sources). Furthermore, Hevo’s fault-tolerant architecture ensures a consistent and secure transfer of your data to a Data Warehouse. Using Hevo will make your life easier and make Data Transfer hassle-free.

Learn more about Hevo

Want to take Hevo for a spin? Sign up for a 14-day free trial and experience the feature-rich Hevo suite firsthand.

Share your experience of this blog and your understanding of the Amazon S3 Rest APIs in the comments section!