Connecting Elasticsearch to S3: 4 Easy Steps

Are you trying to derive deeper insights from your Elasticsearch by moving the data into a larger Database like Amazon S3? Well, you have landed on the right article.

This article will give you a brief overview of Elasticsearch and Amazon S3. You will also get to know how you can set up your Elasticsearch to S3 integration using 4 easy steps. Moreover, the limitations of the method will also be discussed in further sections. Read along to know more about connecting Elasticsearch to S3 in the further sections.

Note: Currently, Hevo Data doesn’t support S3 as a destination.

What is Elasticsearch?

Elasticsearch accomplishes its super-fast search capabilities through the use of a Lucene-based distributed reverse index. When a document is loaded to Elasticsearch, it creates a reverse index of all the fields in that document.

A reverse index is an index where each of the entries is mapped to a list of documents that contains them. Data is stored in JSON form and can be queried using the proprietary query language.

Elasticsearch has four main APIs – Index API, Get API, Search API, and Put Mapping API:

• Index API is used to add documents to the index.
• Get API allows to retrieve the documents and Search API enables querying over the index data.
• Put Mapping API is used to add additional fields to an already existing index. 

The common practice is to use Elasticsearch as part of the standard ELK stack, which involves three components – Elasticsearch, Logstash, and Kibana:

• Logstash provides data loading and transformation capabilities.
• Kibana provides visualization capabilities.

Together, three of these components form a powerful Data Stack. 

Behind the scenes, Elasticsearch uses a cluster of servers to deliver high query performance.

An index in Elasticsearch is a collection of documents.

Each index is divided into shards that are distributed across different servers. By default, it creates 5 shards per index with each shard having a replica for boosting search performance.

Index requests are handled only by the primary shards and search requests are handled by both the shards. 

The number of shards is a parameter that is constant at the index level.

Users with deep knowledge of their data can override the default shard number and allocate more shards per index. A point to note is that a low amount of data distributed across a large number of shards will degrade the performance.

Amazon offers a completely managed Elasticsearch service that is priced according to the number of instance hours of operational nodes. 

To know more about Elasticsearch, visit this link.

What is Amazon S3?

AWS S3 is a fully managed object storage service that is used for a variety of use cases like hosting data, backup and archiving, data warehousing, etc.

Amazon handles all operational activities related to capacity scaling, pre-provisioning, etc and the customers only need to pay for the amount of space that they use. Here are a couple of key Amazon S3 features:

• Access Control: It offers comprehensive access controls to meet any kind of organizational and business compliance requirements through an easy-to-use control panel interface. 
• Support for Analytics: S3 supports analytics through the use of AWS Athena and AWS redshift spectrum through which users can execute SQL queries over data stored in S3.
• Encryption: S3 buckets can be encrypted by S3 default encryption. Once enabled, all items in a particular bucket will be encrypted. 
• High Availability: S3 achieves high availability by storing the data across several distributed servers. Naturally, there is an associated propagation delay with this approach and S3 only guarantees eventual consistency.

But, the writes are atomic; which means at any time, the API will return either the new data or old data. It’ll never provide a corrupted response. 

Conceptually S3 is organized as buckets and objects.

• A bucket is the highest-level S3 namespace and acts as a container for storing objects. They have a critical role in access control and usage reporting is always aggregated at the bucket level.
• An object is the fundamental storage entity and consists of the actual object as well as the metadata. An object is uniquely identified by a unique key and a version identifier. 

Customers can choose the AWS regions in which their buckets need to be located according to their cost and latency requirements.

A point to note here is that objects do not support locking and if two PUTs come at the same time, the request with the latest timestamp will win. This means if there is concurrent access, users will have to implement some kind of locking mechanism on their own. 

To know more about Amazon S3, visit this link.

Steps to Connect Elasticsearch to S3 Using Custom Code

Moving data from Elasticsearch to S3 can be done in multiple ways.

The most straightforward is to write a script to query all the data from an index and write it into a CSV or JSON file. But the limitations to the amount of data that can be queried at once make that approach a nonstarter.

You will end up with errors ranging from time outs to too large a window of query. So, you need to consider other approaches to connect Elasticsearch to S3.

Logstash, a core part of the ELK stack, is a full-fledged data load and transformation utility.

With some adjustment of configuration parameters, it can be made to export all the data in an elastic index to CSV or JSON. The latest release of log stash also includes an S3 plugin, which means the data can be exported to S3 directly without intermediate storage.

Thus, Logstash can be used to connect Elasticsearch to S3. Let us look in detail into this approach and its limitations.

Using Logstash

Logstash is a service-side pipeline that can ingest data from several sources, process or transform them and deliver them to several destinations.

In this use case, the Logstash input will be Elasticsearch, and the output will be a CSV file.

Thus, you can use Logstash to back up data from Elasticsearch to S3 easily.

Logstash is based on data access and delivery plugins and is an ideal tool for connecting Elasticsearch to S3. For this exercise, you need to install the Logstash Elasticsearch plugin and the Logstash S3 plugin. Below is a step-by-step procedure to connect Elasticsearch to S3:

Step 1: Execute the below command to install the Logstash Elasticsearch plugin.

logstash-plugin install logstash-input-elasticsearch

Step 2: Execute the below command to install the logstash output s3 plugin.

logstash-plugin install logstash-output-s3

Step 3: Next step involves the creation of a configuration for the Logstash execution. An example configuration to execute this is provided below.

 input {  elasticsearch {     hosts => "elastic_search_host"     index => "source_index_name"     query => '     {     "query": {     "match_all": {}     }     }    '   } } output {    s3{      access_key_id => "aws_access_key"      secret_access_key => "aws_secret_key"      bucket => "bucket_name"    } }

In the above JSON, replace the elastic_search_host with the URL of your source Elasticsearch instance. The index key should have the index name as the value.

The query tries to match every document present in the index. Remember to also replace the AWS access details and the bucket name with your required details.

Create this configuration and name it “es_to_s3.conf”.

Step 4: Execute the configuration using the following command.

logstash -f es_to_s3.conf

The above command will generate JSON output matching the query in the provided S3 location. Depending on your data volume, this will take a few minutes.

Multiple parameters that can be adjusted in the S3 configuration to control variables like output file size etc. A detailed description of all config parameters can be found in Elastic Logstash Reference [8.1].

By following the above-mentioned steps, you can easily connect Elasticsearch to S3.

Limitations of Connecting Elasticsearch to S3 Using Custom Code

The above approach is the simplest way to transfer data from an Elasticsearch to S3 without using any external tools. But it does have some limitations. Below are two limitations that are associated while setting up Elasticsearch to S3 integrations:

• This approach to connecting Elasticsearch to S3 works fine for a one-time load, but in most situations, the transfer is a continuous process that needs to be executed based on an interval or triggers. To accommodate such requirements, customized code will be required.
• This approach to connecting Elasticsearch to S3 is resource-intensive and can hog the cluster depending on the number of indexes and the volume of data that needs to be copied. 

Conclusion

This article provided you with a comprehensive guide to Elasticsearch and Amazon S3. You got to know about the methodology to backup Elasticsearch to S3 using Logstash and its limitations as well. Now, you are in the position to connect Elasticsearch to S3 on your own.

The manual approach of connecting Elasticsearch to S3 using Logstash will add complex overheads in terms of time and resources. Such a solution will require skilled engineers and regular data updates. Furthermore, you will have to build an in-house solution from scratch if you wish to transfer your data from Elasticsearch or S3 to a Data Warehouse for analysis.

Hevo Data provides an Automated No-code Data Pipeline that empowers you to overcome the above-mentioned limitations. Hevo caters to 150+ data sources (including 40+ free sources) and can seamlessly transfer your Elasticsearch data to a data warehouse or a destination of your choice in real-time. Hevo’s Data Pipeline enriches your data and manages the transfer process in a fully automated and secure manner without having to write any code. It will make your life easier and make data migration hassle-free.

Visit our Website to Explore Hevo

Want to take Hevo for a spin? Sign Up for a 14-day free trial and experience the feature-rich Hevo suite firsthand.

What are your thoughts on moving data from Elasticsearch to S3? Let us know in the comments.