Connecting Elasticsearch to S3: 4 Easy Steps

on Tutorial, Database, ETL • February 7th, 2020 • Write for Hevo

Are you trying to derive deeper insights from your Elasticsearch by moving the data into a larger Database like Amazon S3? Well, you have landed on the right article. Now, it has become easier to replicate data from Elasticsearch to S3.

This article will give you a brief overview of Elasticsearch and Amazon S3. You will also get to know how you can set up your Elasticsearch to S3 integration using 4 easy steps. Moreover, the limitations of the method will also be discussed in further sections. Read along to know more about connecting Elasticsearch to S3 in the further sections.

Table of Contents

Introduction to Elasticsearch

ElasticSearch Logo
Image Source

Elasticsearch accomplishes its super-fast search capabilities through the use of a Lucene-based distributed reverse index. When a document is loaded to Elasticsearch, it creates a reverse index of all the fields in that document. A reverse index is an index where each of the entries is mapped to a list of documents that contains them. Data is stored in JSON form and can be queried using the proprietary query language.

Elasticsearch has four main APIs – Index API, Get API, Search API and Put Mapping API. Index API is used to add documents to the index. Get API allows to retrieve the documents and Search API enables querying over the index data. Put Mapping API is used to add additional fields to an already existing index. 

The common practice is to use Elasticsearch as part of the standard ELK stack, which involves three components – Elasticsearch, Logstash, and Kibana. Logstash provides data loading and transformation capabilities. Kibana provides visualization capabilities. Together, three of these components form a powerful Data Stack. 

Behind the scenes, Elasticsearch uses a cluster of servers to deliver high query performance. An index in Elasticsearch is a collection of documents. Each index is divided into shards that are distributed across different servers. By default, it creates 5 shards per index with each shard having a replica for boosting search performance. Index requests are handled only by the primary shards and search requests are handled by both the shards. 

The number of shards is a parameter that is constant at the index level. Users with deep knowledge of their data can override the default shard number and allocate more shards per index. A point to note is that a low amount of data distributed across a large number of shards will degrade the performance. 

Scaling in Elasticsearch is accomplished by adding more servers. The architecture can automatically rebalance the data and query load across available nodes. Fault tolerance in Elasticsearch is accomplished through cross-cluster replication. A remote cluster can be set up to sync with the primary cluster and serve as hot standby.

Amazon offers a completely managed Elasticsearch service that is priced according to the number of instance hours of operational nodes. 

To know more about Elasticsearch, visit this link.

Introduction to Amazon S3

Amazon S3 Logo
Image Source

AWS S3 is a fully managed object storage service that is used for a variety of use cases like hosting data, backup and archiving, data warehousing, etc. Amazon handles all operational activities related to capacity scaling, pre-provisioning, etc and the customers only need to pay for the amount of space that they use. It offers comprehensive access controls to meet any kind of organizational and business compliance requirements through an easy-to-use control panel interface. 

S3 supports analytics through the use of AWS Athena and AWS redshift spectrum through which users can execute SQL queries over data stored in S3. S3 buckets can be encrypted by S3 default encryption. Once enabled, all items in a particular bucket will be encrypted. 

S3 achieves high availability by storing the data across a number of distributed servers. Naturally, there is an associated propagation delay with this approach and S3 only guarantees eventual consistency. But, the writes are atomic; which means at any time, the API will return either the new data or old data and never will it provide a corrupted response. 

Conceptually S3 is organized as buckets and objects. A bucket is the highest level S3 namespace and acts as a container for storing objects. They have a critical role in access control and usage reporting is always aggregated at the bucket level. An object is the fundamental storage entity and consists of the actual object as well as the metadata. An object is uniquely identified by a unique key and a version identifier. 

Customers can choose the AWS regions in which their buckets need to be located according to their cost and latency requirements. A point to note here is that objects do not support locking and if two PUTs come at the same time, the request with the latest timestamp will win. This means if there is concurrent access, users will have to implement some kind of locking mechanism on their own. 

To know more about Amazon S3, visit this link.

Simplify Integrations Using Hevo’s No-code Data Pipeline

Hevo Data helps you directly transfer data from Asana, Github, and 100+ data sources (including 30+ free sources) to Business Intelligence tools, Data Warehouses, or a destination of your choice in a completely hassle-free & automated manner. Hevo is fully managed and completely automates the process of not only loading data from your desired source but also enriching the data and transforming it into an analysis-ready form without having to write a single line of code. Its fault-tolerant architecture ensures that the data is handled in a secure, consistent manner with zero data loss.

Hevo takes care of all your data preprocessing needs required to set up the integration and lets you focus on key business activities and draw a much powerful insight on how to generate more leads, retain customers, and take your business to new heights of profitability. It provides a consistent & reliable solution to manage data in real-time and always have analysis-ready data in your desired destination. 

Get Started with Hevo for Free

Check out what makes Hevo amazing:

  • Real-Time Data Transfer: Hevo with its strong Integration with 100+ Sources (including 30+ Free Sources), allows you to transfer data quickly & efficiently. This ensures efficient utilization of bandwidth on both ends.
  • Data Transformation: It provides a simple interface to perfect, modify, and enrich the data you want to transfer. 
  • Secure: Hevo has a fault-tolerant architecture that ensures that the data is handled in a secure, consistent manner with zero data loss.
  • Tremendous Connector Availability: Hevo houses a large variety of connectors and lets you bring in data from numerous Marketing & SaaS applications, databases, etc. such as HubSpot, Marketo, MongoDB, Oracle, Salesforce, Redshift, etc. in an integrated and analysis-ready form.
  • Simplicity: Using Hevo is easy and intuitive, ensuring that your data is exported in just a few clicks. 
  • Completely Managed Platform: Hevo is fully managed. You need not invest time and effort to maintain or monitor the infrastructure involved in executing codes.
  • Live Support: The Hevo team is available round the clock to extend exceptional support to its customers through chat, email, and support calls.
Sign up here for a 14-Day Free Trial!

Steps to Connect Elasticsearch to S3 using Custom Code

Moving data from Elasticsearch to S3 can be done in multiple ways. The most straightforward is to write a script to query all the data from an index and write it into a CSV or JSON file. But the limitations to the amount of data that can be queried at once make that approach a nonstarter. You will end up with errors ranging from time outs to too large a window of query. So, you need to consider other approaches to connect Elasticsearch to S3.

Logstash, which is a core part of the ELK stack, is a full-fledged data load and transformation utility. With some adjustment of configuration parameters, it can be made to export all the data in an elastic index to CSV or JSON. The latest release of log stash also includes an S3 plugin, which means the data can be exported to S3 directly without intermediate storage. Thus, Logstash can be used to connect Elasticsearch to S3. Let us look in detail into this approach and its limitations.

Using Logstash

Image Source

Logstash is a service side pipeline that can ingest data from a number of sources, process or transform them and deliver to a number of destinations. In this use case, Log stash input will be Elasticsearch and output will be a CSV file. Thus, using logstash, Elasticsearch to S3 integration can be easily done.

Logstash is based on data access and delivery plugins. For this exercise, you need to install the Logstash Elasticsearch plugin and the Logstash S3 plugin. Below is a step-by-step procedure to connect Elasticsearch to S3:

  1. Execute the below command to install logstash Elasticsearch plugin.
    logstash-plugin install logstash-input-elasticsearch
  2. Execute the below command to install logstash output s3 plugin.
    logstash-plugin install logstash-output-s3
  3. Next involves creating a configuration for the logstash execution. An example configuration to execute this is provided below.

     

    input {  elasticsearch {     hosts => "elastic_search_host"     index => "source_index_name"     query => '     {     "query": {     "match_all": {}     }     }    '   } } output {    s3{      access_key_id => "aws_access_key"      secret_access_key => "aws_secret_key"      bucket => "bucket_name"    } }

    In the above JSON, replace the elastic_search_host with the URL of your source Elasticsearch instance. The index key should have the index name as the value. The query tries to match every document present in the index. Remember to also replace the AWS access details and the bucket name with your required details.

  4. Create this configuration and name it as es_to_s3.conf

  5. Execute the configuration using the following command.
    logstash -f es_to_s3.conf

    The above command will generate JSON output matching the query in the provided S3 location. Depending on your data volume, this will take a few minutes. There are multiple parameters that can be adjusted in the S3 configuration to control variables like output file size etc. A detailed description of all config parameters can be found here.

    By following the above-mentioned steps, you can easily connect Elasticsearch to S3.

Limitations of Connecting Elasticsearch to S3 using Custom Code

The above approach is the simplest way to transfer data from an Elasticsearch to S3 without using any external tools. But it does have some limitations. Below are to limitations that are associated while setting up Elasticsearch to S3 integrations:

  1. This approach to connect Elasticsearch to S3 works fine for a one-time load, but in most situations, the transfer is a continuous process that needs to be executed based on an interval or triggers. To accommodate such requirements, customized code will be required.
  2. This approach to connect Elasticsearch to S3 is resource-intensive and can hog the cluster depending upon the number of indexes and the volume of data that needs to be copied. 

Conclusion

This article provided you with a comprehensive guide to Elasticsearch and Amazon S3. You got to know about the methodology to connect Elasticsearch to S3 using Logstash and its limitations as well. Now, you are in the position to connect Elasticsearch to S3 on your own.

Visit our Website to Explore Hevo

Businesses can use automated platforms like Hevo Data to set this integration and handle the ETL process. It helps you directly transfer data from a source of your choice to a Data Warehouse, Business Intelligence tools, or any other desired destination in a fully automated and secure manner without having to write any code and will provide you a hassle-free experience.

Want to take Hevo for a spin? Sign Up for a 14-day free trial and experience the feature-rich Hevo suite first hand. You can also have a look at the unbeatable pricing that will help you choose the right plan for your business needs.

What are your thoughts on moving data from Elasticsearch to S3? Have you explored other approaches that have worked for you? Let us know in the comments.

No-code Data Pipeline for S3