Building Scalable REST APIs using Heroku Flask MongoDB: 6 Easy Steps

Sometimes, it can be challenging to build scalable REST APIs, especially for new developers with limited experience and understanding. This article demonstrates how to build a scalable REST API web service using Flask, a very nice Python web microframework paired with MongoDB, the most popular No-SQL database, and Heroku as the deployment platform. 

But before we start digging into the details, let’s cover some basics.

Table of Contents:

• What is an API?
• APIs: how do they work?
• What is an API Call?
• Why Flask?
• Why use MongoDB and Heroku?
  • Heroku Flask MongoDB: Setting up a Git Repository and a Virtual Environment
  • Heroku Flask MongoDB: Building a Database with MongoDB
  • Heroku Flask MongoDB: Loading Data in MongoDB
  • Heroku Flask MongoDB: Building the Web API with Flask
  • Heroku Flask MongoDB: Setting up Heroku
  • Heroku Flask MongoDB: Deploying Heroku
• Conclusion 

What is an API?

APIs are application programming interfaces. Essentially, it facilitates communication between two software systems. For example, when users enter a URL into their browser, like www.google.com, they make a GET request to Google’s server. The result of the request will be a Google response, which may contain HTML, CSS, JavaScripts, or other assets consumed by, executed by, and rendered by the user’s system. Likewise, when a system requests Raw Data from an API endpoint, the system receives Raw Data in return. Once the raw data is received, further processing can be performed. 

Web services come in many forms. They include:

1. SOAP — Simple Object Access Protocol
2. XML-RPC — XML Format for Data Transfer
3. JSON-RPC — JSON Format for Data Transfer
4. REST — Representational State Transfer

Our focus will be on utilizing REST as an architectural principle in this article.

APIs: how do they work?

An API function is a mechanism that allows data to be transferred from the client to an API server that communicates with a database. The client receives the data based on certain conditions.

APIs are commonly used in the stock market, companies, and daily prices. Considering this diagram, a client (user) might request price history details for a specific company. API calls made by the client specifying a particular company would trigger the API server to access the database and return the information to the client. In other words, there is no interaction with the database by the user, only with the API server.

What is an API Call?

A server API call involves specifying a request and getting the requested content from an endpoint on the server. Let’s look at logging into social media websites as an example. After filling out their login information, users would click “login.” As soon as the user clicks such a button, API calls request the user’s data and authenticate the user. This constitutes an API call itself. Below is an example URL used to request stock price data for an individual company.

As you can see above, the URL request has been broken. First, we start with the HTTPS protocol, then type the domain’s name. The path to the API and its version is then specified, following the best practices of a RESTful API. Finally, the endpoint is defined. 

A unique “funnel down” approach occurs here, where we narrow down our scope of data as we move from left to right. For example, we start by collecting all of the prices from all companies and then specifically request Apple’s prices as the company of interest. It’s a classic RESTful architecture pattern at its best. Next, we will discuss why we chose Flask, MongoDB for the development, and Heroku for deployment. 

Why Flask?

The two most popular Python frameworks for web development are Django and Flask. Django is a bit older and more popular. Although Django is older and has a larger community, Flask has its own strengths. The Flask framework was designed for simplicity and scalability from the ground up. Compared to Django applications, Flask applications are known for their lightweight nature. The goal of Flask developers is to keep its core extendable but straightforward and simple. Flask developers call it a microframework. As far as database choices and template engines are concerned, Flask doesn’t make many decisions. Additionally, Flask provides comprehensive documentation for developers.

Why use MongoDB and Heroku?

Setting up MongoDB is relatively inexpensive. Unlike Relational Database Management Systems (RDMS), this is a Document-oriented, NoSQL database where each record is independent of a defined schema. 

As a result, MongoDB can scale out for dealing with large volumes of data and be flexible to changing data structures.

In no time, you can build and deploy fully managed cloud applications using Heroku, one of the most popular PaaS solutions. It’s easy to integrate MongoDB with Heroku applications. Your only task is to set the connection string for your MongoDB cluster to a Heroku config variable. It’s that simple! What’s more? 

Development Steps for Heroku Flask MongoDB

This demo creates a simple web service with a single API endpoint. This application will be hosted on the cloud using Heroku to provide reliability and availability, amongst other benefits.

Heroku Flask MongoDB: Setting up a GitHub repository and a virtual environment

Git is an open-source distributed version control system. In this project, we will create a repo on GitHub and clone it to our local machine with $ git clone “https://github.com/[github_name]/[repo_name].git.” To ensure commands are run with the new repository as the root, use $ cd [repo_name] to change the directory. 

The next step in Heroku Flask MongoDB is to create virtual environments to isolate the projects.

Therefore, every project can have its own dependencies, regardless of other projects’ dependencies.

Set up a new environment for Heroku Flask MongoDB by running $conda create –name env_name python=3.6, then activate it with conda activate env_name. Then, use $ conda deactivate to disable the virtual environment. It will be necessary to install all the dependencies through $ pip install before importing them.

Heroku Flask MongoDB: Building a Database with MongoDB 

The MongoDB database is intuitive and easy to use and set up. There is no need to define a schema before dumping large amounts of data. This enables rapid development and iteration. 

Create an account with MongoDB, then create an organization and a project. Next, you’ll need to create a cluster. To quickly prototype our database, we used the MongoDB Atlas Free Tier Cluster, which gives us 512MB of storage and tremendous flexibility.

Before we can use MongoDB Atlas, we need to secure the cluster. For clients to access clusters, Atlas requires authentication, so we must identify what IP addresses and users have access to. First, create a username and password for the database access section, then add an IP address to the network access section.

Our setup was to have only one shared username/password accessed from various IP addresses. Copy the connection string following the selection of Python as the driver. The connection string is formatted accordingly to help the driver interpret it. Database URLs typically take the form dialect+driver://username:password@host:port/database.

In the case of an application, the connection string should be set as an environment variable. Environment variables exist independently of any script and are considered part of the operating system. 

With Python-dotenv, we can read key-value pairs from a .env file and set them as environment variables at run-time. Using this approach for Heroku Flask MongoDB, we can develop scripts as if we already have environment variables set. To ensure that .env files aren’t included in git commits locally or remotely, add them to .gitignore.

$ echo "password=[mongo_password]" > .env
$ echo ".env" >> .gitignore

Heroku Flask MongoDB: Loading Data into MongoDB

We will load some data into the database for Heroku Flask MongoDB now that it has been set up. Data in the database can be found in JSON format.

Sample data

Python works with MongoDB using PyMongo. The syntax for MongoDB is quite logical when compared to Object-Oriented Programming, at least for basic queries. 

To connect to MongoDB, you can simply pass the connection string to the pymongo.MongoClient() constructor. The syntax load_dotenv() loads the .env file into the environment. This will be included as part of the connection string. We need to open the JSON file first before importing it into MongoDB. This does not require any transformation since MongoDB demands documents. 

In PyMongo, there are two types of insertion: 1) insert_one() and 2) insert_many() for a single document or a list of documents. In many ways, PyMongo’s syntax is similar to that of MongoDB. It’s okay to use the insert() command in Mongo Shell, but it’s deprecated for PyMongo. Once the required dependencies have been installed, you only need to run this script once to load the data: $ python load_data.py:

$ pip install python-dotenv
$ pip install pymongo
$ pip install dnspython

Heroku Flask MongoDB: Building the Web API with Flask

We create the web API with Flask for Heroku Flask MongoDB, part of the Web Application. Flask is used in a minimal capacity in this demonstration, but it can be fully utilized to build a robust website in Heroku Flask MongoDB.

Next, an instance of the Flask class was created. Then, the application’s name was passed as an argument to the constructor.

An application built with Flask is called a Web Server Gateway Interface application, which means the webserver forwards all requests to this instance for further processing. 

You need to install Flask for Heroku Flask MongoDB with $ pip install flask, run the app with $ python app_basic.py and visit localhost:5000. Flask serves the Web Application on port 5000, or http://127.0.0.1:5000, on the local machine. To terminate the server, press CTRL+C on the terminal. 404 Not Found errors will be encountered for URLs not mapped to view functions.

Finally, we will build an endpoint that is useful to our users. In this case, we want to provide data in the form of a timeline:

In PyMongo, Aggregate Pipeline Operators can construct expressions for use in Aggregate Pipeline Stages. Documents are passed from stage to stage in the pipeline. We want to group our original data by date and create a list of all launches. The $group operator is used on the launch_date key in this pipeline. In addition, the $group operator searches for an accumulator. $avg , $max , or $sum are the most common accumulators. Rather than having the accumulator return a list of documents, we will use $push. It showcases what I see as a major advantage of MongoDB’s Object-Oriented nature and flexibility when combined with the querying language.

Heroku Flask MongoDB: Setting Up Heroku 

Our Heroku Flask MongoDB application must be hosted on Heroku or another Platform as a Service to allow global access. Therefore, getting started with Heroku is very easy. To simplify the deployment of the Heroku Flask MongoDB application, we do not use the Heroku CLI. 

Heroku lets you deploy code with ease using GitHub integration. Start by creating a new Heroku application. After logging in, click “Create new app” and give it a unique name to finish the process. Please search for the application’s source code in the GitHub repository using the name, and connect it to the repository.

Several things need to be configured before we deploy Heroku Flask MongoDB application: Config Vars, requirements.txt, and Procfile.

Heroku Flask MongoDB: Heroku Configuration

To safeguard MongoDB passwords, we need to set them as an environment variable. Go to the Dashboard and select “Reveal Config Varis” from the Setting page. Unfortunately, the password environment variable is missing from our code’s connection string to the database. Therefore, a MongoDB connection password must be added manually as an environment variable.

The following two files are required to deploy the application to Heroku: 1) requirements.txt and 2) Profile. These files are both required to configure Heroku’s hosting server. The files must be manually created in the repository and reside in the root so Heroku can find them during deployment.

1. Profile— This file specifies how Heroku should run the application and defines which commands should be run. The format is *process type>: *command>, where the process type is the prototype from which a dyno is instantiated.
2.  requirements.txt— You can find a list of all the packages and dependencies required to run the application. Run the $ pip freeze > requirements.txt command in the project directory to create the requirements.txt file. The requirements.txt file will be updated with a list of all installed libraries in the active environment. As a result, Heroku can install all required dependencies.

Heroku Flask MongoDB: Deploying to Heroku

As soon as both configuration files ( Procfile and requirements.txt ) are created, save all working files in the repository before adding/committing/pushing them to the remote repository.

$ git add
$ git commit -m' ready to deploy
$ git push

Once the code is on GitHub, we can deploy the Heroku Flask MongoDB application to Heroku. From the Deploy page, select the branch to deploy. Heroku will then start building your application.

Conclusion

Here we covered what an API is, how it works, and what an API call is. Then we walked through creating a RESTful API using Heroku Flask MongoDB. Lastly, we discussed how to deploy the app on Heroku to give us access to our API from anywhere.

Heroku and MongoDB are trusted source that a lot of companies use as it provides many benefits but transferring data from it into a data warehouse is a hectic task. The Automated data pipeline helps in solving this issue and this is where Hevo comes into the picture. Hevo Data is a No-code Data Pipeline and has awesome 100+ pre-built Integrations that you can choose from.

visit our website to explore hevo[/hevoButton]

Hevo can help you Integrate your data from numerous sources and load them into a destination to Analyze real-time data with a BI tool such as Tableau. It will make your life easier and data migration hassle-free. It is user-friendly, reliable, and secure.

SIGN UP for a 14-day free trial and see the difference!

Share your experience of learning about Heroku Flask MongoDB in the comments section below.