Modern application development is based on cloud service providers like AWS, Azure, Google cloud services, etc, and does not need organizations to set up and maintain infrastructure. This paradigm is represented by the word serverless development usually.
Deploying and managing such applications poses new problems compared to the traditional way of handling Serverless CI/CD. This has given rise to numerous products that specialize in this area and naturally, most of them are named after the generic concept of serverless computing. Serverless ( https://www.serverless.com/) is one such product that helps organizations to deploy their cloud services with just a click.
This blog is about how to implement Serverless CI/CD using the serverless framework.
Table of Contents
- The Need for Serverless Framework
- What is Serverless?
- How to set up Serverless CI/CD?
- AWS Account with Access key and the secret.
- The basic idea of cloud-based application development.
The Need for Serverless Framework
Before going deep into the Serverless framework, let’s understand the paradigm of cloud functions and services better. Cloud functions allow one to completely ignore the infrastructure and focus on the implementation logic. This is made possible by the horizontally scalable infrastructure provided by hyper-scalers like AWS Lambda functions, Google Cloud functions, etc, or by platforms set up by the organization’s infrastructure team using container environments like Kubless or KNative. Lambda functions. Deploying applications to such scalable function environments are a different ball game compared to traditional Serverless CI/CD and the mechanism are often tightly coupled to the function environment provider.
For example, if your provider is AWS, then you would use a combination of AWS SAM and AWS Cloud formation templates to define the infrastructure and deploy the code. In the case of Azure, this will be Azure pipelines and Azure Resource Manager templates.
The Serverless CI/CD framework tries to solve this problem of tight coupling with a provider by acting as a middleman module that abstracts away the provider-specific details from the application-specific deployment scripts. Now that the background is clear, let us understand more about Serverless and its abilities.
Hevo Data, a Fully-managed Data Pipeline platform, can help you automate, simplify & enrich your data replication process in a few clicks. With Hevo’s wide variety of connectors and blazing-fast Data Pipelines, you can extract & load data from 100+ Data Sources straight into your Data Warehouse or any Databases. To further streamline and prepare your data for analysis, you can process and enrich raw granular data using Hevo’s robust & built-in Transformation Layer without writing a single line of code!
Get Started with Hevo for Free
Try our 14-day full access free trial today!
What is Serverless?
Serverless helps you develop, deploy and troubleshoot your serverless applications quickly and easily. It can also help you create an infrastructure that your applications depend on. It does it by accessing the provider-specific mechanisms under the hood and abstracting away the complexities from the developer.
For example, while working with AWS, Serverless internally uses AWS Cloud formation and gives a clean interface for the developer to work on it.
Serverless supports many providers like AWS, Azure, GCP, Tencent, KNative, and Kubeless to name a few. KNative and Kubeless are open-source container-based infrastructure platforms that can be deployed on-premise. This means Serverless can be useful to you even if you are not part of any hyper-scalers like AWS, GCP, and Azure.
Serverless provides a dashboard where you can manage your applications and providers through an intuitive user interface. It is possible to deploy or terminate your applications directly from the dashboard. This is a hosted dashboard that is offered free of cost. While the framework supports languages like Python, Java, NodeJS, C#, Go, Ruby, etc, the dashboard only supports Python and NodeJS run times as of now.
Serverless CI/CD currently supports GitHub and BitBucket as your version control system. AWS is the only provider where you can use Serverless for one-click deployment. Support for more cloud providers is underway.
Let’s now try to use Serverless to create a simple HTTP function and deploy it.
How to set up Serverless CI/CD?
Setting up Serverless CI/CD starts with installing Serverless in your development machine and having access to the Serverless dashboard.
- Install Serverless in your Linux-based instance by using the below command.
curl -o- -L https://slss.io/install | bash
- Install AWS CLI on your system using the below commands.
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
- Configure the AWS credentials by using the below command. You must enter your AWS access key and secret when prompted.
- Once this is done, use the below command to initiate a project using Serverless. The console will now prompt asking what kind of project to start.
For now, we will select an AWS – Python Starter project and use the templates available to create one.
The console will then ask you to log in to your Serverless dashboard for setting up monitoring and deployment. Follow the below sequence to get set up in Serverless
Manually connecting data sources to Databases and then creating Data Pipelines is a lackluster task. Experience Hevo’s automated No Code Data Pipelining solution that not only helps you replicate data but also automates the ETL process and you don’t have to write a single line of code.
Check out why Hevo is the Best:
- Secure: Hevo has a fault-tolerant architecture that ensures that the data is handled in a secure, consistent manner with zero data loss.
- Schema Management: Hevo takes away the tedious task of schema management & automatically detects the schema of incoming data and maps it to the schema of your Relational Database.
- Auto Schema Mapping: Hevo takes away the tedious task of schema management & automatically detects the format of incoming data from a source or Relational Database and replicates it to the destination schema.
- Transformations: Hevo provides preload transformations to make your incoming data fit for the chosen destination. You can also use drag and drop transformations like Date and Control Functions, JSON, and Event Manipulation to name a few.
- Incremental Data Load: Hevo allows the transfer of data that has been modified in real-time. This ensures efficient utilization of bandwidth on both ends.
- Live Support: The Hevo team is available round the clock to extend exceptional support to its customers through chat, E-Mail, and support calls.
Want to take Hevo for a spin? Use Sign Up For a 14 Day Free Trial here for a 14-day free trial and experience the feature-rich Hevo.
- If your AWS credentials are correct you should now be set up in the Serverless dashboard. Head to the dashboard at https://app.serverless.com/ and log in.
- Let’s now add this project to the Github repository. This can be done by using the below commands
cd test-2git init .
git add .
git remote add origin https://github.com/yourusername/your-repo-name.git
- You will now connect the project with Github to set up Serverless CI/CD. Click on the right-hand side three-dot menu and Enter the setting screen.
- Click connect and enter the credentials for your Github account to enable the connection. The repositories with a valid Servless.yml file should automatically get listed
- Once this is done, you should be able to push to the GitHub repository directly and the deployment should be taken care of automatically. Before closing this, let us spend some time understanding the serverless.yml file where all the core specification of deployment and infrastructure is defined.
In the yml file, you can see a handler function mentioned. This function will contain the business logic for the function. You can also see the provider being mentioned here. For now, AWS is the only provider for which Serverless CI/CD is supported, but other functionalities of Serverless like monitoring, deploying via the dashboard, etc are supported by other providers too.
You have now learned the basics of Serverless and how to set up Serverless CI/CD based on Serverless. Serverless provides a cloud-agnostic way of developing, troubleshooting, and deploying your serverless applications. Its support for most of the popular cloud providers like AWS, Azure etc, and on-premise platforms like KNative, Kubeless, etc makes it a good choice for standardizing your cloud-based application development.
Visit our Website to Explore Hevo
Hevo Data, a No-code Data Pipeline can seamlessly transfer data from a vast sea of 100+ sources to a Data Warehouse or a Destination of your choice. It is a reliable, completely automated, and secure service that doesn’t require you to write any code!
If you are using CRMs, Sales, HR, and Marketing applications and searching for a no-fuss alternative to Manual Data Integration, then Hevo can effortlessly automate this for you. Hevo, with its strong integration with 100+ sources (Including 40+ Free Sources), allows you to export, load, and transform data — and also make it analysis-ready in a jiffy!
Also, let’s know about your thoughts and building process for Serverless CI/CD in the comments section below.