GitLab Pipeline APIs & Triggers: Made Easy 101

In this modern and advanced technical world, the demand for Continuous Integration and Development (CI/CD) in day-to-day business operations has increased. Businesses have increased their dependency on various  DevOps tools to optimize their workflow to deal with the fast-paced environment. A DevOps Tool is essentially an application that aids in the automation of the software development process. It primarily focuses on communication and collaboration among various teams in product management, software development, and operations. Numerous DevOps tools, such as GitLab, have come into the picture that helps developers in optimizing their software development and workflow.

In this article, you will gain information about GitLab Pipeline API. You will also gain a holistic understanding of GitLab, its key features, the usage of GitLab API, the definition of CI/CD Pipeline, and its working in GitLab. It will also provide information on how to trigger a pipeline status report using the GitLab Pipeline API.

Read along to find out in-depth information about GitLab Pipeline API.

What is GitLab?

GitLab is a free, open-source, web-based DevOps platform that enables developers to perform all project tasks in a single application, including planning, development, security, operation, and monitoring. It began as an open-source project in which teams collaborated to create better software. The goal of GitLab is to provide a common platform where each team member can have a direct impact on the company’s roadmap.

GitLab assists teams in shortening product lifecycles and increasing productivity, which creates value for customers. Users are not required to manage authorizations for each tool in the application. If permissions are only set once, everyone in the organization has access to all components. GitLab assists developers and other technical teams in reducing complexity and speeding up DevOps. Users can use GitLab to shorten product lifecycles and increase productivity.

Key Features of GitLab

Some of the key features of GitLab are as follows:

• Activity Stream: GitLab displays a list of the most recent commits, merges, comments, and team members on your project.
• Package Management: GitLab includes built-in packages like Maven, C++, etc. It allows teams to create a reliable and consistent software supply chain and workflow.
• Powerful Branching: Git has a branch that contains information about its history. This can be created, moved, or shared instantly.
• Container Scanning: GitLab users can run security scans to ensure that Docker images are not vulnerable in the environment.
• Auto DevOps: GitLab automatically configures the software development lifecycle by default. It creates, detects, tests, deploys, and tracks applications.

How & When to Use GitLab API?

GitLab API will enable your team with the flexibility to perform numerous actions you would normally carry out using the user interface of GitLab. 

There are two major ways to interact with the GitLab API: 

• Rest API
• GraphQL API

Using REST API in GitLab, you can easily access using the regular API requests. Using GraphQL, you can also access the GitLab API. Here, you can make an API request according to your specific needs, and it is versioned by default.

The GitLab API allows you to do everything you can do with the web UI using the input devices but through a REST API or GraphQL API.

There isn’t any specific use case for using GitLab APIs. Most of the time, there won’t be any requirement of using the GitLab because the Web UI of GitLab is very straightforward and feature-rich. However, to minimize the excessive usage of keyboard and mouse, the GitLab API enables you to automate your tasks and derive information in a convenient manner.

What is a CI/CD Pipeline? How does it Work in GitLab?

CI and CD stand for Continuous Integration and Continuous Delivery/Continuous Deployment. A CI/CD pipeline is a set of steps that must be performed in a chronological manner to deliver new versions of software in increments. This practice aims to improve the software delivery throughput. Simply put, it delivers software in a faster and more efficient manner.

The pipelines are made up of:

• Jobs: They define what you do. For example, jobs that compile or test code
• Stages: They specify when the jobs should be run. For example, there are different stages of compiling the code and then running the tests.

Runners carry out the jobs. If there are an ample number of concurrent runners, multiple jobs in the same stage are executed in parallel.

• If all jobs in a stage are successful, the pipeline advances to the next stage.
• If any of the jobs in a stage fail, the next stage is not (usually) executed, and the pipeline terminates early.

A typical pipeline might have four stages that are carried out in the following order:

• A build stage with the compile job.
• A test stage with two jobs, test1, and test2.
• A staging stage with the job deploy-to-stage.
• A deployment stage with a job called deploy-to-prod.

Pipelines, in general, are executed automatically and do not require any intervention once created. However, there are times when you have to interact with a pipeline manually. Pipelines can be run manually with predefined or user-specified variables.

To execute a pipeline manually, the steps to be followed are as follows:

• Step 1: Go to your GitLab account.
• Step 2: On the top bar, select the “Menu” button. Then, click on the “Projects” option. You can search and find your project.
• Step 3: Navigate to the left sidebar. Then, select the “CI/CD” option. Then, choose the “Pipelines” option.
• Step 4: Now, select the “Run pipeline” option.
• Step 5: In the form that appears, you can fill the fields accordingly. In the “Run for branch name or tag” field, you can select the branch or tag you want to run the pipeline for.
• Step 6: Enter the name of the environment variables that are required for the pipeline to run. You can also set some default values for some specific variables so that the values are prefilled.
• Step 7: Now, select the “Run pipeline” button.

The pipeline now executes the jobs as configured.

Streamline your CI/CD processes with GitLab’s scheduled pipelines to automate routine tasks efficiently. Learn more at GitLab Scheduled Pipeline.

How to Trigger a Pipeline Status Report Using the GitLab Pipeline API?

The GitLab Pipeline API will help you in your pipeline by leveraging the web UI with the help of a keyboard and mouse but via a REST API or GraphQL API.

Triggering a pipeline status report using the GitLab Pipeline API can be showcased using a simple example. The example illustrated the creation of a wiki-based status report using the GitLab Pipeline API.

• GitLab Pipeline API: Open the GitLab Project
• GitLab Pipeline API: The Project for Reporting
• GitLab Pipeline API: Breakdown of the .gitlab-ci.yml file
• GitLab Pipeline API: The Nodejs Application

1) GitLab Pipeline API: Open the GitLab Project

• First, you need to open the Project in GitLab where you will be using the GitLab Pipeline API.
• If you have a private project, you can use a personal access token or a private access token. And if you want to access numerous projects, you can use the personal access token.
• Suppose you’re working on this GitLab project, here, the project ID is 278964.

• You can use the following command to get the details of the project.

$ curl  https://gitlab.com/api/v4/projects/278964 | jsonpp

Output:

{
  "id": 278964,
  "description": "GitLab is an open-source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more. Self-host GitLab on your own servers, in a container, or on a cloud provider.",
  "name": "GitLab",
  "name_with_namespace": "GitLab.org / GitLab",
  "path": "gitlab",
  "path_with_namespace": "gitlab-org/gitlab",
  "created_at": "2015-05-20T10:47:11.949Z",
  "default_branch": "master",
  "tag_list": [],
  "topics": [],
  "ssh_url_to_repo": "git@gitlab.com:gitlab-org/gitlab.git",
  "http_url_to_repo": "https://gitlab.com/gitlab-org/gitlab.git",
  "web_url": "https://gitlab.com/gitlab-org/gitlab",
  "readme_url": "https://gitlab.com/gitlab-org/gitlab/-/blob/master/README.md",
  "avatar_url": "https://assets.gitlab-static.net/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png",
  "forks_count": 3936,
  "star_count": 2858,
  "last_activity_at": "2021-06-21T16:36:22.382Z",
  "namespace": {
    "id": 9970,
    "name": "GitLab.org",
    "path": "gitlab-org",
    "kind": "group",
    "full_path": "gitlab-org",
    "parent_id": null,
    "avatar_url": "/uploads/-/system/group/avatar/9970/logo-extra-whitespace.png",
    "web_url": "https://gitlab.com/groups/gitlab-org"
  }
}

• If you want to see the latest pipelines in your project, you can use the following command.

$ curl  https://gitlab.com/api/v4/projects/278964/pipelines | jsonpp

2) GitLab Pipeline API: The Project for Reporting

In any project, Wiki pages are a git repository that you can access from the pipeline. You can publish information on your Wiki page from the pipeline.

• In this example, the Wiki page of the project is used to create a simple table in markdown that will show all projects in a group. It will also show the status of the last run.
• The proposal is to be able to write the following code in the .gitlab-ci.yml file.

update-doc:
  stage: update-doc
  script:
     - git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@domain.com/user/project.wiki.git
     - cd project.wiki.git
     - echo "test add text" > home.md
     - git add home.md
     - git commit -m "test update home.md"
     - git push https://gitlab-ci-token:${CI_BUILD_TOKEN}@domain.com/user/project.wiki.git

• The output of the GitLab project will look like the following:

3) GitLab Pipeline API: Breakdown of the .gitlab-ci.yml file

• You first need to install the packages before building the application. Then you can run the application.
• In line 40, you can see that the application is first run with the “— group” option and the “himinds-boosting-innovation” GitLab Group. Since. this is a private group, the project’s Personal Access Token is set to “BOT_TOKEN.”
• The token will be printed on the command line by the NodeJS application. The “BOT_TOKEN” is also masked so there is no need to worry.

• The application will generate a file called “ci-report.md” that you will need to push to the Wiki git repository.
• In this example, the file is being overwritten each time.

The .gitlab-ci.yml file looks something like shown below:

image: node:14.16.1

stages:
  - ci-report-wiki

ci-report-wiki:
  stage: ci-report-wiki
  variables:
    # specify the filename you want to modify
    FILENAME: "ci-report.md"
  script:
    # define URL for the wiki in terms of project-agnostic predefined variables
    - export WIKI_URL="${CI_SERVER_PROTOCOL}://project_${CI_PROJECT_ID}_bot:${BOT_TOKEN}@${CI_SERVER_HOST}:${CI_SERVER_PORT}/${CI_PROJECT_PATH}.wiki.git"

    # Run node application
    - yarn install
    - yarn build
    - yarn start --group "himinds-boosting-innovation"
    - ls -all
    - pwd

    # clone this project's wiki under /tmp
    - rm -rf "/tmp/${CI_PROJECT_NAME}.wiki"
    - cd /tmp
    - git clone "${WIKI_URL}"

    # enter the cloned repo
    - cd "${CI_PROJECT_NAME}.wiki"

    - cp "/builds/${CI_PROJECT_PATH}/$FILENAME" .
    - ls -all

    # set committer info
    - git config user.name "$GITLAB_USER_NAME"
    - git config user.email "$GITLAB_USER_EMAIL"

    # commit the file
    - git add "$FILENAME"
    - git commit -m "Auto-updated file in CI"

    # push the change back to the master branch of the wiki
    - git push origin "HEAD:master"

4) GitLab Pipeline API: The Nodejs Application

• For interacting with the GitLab Pipeline API, the @nerdvision/gitlab-js package is used.
• Using the package, you can scrape GitLab. And with the json2md package, you can create a markdown file.
• The source code of the project is:

import { Client } from '@nerdvision/gitlab-js';
import json2md from 'json2md'

import dotenv from 'dotenv'
dotenv.config()
const fs = require('fs');
import { Command } from 'commander';
import { posix } from 'node:path';
const program = new Command();

program
  .option('-g, --group <type>', 'specify group')
program.parse(process.argv);
const options = program.opts();
interface ProjectCIStatus {
  name: string
  branch: string
  status: string
}
const token = process.env.BOT_TOKEN
console.log(token); //Just to show that it is masked

const client = new Client({
  token: token
});

let getProjectCollection = async (groupName: string) => {
  const group = await client.groups.find(groupName);
  const projects = await group.projects.list();

  let projectCIStatusCollection: ProjectCIStatus[] = [];

  for (let project of projects) {
    const pipelines = await project.pipelines.list();
    //only for projects that uses pipelines
    if (pipelines.length != 0) {
      const latestPipeline = pipelines[0].data;
      const element = { name: project.data.name, branch: latestPipeline.ref, status: latestPipeline.status };
      projectCIStatusCollection.push(element);
    }
  }
  return projectCIStatusCollection;
}

let convertToMarkDown = (collection: ProjectCIStatus[]) => {
  return json2md([
    { h1: 'Report' },
    { h2: `created: ${new Date().toLocaleString('se-SE')}` },
    { p: "Results from the last run." },
    {
      table: {
        headers: ['Name', 'Ref', 'Status'],
        rows: collection.map((element) => ({
          Name: element.name,
          Ref: element.branch,
          Status: element.status
        })),
      },
    },
  ]);
}

(async () => {

  if (!options.group) {
    console.error("You need to specify a group!");
    process.exit(0);
  }

  let groupName = program.opts().group;
  let projectCollection: ProjectCIStatus[] = await getProjectCollection(groupName);
  let markdown = convertToMarkDown(projectCollection);
  fs.writeFileSync("ci-report.md", markdown);
})();

• In this case, there are two functions:
  • The “getprojectCollection” function scrapes all the data.
  • The “convertToMarkDown” function converts the array which you get from “getProjectCollection” into a markdown file.

Conclusion

In this article, you have learned about GitLab Pipeline API. This article also provided information on GitLab, its key features, the usage of GitLab API, the definition of CI/CD Pipeline, and its working in GitLab. It also tells us how to trigger a pipeline status report using the GitLab Pipeline API.

Hevo Data, a No-code Data Pipeline, provides a consistent and reliable solution for managing data transfer between various sources and Desired Destinations with a few clicks. Connect with us today to improve your data management experience and achieve more with your data.

FAQs