Apache Spark Batch Processing: 5 Easy Steps

Apache Spark is an open-source and unified data processing engine popularly known for implementing large-scale data streaming operations to analyze real-time data streams. According to a report, Apache Spark is capable of streaming and managing more than 1 PetaBytes of data per day. Apache Spark not only allows users to implement real-time stream processing operations but also enables users to perform Apache Spark Batch Processing. Since Apache Spark natively supports both batch and streaming workloads, users can seamlessly process and analyze data using inbuilt processing libraries like Spark SQL, MLlib, and GraphX. 

In this article, you will learn about Apache Spark, batch processing, and how to implement Apache Spark Batch Processing using .NET. 

Table of Contents

• Prerequisites
• What is Apache Spark?
• What is Batch Processing?
• How to implement Apache Spark batch processing?
  • Downloading the Sample Data
  • Creating a Console Application
  • Creating a Spark Session
  • Preparing the Data
  • Analyzing the real-time data
• Conclusion

Prerequisites

A fundamental knowledge of data processing. 

What is Apache Spark?

Launched in 2014, Apache Spark is an open-source and multi-language data processing engine that allows you to implement distributed stream and batch processing operations for large-scale data workloads. In other words, Apache Spark is prominently known as a Distributed General-Purpose Computing Engine that is used to analyze and process massive data files from a variety of sources such as S3, Azure, HDFS, and others. Since Apache Spark is a multi-language data processing framework, you can customize and reuse code across a wide range of workloads like batch processing, interactive querying, machine learning, graph processing, and real-time analytics. 

What is Batch Processing?

Batch processing is used to deal with enormous amounts of data for implementing high-volume and repeating data jobs, each of which performs a specific operation without the need for user intervention. When sufficient computer resources are available, the batch processing process allows you to process and manage data with little to no user interaction. Furthermore, batch processing is critical in organizations and businesses in efficiently managing large volumes of data.

This technique is especially useful for repetitive and monotonous operations for supporting several data workflows. Since batch processing automates the workflows, it highly minimizes the possibility of manual errors or anomalies. Due to significant gains in precision and accuracy through automation, organizations can achieve superior data quality while decreasing bottlenecks in data processing activities.

How to implement Apache Spark Batch Processing?

In the further steps, you will learn how to implement Apache Spark Batch Processing using .NET. Initially, we have to create and run an Apache Spark application using .NET. Then, we will read data into a DataFrame and prepare it for analysis. Finally, we use Spark SQL to process the data. 

Before implementing Apache Spark Batch Processing activities, ensure you have a preconfigured Apache Spark environment. If this is your first-time using .NET for Apache Spark, check out the Get Started with .NET for Apache Spark tutorial to discover how to configure your environment and run your first .NET for Apache Spark application.

1. Downloading the Sample Data 

To implement Apache Spark Batch Processing operations with high-scale data, you can use data from the GHtorrent website. GHtorrent tracks all public GitHub events like a project, commit, and watcher information to continuously record the real-time events and their structure in databases. Data from various time periods are accessible as archives. Because the dump files are massive and take a lot of memory to process, we use a truncated version of the dump file, which can be downloaded from GitHub. You can download the truncated version of the dump file from this GitHub repository.

2. Creating a Console Application

• After downloading the sample data for Apache Spark Batch Processing, we have to create a console application in Apache Spark. Execute the following command to create a new console application.

dotnet new console -o mySparkBatchApp
cd mySparkBatchApp

• The above-given Dotnet command generates a new console application for Apache Spark Batch Processing. The -o parameter in the command creates a directory named mySparkBatchApp, in which your app will be saved and populated with the necessary files. The command cd mySparkBatchApp navigates to the app directory you just generated using the above command.
• In the next step of Apache Spark Batch Processing, you have to install Microsoft.Spark package to use .NET for Apache Spark in an application. Run the following command to install Microsoft.Spark package.

dotnet add package Microsoft.Spark

3. Creating a Spark Session

• Execute the following command to initialize the process of Spark Session creation which will be used in Apache Spark Batch Processing.

using System;
using Microsoft.Spark.Sql;
using static Microsoft.Spark.Sql.Functions;

• Then, incorporate the following code into your project’s namespace. The “s_referenceDate” parameter is later used in the program to filter the data based on date.

static readonly DateTime s_referenceDate = new DateTime(2015, 10, 20);

• To create a new SparkSession for Apache Spark Batch Processing, you have to add the following code inside the Main function. The SparkSession is the starting point for programming with the dataset and DataFrame APIs in Spark. You can utilize Spark and DataFrame functionality throughout your programme by invoking the spark object for Apache Spark Batch Processing.

SparkSession spark = SparkSession
     .Builder()
     .AppName("GitHub and Spark Batch")
     .GetOrCreate();

4. Preparing the Data

• In this step of Apache Spark Batch Processing, you have to prepare your data sample for implementing batch processing operations. Initially, you have to change the input file into a DataFrame, which is a distributed collection of data structured into named columns. Then, you have to define the columns of your data through the Schema parameter. Ensure that you change the path to the CSV file to the location of the GitHub data you downloaded previously.

DataFrame projectsDf = spark
    .Read()
    .Schema("id INT, url STRING, owner_id INT, " +
    "name STRING, descriptor STRING, language STRING, " +
    "created_at STRING, forked_from INT, deleted STRING," +
    "updated_at STRING")
    .Csv("filepath");
projectsDf.Show();

• Further, to delete rows with NA (null) values, use the Na method, and to remove specific columns from your data, use the Drop method. This helps you avoid errors when attempting to examine null data or columns that are irrelevant to your final analysis of Apache Spark Batch Processing.

// Drop any rows with NA values
DataFrameNaFunctions dropEmptyProjects = projectsDf.Na();
DataFrame cleanedProjects = dropEmptyProjects.Drop("any");

// Remove unnecessary columns
cleanedProjects = cleanedProjects.Drop("id", "url", "owner_id");
cleanedProjects.Show();

5. Analyzing the real-time data

• Now, you are all set to analyze the data. You can use Spark SQL, which enables you to run SQL queries on your data in order to carry out data analysis operations. To query all rows in a DataFrame, it’s common to utilize a combination of user-defined functions and Spark SQL. 
• To simulate standard SQL queries as similar as encountered in other sorts of database management applications, you can specifically call “spark.Sql.” You can also use popular SQL functions like GroupBy and Agg to precisely combine, filter, and analyze your data.
• Now, you are all set to analyze the GitHub data. Execute the following code to determine how many times each language has been forked in GitHub.

// Average number of times each language has been forked
DataFrame groupedDF = cleanedProjects
    .GroupBy("language")
    .Agg(Avg(cleanedProjects["forked_from"]));

• In the above code, the information is organized by language with the help of the GroupBy function, and then the average number of forks from each language is calculated using the aggregate function.
• To further proceed with data analysis operations for determining which languages are the most forked, execute the following block of code. After executing the above code, the average number of forks is sorted in descending order.

// Sort by most forked languages first
groupedDF.OrderBy(Desc("avg(forked_from)")).Show();

• Now in the process to make apache spark batch processing, execute the following code to determine how recently projects are updated or modified in GitHub.

spark.Udf().Register<string, bool>
(
"MyUDF",
(date) => DateTime.TryParse(date, out DateTime convertedDate) && (convertedDate > s_referenceDate));
cleanedProjects.CreateOrReplaceTempView("dateView"
);
DataFrame dateDf = spark.Sql(
    "SELECT *, MyUDF(dateView.updated_at) AS datebefore FROM dateView");
dateDf.Show();

• In the above code, you created a new user-defined function named MyUDF and compared it to a date called “s referenceDate,” which was declared earlier. The UDF is then called on each row of data in the dataset using Spark SQL to examine each project in the dataset.
• After implementing various data analysis operations, call the spark.Stop() function to end the SparkSession.
• To further transform the previous operations to build a .NET application, execute the following command.

dotnet build

• you have to use the spark-submit function to run your app and make sure to provide the appropriate paths to your Microsoft Spark jar file.

spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local /<path>/to/microsoft-spark-<spark_majorversion-spark_minorversion>_<scala_majorversion.scala_minorversion>-<spark_dotnet_version>.jar dotnet /<path>/to/netcoreapp<version>/mySparkBatchApp.dll

• After executing the above code, you successfully build a .NET application that implements end-to-end Apache Spark Batch Processing operations. 

Conclusion

In this article, you learned about Apache Spark, batch processing, and how to implement Apache Spark Batch Processing using .NET. However, you can also explore and try to implement different Apache Spark Batch Processing techniques like distributed processing, structured streaming, and graph processing in Apache Spark.

There are various trusted sources like Apache spark that companies use as it provides many benefits, but, transferring data from it into a data warehouse is a hectic task. The Automated data pipeline helps in solving this issue and this is where Hevo comes into the picture. Hevo Data is a No-code Data Pipeline and has awesome 100+ pre-built Integrations that you can choose from.

visit our website to explore hevo

Hevo can help you Integrate your data from 100+ data sources and load them into a destination to Analyze real-time data. It will make your life easier and data migration hassle-free. It is user-friendly, reliable, and secure.

SIGN UP for a 14-day free trial and see the difference!

Share your experience of learning about the Spark batch processing in the comments section below.