2 Critical GCP Message Queue Deployment Processes Explained

Google Cloud is a suite of Cloud computing services that works on the same infrastructure that Google leverages internally for its end-use products such as Gmail, Google Search, and Google Drive, to name a few. Its strength lies in Artificial Intelligence, Big Data processing tools, and Machine Learning initiatives along with container support.

This blog talks about the two methods you can leverage to deploy the GCP Message Queue for your Data Pipeline, namely, Cloudtasks and Pub/Sub. It also gives a brief introduction to Google Cloud Platform before diving into the nitty-gritty of Google Message Queue.

What is GCP?

Google Cloud Platform provides the same virtual machine functionality and core data storage as Azure and AWS, or any other cloud service provider.

What is Google Cloudtasks?

With Google Cloudtasks you can separate pieces of work that can be executed independently, outside of your main application flow, and send them for processing.

What is Google Pub/Sub?

Similar to Google Cloudtasks, Pub/Sub is a reliable and highly-available messaging service that can be used to send messages between applications.

Methods you can use to deploy the GCP Message Queue

• Method 1 – GCP Message Queue Deployment: Using Google Cloudtasks
• Method 2 – GCP Message Queue Deployment: Using Google Pub/Sub

GCP Message Queue Deployment: Using Google Cloudtasks

You can create a general workflow for GCP Message Queue using Google Cloud Tasks as follows:

• First, you need to generate a worker to process the tasks.
• Next, you need to create a GCP Message queue.
• After you’ve created a GCP Message queue, you can create the tasks programmatically and add them to the GCP Message queue.
• The Cloud Tasks service returns an OK to the originating application. This shows that the task has been successfully written to Cloud Tasks storage, making the create task request both highly durable and available.
• Next, the tasks get passed to the worker who will then process them.
• Finally, to complete the sequence, the worker will return a 2xx success status code to the Cloud Tasks service.

Here are the steps you can follow to deploy the GCP Message Queue by leveraging Google Pub/Sub

Step 1: Getting Started

• first, you need to download and unzip the source repository for this process, or you can simply clone it by leveraging Git, with the following command:

git clone https://github.com/spring-guides/gs-messaging-gcp-pubsub.git

• Next, you need to cd into gs-messaging-gcp-pubsub/initial.

Step 2: Adding Required Dependencies

• To add required dependencies to GCP Message Queue through Maven, you need to add the following snippet to your pom.xml file:

<dependencies>
    ...
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-gcp-starter-pubsub</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.integration</groupId>
        <artifactId>spring-integration-core</artifactId>
    </dependency>
    ...
</dependencies>

• If you’re using Maven, it is highly recommended to leverage the Spring Cloud GCP bill of materials to handle the various versions of your GCP Message Queue dependencies:

<properties>
    ...
    <spring-cloud-gcp.version>1.2.5.RELEASE</spring-cloud-gcp.version>
    ...
</properties>

<dependencyManagement>
    <dependencies>
       ...
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-gcp-dependencies</artifactId>
            <version>${spring-cloud-gcp.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
        ...
    </dependencies>
</dependencyManagement>

Step 3: Setting up Google Cloud Pub/Sub Environment

• To set up the Google Cloud Pub/Sub environment for GCP Message Queue, you will require a subscription and a topic to receive and send messages from Google Cloud Pub/Sub. You can generate them in the Google Cloud Console, or programmatically with the help of the PubSubAdmin class.
• For this example, you need to create a topic named ‘testTopic’ and a subscription for that topic named ‘testSubscription’.

Step 4: Creating Application Files

• To create application files for GCP Message Queue, you’ll require a class to incorporate the channel adapter along with GCP Message Queue’s configuration. Generate a PubSubApplication Class with the @SpringBootApplication header, as is usually seen with a Spring Boot application:

src/main/java/hello/PubSubApplication.java
@SpringBootApplication
public class PubSubApplication {

  public static void main(String[] args) throws IOException {
    SpringApplication.run(PubSubApplication.class, args);
  }

}

• On top of this, since you are creating a web application, you can make a WebAppController class to differentiate controller and configuration logic.

src/main/java/hello/WebAppController.java
@RestController
public class WebAppController {
}

• Next, you need to add the two missing files for HTML and properties:

src/main/resources/static/index.html
	<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Spring Integration GCP sample</title>
</head>
<body>
<div name="formDiv">
  <form action="/publishMessage" method="post">
    Publish message: <input type="text" name="message" /> <input type="submit" value="Publish!"/>
  </form>
</div>
</body>
</html>

src/main/resources/application.properties
#spring.cloud.gcp.project-id=[YOUR_GCP_PROJECT_ID_HERE]
#spring.cloud.gcp.credentials.location=file:[LOCAL_FS_CREDENTIALS_PATH]

•  This would allow the Spring Cloud Core Boot Starter to auto-configure these two properties and make them optional.
• Properties from the properties file always take precedence over the Spring Boot configuration. The Spring Cloud Core Boot Starter is lumped with the Spring Cloud GCP Pub/Sub Boot starter.

Step 5: Creating an Inbound Channel Adapter

• An inbound channel adapter listens to messages from a Google Cloud Pub/Sub subscription and forwards them to a Spring channel within an application.
• Instantiating an inbound channel adapter needs a PubSubTemplate instance along with the name of an existing subscription. PubSubTemplate is Spring’s abstraction to subscribe to Google Cloud Pub/Sub topics. The Spring Cloud Pub/Sub Boot starter offers an auto-configured PubSubTemplate instance which can be inserted as a method argument.

src/main/java/hello/PubSubApplication.java
  @Bean
  public PubSubInboundChannelAdapter messageChannelAdapter(
    @Qualifier("pubsubInputChannel") MessageChannel inputChannel,
    PubSubTemplate pubSubTemplate) {
  PubSubInboundChannelAdapter adapter =
    new PubSubInboundChannelAdapter(pubSubTemplate, "testSubscription");
  adapter.setOutputChannel(inputChannel);
  adapter.setAckMode(AckMode.MANUAL);

  return adapter;
  }

•  The message acknowledgment mode gets set in the adapter to automatic, by default. You can override this behavior, as shown in the instance given below.
• After the channel adapter is instantiated, an output channel where the adapter sends the received messages also needs to be configured.

src/main/java/hello/PubSubApplication.java
  @Bean
  public MessageChannel pubsubInputChannel() {
  return new DirectChannel();
  }

• Next, you can observe that attached to an inbound channel is a service activator which can be used to process incoming messages.

src/main/java/hello/PubSubApplication.java
  @Bean
  @ServiceActivator(inputChannel = "pubsubInputChannel")
  public MessageHandler messageReceiver() {
  return message -> {
    LOGGER.info("Message arrived! Payload: " + new String((byte[]) message.getPayload()));
    BasicAcknowledgeablePubsubMessage originalMessage =
    message.getHeaders().get(GcpPubSubHeaders.ORIGINAL_MESSAGE, BasicAcknowledgeablePubsubMessage.class);
    originalMessage.ack();
  };
  }

• The ServiceActivator input channel name needs to be the same as the input channel method name.
• Whenever a new message shows up in that channel, it gets processed by the returned MessageHandler.
• In this instance, the message is processed simply by logging its body and acknowledging it. In the event of manual acknowledgment, a message is acknowledged through the BasicAcknowledgeablePubsubMessage object, which is usually attached to the Message headers and can be easily extracted by leveraging the GcpPubSubHeaders.ORIGINAL_MESSAGE key.

Step 6: Creating an Outbound Channel Adapter

• An outbound channel adapter will listen to new messages from a Spring Channel and publish them to a Google Cloud Pub/Sub topic.
• To instantiate an outbound channel adapter, you need a PubSubTemplate along with the name of an existing topic.
• PubSubTemplate is Spring’s abstraction to publish messages to Google Cloud Pub/Sub topics. The Spring Cloud GCP Pub/Sub Boot starter offers an auto-configured PubSubTemplate instance.

src/main/java/hello/PubSubApplication.java
  @Bean
  @ServiceActivator(inputChannel = "pubsubOutputChannel")
  public MessageHandler messageSender(PubSubTemplate pubsubTemplate) {
  return new PubSubMessageHandler(pubsubTemplate, "testTopic");
  }

• You can leverage a MessageGateway to write messages to a channel and publish them to Google Cloud Pub/Sub.

@MessagingGateway(defaultRequestChannel = "pubsubOutputChannel")
  public interface PubsubOutboundGateway {

  void sendToPubsub(String text);
  }

• From this code snippet, Spring can auto-create an object that can then be auto-wired into a private field within the application.

  @Autowired
  private PubsubOutboundGateway messagingGateway;

Step 7: Adding Controller Logic

• Next, you need to add logic to your controller that allows you to write to a Spring channel:

src/main/java/hello/WebAppController.java
package hello;

import hello.PubSubApplication.PubsubOutboundGateway;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.servlet.view.RedirectView;

@RestController
public class WebAppController {

  // tag::autowireGateway[]
  @Autowired
  private PubsubOutboundGateway messagingGateway;
  // end::autowireGateway[]

  @PostMapping("/publishMessage")
  public RedirectView publishMessage(@RequestParam("message") String message) {
  messagingGateway.sendToPubsub(message);
  return new RedirectView("/");
  }
}

Step 8: Authentication

• You can choose to run the application from the command line with Maven or Gradle.
• You can also construct a single executable JAR file that possesses all the required classes, dependencies, and resources you would need to run it seamlessly.
• By building an executable JAR file, you can make it easier to version, ship, and deploy the service as an application throughout the development lifecycle, spanning different environments, and much more.
• If you choose to use Gradle, you can run your application by leveraging ./gradlew bootRun.
• Alternatively, you can also build the JAR file by using the ./gradlew build command followed by running the JAR file, as follows:

java -jar build/libs/gs-messaging-gcp-pubsub-0.1.0.jar

•  If you opt for Maven, you can run the application by leveraging ./mvnw spring-boot:run command. On the other hand, you can also build the JAR file with the help of the ./mvnw clean package followed by running the JAR file, as follows:

java -jar target/gs-messaging-gcp-pubsub-0.1.0.jar

• Finally, the logging output is depicted. The service should now be up and running within a matter of seconds. 

Step 9: Testing the Application

• Once the application is up and running, you can test it out. Open up http://localhost:8080, pen down a message in the input text box, and press the “Publish!” button. You also need to verify that the message was correctly logged within your process terminal window.

Conclusion

1. This blog talks about the different salient aspects of Google Message Queue
2. Different methods you can implement to deploy it for your Data Pipeline.
3. It includes the salient differences between Google Cloud Tasks and Google Pub/Sub as well.