Skip to main content

Handling Streaming Messages with Cloud Pub/Sub

Hands-On Lab

 

Photo of Joseph Lowery

Joseph Lowery

Google Cloud Training Architect II in Content

Length

00:30:00

Difficulty

Beginner

One of the primary benefits of Cloud Pub/Sub is its ability to handle streaming data, as well as occasional and batch data. Streaming data could come from many sources, including multiple Internet of Things devices. In this hands-on lab, we’ll learn how to set up a Cloud Pub/Sub topic and subscription, simulate streaming data from traffic sensors, and pull multiple records of data from the subscription.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Handling Streaming Messages with Cloud Pub/Sub

Introduction

One of the primary benefits of Cloud Pub/Sub is its ability to handle streaming data, as well as occasional and batch data. Streaming data could come from many sources, including multiple Internet of Things devices. In this hands-on lab, we’ll learn how to set up a Cloud Pub/Sub topic and subscription, simulate streaming data from traffic sensors, and pull multiple records of data from the subscription.

Logging In to the Environment

  1. On the lab instructions page, right-click the Open GPC Console button.
  2. From the dropdown, select the option to open the link in a private browser window. (Note: Different browsers have different names for a private browser window. On Chrome, you'll choose Open Link in Incognito Window. If you're using Firefox, click Open Link in New Private Window. Etc.)
  3. On the Google sign-in page, enter the unique username you were provided on the lab instructions page. Click Next.
  4. Enter the unique password you were provided on the lab instructions page. Click Next.
  5. On the Welcome to your new account page, click Accept.
  6. In the Welcome L.A.! menu, check the box under Terms of service.
  7. Choose your country of residence, then click AGREE AND CONTINUE.

Create a Topic

  1. From the Google Cloud Platform dashboard, click the main navigation menu at the top left of the page.
  2. In the dropdown, select Pub/Sub > Topics.
  3. Click Create a topic.
  4. Enter a name for the topic, such as "la-streaming-topic".
  5. Leave the Encryption option at its default setting.
  6. Click Create Topic.

Create a Subscription

  1. Drill down into the topic we just created and click Create Subscription at the bottom of the page in the Subscriptions section.
  2. Enter a name for the subscription, such as "la-streaming-subscription".
  3. Set Delivery Type to Pull.
  4. Under Retain acknowledged messages, click the Enable option.
  5. Leave all of the other options as their defaults.
  6. Click Create.

Retrieve the Files

  1. Click the Cloud Shell icon at the top of the page.
  2. Click START CLOUD SHELL.
  3. In the Cloud Shell, run the following command to clone the GitHub repository:
    git clone https://github.com/GoogleCloudPlatform/training-data-analyst
  4. Change to the training-data-analyst/courses/streaming/publish directory.
    cd training-data-analyst/courses/streaming/publish
  5. Copy the data file from the Cloud Storage bucket.
    gsutil cp gs://cloud-training-demos/sandiego/sensor_obs2008.csv.gz .
  6. Click the pencil icon to open the Cloud Shell code editor.
  7. Review the send_sensor_data.py file in the pubsub-streaming-lab folder.
  8. On line 25, change the TOPIC variable from sandiego to your topic name.
  9. Click File > Save.

Stream Sample Data

  1. Authenticate the shell with the following code:
    gcloud auth application-default login
  2. Enter Y to continue.
  3. Click the generated link to confirm the authentication.
  4. Select the cloud_user Google account, then click Allow.
  5. Click the clipboard icon to copy the generated code to your clipboard.
  6. Return to the shell window.
  7. Paste in the code we just copied, and press Enter to run it.
  8. Execute the following command to install the Google Cloud Pub/Sub library: sudo pip install -U google-cloud-pubsub
  9. Execute the following code to simulate streaming data: ./send_sensor_data.py --speedFactor=60 --project=[PROJECT_ID]
  10. Click the + icon at the top of the Cloud Shell to create a new Cloud Shell instance.
  11. Change to the working directory.
    cd training-data-analyst/courses/streaming/publish
  12. Pull messages from the subscription with the following command:
    gcloud beta pubsub subscriptions pull --auto-ack [SUBSCRIPTION_NAME] --limit=25

Conclusion

Congratulations, you've successfully completed this hands-on lab!