EMnify is hiring. Visit our careers page to see our open positions, and apply today!
    Dec, 3 2020

    EMnify Data Streamer integration for Google Cloud Pub/Sub

    integration_guides

    Stream connectivity metadata directly to Google Cloud Pub/Sub in real-time

    Before each device is able to send data to the application it needs to communicate with the mobile network to ensure that the data is allowed to be sent over the network. The EMnify DataStreamer makes connectivity metadata available in real-time to your cloud platform for detailed insights on the event and usage data of your devices. 

    Data Streamer with Google Cloud Platform

    Prerequisites

    • A device with a cellular modem and EMnify SIM card
    • A Google Cloud account

    Benefits 

    • connectivity metadata integrated into operational dashboards for service teams
    • allows faster triaging between device, connectivity and application issues
    • insights about network attach, data and roaming issues or when device is above service limit
    • view on service usage and cost per device 
    • directly delivered to Google Cloud without need for managing application servers

    Integration steps

    Hint: users who wish to use the gcloud SDK instead of the Google Cloud web console may skip to Configuring via gcloud SDK.

    To create a Pub/Sub topic and subscription, users need to have an associated project. We will create a project from scratch in this guide for illustrative purposes.
    In the Google Cloud console,

    • Create a new project:

    New project Google Cloud

    • Give the project the name EMnify Data. The project ID emnify-data is required in a later step
      Pub Sub menu Google Cloud
    • From the sidebar menu, navigate to the Pub/Sub service
    • Create a Pub/Sub topic called emnify-device-dataGoogle Cloud add EMnify
    • To allow for the EMnify data stream to be written to the topic, click + Add Member under the Permissions tabGoogle Cloud Data Streamer
    • Add the following member with a Pub/Sub Publisher role:
      pubsub@emnify-datastreamer.iam.gserviceaccount.com
      Google Cloud Pub Sub Publisher

    Adding Stream in the EMnify UI

    Log in to the EMnify portal and navigate to the Integrations page:

    Click Add New Data Stream and under Google Pub/Sub click ADD 

      • Enter the Cloud Pub/Sub Topic, in this example emnify-device-data
      • Enter the Project ID  into the "Cloud Pub/Sub Project containing Topic" field, in this example emnify-data

    Newly-created data streams are immediately active and should show a green Running status indicating a 200 HTTP response code when correctly configured.
    If the configuration is invalid, a warning message will be displayed indicating incorrect configuration. 

    Upon successful creation, event data arrives directly into Google Pub/Sub for further processing by other services:

    Google Cloud Pub Sub

    Configuring via gcloud SDK

    For users who already use the gcloud SDK, setting up the topic, adding EMnify publish permissions, creating a subscription and checking the incoming messages can be done directly from the CLI in the following four steps:

    # Create the topic 'emnify-device-data' and allow EMnify publish permissions
    gcloud pubsub topics create emnify-device-data
    gcloud pubsub topics add-iam-policy-binding emnify-device-data \
    --member='serviceAccount:pubsub@emnify-datastreamer.iam.gserviceaccount.com' \
    --role='roles/pubsub.publisher'

    # ! Note: the data stream must be activated via EMnify API at this step !

    # Create a subscription for the 'emnify-device-data' topic and pull published messages
    gcloud pubsub subscriptions create my_subscription --topic=emnify-device-data
    gcloud pubsub subscriptions pull my_subscription

    Verifying the integration

    When incoming messages are successfully sent to a Pub/Sub topic, clicking on view messages allows to preview incoming data.

    • Click View Messages and select Create a SubscriptionGoogle Cloud create a subscription
    • Give the subscription a name, ensure it is of Pull type to read incoming messages and click Next
      Google Cloud subscription topic
    • Click View Messages, select the newly-created subscription and click Pull to show latest published messages:
      Google Cloud messages

    More details of the events can be inspected by expanding the columns icon on the right and selecting message attributes to be previewed:Google Cloud attributes

    /bg-gradient-blue

    Subscribe to our Developer Newsletter