Stream connectivity metadata directly to AWS S3 or Kinesis in real-time

Before each device is able to send data to the application it needs to communicate with the mobile network to ensure that the data is allowed to be send over the network. This communication happens through signaling events which is usually hidden for the application. 

With the EMnify DataStreamer this connectivity meta data is made available real-time to your cloud platform for detailed insights on event and usage data of your devices. 

Prerequisites

  • A device with a cellular modem and EMnify SIM card
  • An AWS account 

Benefits 

  • connectivity metadata integrated into operational dashboards for service teams
  • allows faster triaging between device, connectivity and application issue
  • insights about network attach, data and roaming issues or when device is above service limit
  • overview of service (data, SMS) usage and cost per device 
  • intra-cloud delivered to Kinesis and S3 without need for managing public IPs

Integration Steps

To set up a real-time stream into AWS S3 or Kinesis, log in to the EMnify portal and navigate to the Technical Settings page.
A UI panel for creating, inspecting and managing Data Streams is shown under the Application Tokens panel:

Data Stream with S3 integration

1. In the AWS console, navigate to the S3 service and create a bucket
2. In IAM -> Policies click Create Policy to create a policy which allows PutObject permissions to the S3 bucket.
3. In IAM -> Roles, click Create Role for the S3 use case and click Next: Permissions
4. Attach the policy created in step 2 and click Next: Tags, then Create Role. Edit the newly-created role and click Trust Relationships -> Edit Trust Relationships
5. Copy the policy document JSON with your organisation ID to allow role write access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::884047677700:role/datastreamer"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "org-1234"
        }
      }
    }
  ]
}

Newly-created data streams are immediately active and should show a 200 HTTP response code when correctly configured.
If the configuration is invalid, 500 errors will be displayed in the status column of the Data Stream.

Connectivity data as .csv files directly in S3

When using the S3 integration, platform details arrive in the destination bucket as .csv files which can be directly read by other AWS services:

Data Stream with Kinesis integration

Data streams can also be directly ingested by AWS Kinesis which then allows for real-time analytics on streaming data.

1. In the AWS console, navigate to the Kinesis service and create a stream:

2. In IAM -> Policies click Create Policy which allows PutRecord and PutRecords write permissions to the Kinesis stream.

3. In IAM -> Roles, click Create Role for Kinesis Analytics and click Next: Permissions
4.
Attach the policy created in step 2 and click Next: Tags -> Create Role.
5. Edit the newly-created role and click Trust Relationships -> Edit Trust Relationships
6. 
Copy the policy document JSON with your organisation ID to allow role write access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::884047677700:role/datastreamer"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "org-1234"
        }
      }
    }
  ]
}

Verifying the integration

The data stream should show a 200 HTTP response code in the EMnify portal when correctly configured. In the AWS console, navigate to the 'Monitoring' tab of the kinesis stream. The number of records ingested into the stream via PUT requests should be visible in the lower half of the monitoring console:

 

Next Steps

As connectivity data is now arriving in real-time, it's simple to implement other services which can react to any events which interest us. One useful integration that can be quickly launched is a simple notification system which delivers Slack alerts based on the contents of platform events.

To get started with this example integration, see the Slack notifications via AWS Lambda article or directly launch the project using a cloudformation template provided in the EMnify Codelab GitHub respository.