bg-gradient-blue
Dec, 4 2020

EMnify DataStreamer integration into AWS Kinesis and S3

integration_guides

Stream connectivity metadata directly to AWS S3 or Kinesis in real-time

Before each device is able to send data to the application it needs to communicate with the mobile network to ensure that the data is allowed to be send over the network. This communication happens through signaling events which is usually hidden for the application. 

With the EMnify DataStreamer this connectivity metadata is made available real-time to your cloud platform for detailed insights on event and usage data of your devices. 

UI screenshot for Kinesis&S3 blurred

 

Prerequisites

  • A device with a cellular modem and EMnify SIM card
  • An AWS account 

Benefits 

  • connectivity metadata integrated into operational dashboards for service teams
  • allows faster triaging between device, connectivity and application issue
  • insights about network attach, data and roaming issues or when device is above service limit
  • overview of service (data, SMS) usage and cost per device 
  • intra-cloud delivered to Kinesis and S3 without need for managing public IPs

S3 Integration Steps

1. In the AWS console, navigate to the S3 service and create a bucket
2. In IAM -> Policies click Create Policy to create a policy which allows PutObject permissions to the S3 bucket.
3. In IAM -> Roles, click Create Role for the S3 use case and click Next: Permissions
4. Attach the policy created in step 2 and click Next: Tags, then Create Role.
(Important: Make note of the new Role's ARN as it will be needed for adding a new data stream in the EMnify UI)
5. Edit the newly-created role and click Trust Relationships -> Edit Trust Relationships
6. Copy the policy document JSON below and replace your organisation ID in after org- to allow write access:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::884047677700:role/datastreamer"
]
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "org-1234"
}
}
}
]
}

Adding Stream in the EMnify UI

In to the EMnify portal, navigate to Technical Settings to create, inspecting and manage Data Streams:

  • Click Add New Data Stream and under AWS S3 click ADD
  • Enter the details for the S3 bucket:
    • User Role ARN - the Role ARN created at Step 4 above
    • Region - the region of the S3 bucket, this example uses eu-central-1 (Frankfurt)
    • S3 bucket - the name of the S3 bucket to deliver data to, this example uses emnify-data

Adding Stream for S3 adjusted

Newly-created data streams are immediately active and should show a green Running status indicating a 200 HTTP response code when correctly configured.
If the configuration is invalid, a warning message will be displayed indicating incorrect configuration.

Connectivity data as .CSV files directly in S3

When using the S3 integration, platform details arrive in the destination bucket as .csv files which can be directly read by other AWS services:

Kinesis integration steps

Data streams can also be directly ingested by AWS Kinesis which then allows for real-time analytics on streaming data.

1. In the AWS console, navigate to the Kinesis service and create a stream:

2. In IAM -> Policies click Create Policy which allows PutRecord and PutRecords write permissions to the Kinesis stream.

3. In IAM -> Roles, click Create Role for Kinesis Analytics and click Next: Permissions
4.
Attach the policy created in step 2 and click Next: Tags -> Create Role.
(Important: Make note of the Role's ARN as it will be needed for adding a new data stream in the EMnify UI)
5. Edit the newly-created role and click Trust Relationships -> Edit Trust Relationships
6. 
Copy the policy document JSON below and replace your organisation ID in after org- to allow write access:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::884047677700:role/datastreamer"
]
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "org-1234"
}
}
}
]
}

Adding Stream in the EMnify UI

  • In the EMnify portal, navigate to Technical Settings.
  • Click Add New Data Stream and under AWS Kinesis click ADD
  • Enter the details for the Kinesis Stream:
    • User Role ARN - the Role ARN created at Step 4 above
    • Region - the region of the stream, this example uses eu-central-1 (Frankfurt)
    • Kinesis stream - the name of the kinesis stream to deliver data to, this example uses emnify_usage_data

Adding Stream for AWS Kinesis adjusted

Verifying the integration

The data stream should show a 200 HTTP response code in the EMnify portal when correctly configured. In the AWS console, navigate to the 'Monitoring' tab of the kinesis stream. The number of records ingested into the stream via PUT requests should be visible in the lower half of the monitoring console:

Next Steps

As connectivity data is now arriving in real-time, it's simple to implement other services which can react to any events which interest us. One useful integration that can be quickly launched is a simple notification system which delivers Slack alerts based on the contents of platform events.

To get started with this example integration, see the Slack notifications via AWS Lambda article or directly launch the project using a cloudformation template provided in the EMnify Codelab GitHub respository.