Mobile Core Network: Automated Testing on Jenkins

Jenkins is the leading open source automation server which provides hundreds of plugins to support building, deploying and automating any project. In this article, we would like to take a closer look on how we significantly reduced testing times in our Jenkins setup.

emnify runs a custom-built mobile core network as software on Amazon Web Services. Since we have a complete software stack under our control, we implemented unique features giving our customers the power to configure mobile networks according to their needs through both, the Web UI and directly via the REST API.

Our platform provides connectivity for our customers' IoT devices distributed around the globe and used for a multitude of exciting use cases. Therefore, writing automated tests to ensure correct functionality is our daily bread and butter. The granularity of these tests allows us to deliver frequent updates without the need for routine manual testing. However, over the course of the last three years, since emnify was founded, our team created so many complex automated tests that the overall execution time grew to an unacceptable duration of more than 2 hours.

As we were no longer willing to wait that long for feedback after filing a pull request, we invested some time in improving our Jenkins test setup with the ambitious goal: running all tests within 30 minutes.

Problem Statement

emnify core application is based on Akka, a JVM-based implementation of the Actor Model. This application runs on multiple Akka nodes with different roles, implementing the functionalities that we need in order to run our mobile core network, like the Home Location Register (HLR) or gateways forwarding all user traffic.

The pipeline executing the test suite in Jenkins looks as follows - mind the annotated runtime:

• 1400+ unit and integration tests targeting only a single node with a runtime of ~15min
• 200+ end-to-end system tests with a runtime of ~105min, the longest of them taking 5-10min

The main reason for this enormous runtime of the system tests is that the different Akka nodes implementing various entities of the mobile core network are spun up dozens of times with different configurations. In addition to the end-to-end tests for our REST API, the usage of an emulator allows us to confirm the correct communication with other mobile network operators using protocols like MAP and GTP.

As these tests provide enormous value to our development team, we cannot simply remove any of them. The easiest option to reduce the time until we get feedback from Jenkins is to run these tests in parallel on multiple build agents.

Updates to the Jenkins Setup

While overhauling our Jenkins setup, we recently introduced the following changes:

• Our Jenkins master runs as Docker container in our existing AWS Elastic Container Service (ECS) cluster
• Build agents are launched by the Jenkins master on demand using the Amazon EC2 plugin
• Jenkins Pipelines allow us to define the build pipeline as code and version it together with the application. Also, new branches automatically appear as separate jobs in Jenkins. As enough has been said (including by ourselves) about Jenkins Pipelines already, we assume familiarity of the reader in the following.

This is the resulting pipeline, executing within roughly 5 minutes per stage:

• Retrieve the JUnit test results from one of the previous builds and order the list of tests descending by execution time:

• Set up n=3 buckets and assign tests in established sort order:

• Continue assigning tests one-by-one to the bucket containing the lowest runtime:

In case you find this interesting and like to drink coffee with us (or tea): we're hiring in our developers' HQ located in beautiful Würzburg.