How EMnify contributes to AWS Certification development

As EMnify operates all its systems on AWS, many of our colleagues hold one or more AWS certifications. Becoming certified not only proves a certain level of knowledge, but also helps the individual to focus their learnings on our cloud platform of choice – as there is a lot to learn when it comes to AWS.

Last autumn I was about to take the last certification on Pro/Specialty level that I hadn't done so far – the Machine Learning Specialty.

Being happy about what I achieved; I was about to hunt for the next challenge – until I received an e-mail from the AWS certification team. There was this little checkbox that (I assume) many of us have ticked: Apply to join our Subject Matter Expert (SME) Program – an opportunity that I already received after passing my very first certification. Back then, I did not expect to hear from AWS, and I didn’t for more than two years (I was told that there is no official waiting time involved to become an SME, so this might be a lot quicker for you).

In this email, I was invited to apply to become an SME. Just a few hours after entering details about myself and explaining why I thought I would be a good SME for the Advanced Networking Specialty, I already received positive feedback. With that came the invitation to the first activity…

My first Item Development Workshop 

What people commonly call a question is called an item in certification terminology. Consequently, item development means creation and review of new questions for the certification exams.

In non-Corona times, we would have met for three days in the AWS Dublin office. Due to Corona, the workshop took place remotely with around a dozen participants – two-thirds from AWS and the rest external.

As one would expect, this all sounds easier than it is. And I can tell you: it was a lot harder than I imagined. For a Specialty level question, just asking what an AWS feature means is nowhere near enough. So, I ended up spending hours designing overly complex architectures (based on what we might have ended up with in our company – maybe), only to realize that nobody would ever understand the scenario.

In the peer-review part of the workshop, it quickly became apparent just how common overly ambiguous questions are. Almost every question circled back at least once to the author with suggestions for improvement. Only after comprehensive review by the group did an item start its journey through additional AWS-internal stages and finally into the exam.

Before you think that this was simple: keep in mind that every new question first enters an evaluation phase, during which it is not counted to the exam taker’s result. During this phase, statistical data is collected, and only when the question performs well does it end up counted in real exams. I was also told that the comments which one can give during the exam are all being read and considered in deciding for or against a question (while I must admit that I myself gave such feedback only during a beta exam a while ago).

After that workshop, three of my questions made it into an exam. Phew… one per day? I am sure I can become more effective in later workshops.

Job Task Analysis Workshops

After the interesting opportunity to contribute items, I was invited to a workshop entitled Job Task Analysis (JTA). As you can see in the following diagram, the JTA is where the whole exam development process starts - and interesting things are happening! This has been done so far only once when the Advanced Networking Specialty certification was created. Now, a little over three years later, everything must be re-evaluated to define how the certification will look during the next three years.

I was honored to be invited as a panelist to the JTA committee, again consisting of around a dozen AWS solution architects, Technical Field Experts and more - and just three external experts.

The JTA workshops were again held remotely over the course of a few weeks in March 2021, in total around 40 hours. And we really started with the fundamentals, including: Is the title ‘Advanced Networking’ still the right one, or should we give it a different one? 

For much of the workshop, we were discussing what typical job titles and responsibilities of the people in scope for the certification are and even more, what typical tasks they perform during work. What knowledge and what skills does the so-called Minimally Qualified Candidate (MQC) have - the person who is expected to just pass the certification. 

One artifact created after those hours and hours of group work and discussion include an updated version of the current Exam Guide, which outlines to exam takers what they are expected to know. In the new version created by us, this will also include the knowledge, skills, and abilities that we determined to demonstrate exam takers’ competency.

What was particularly interesting to me were the different views that people brought into the discussions. While we are cloud-natives and networking on AWS for me never involves connecting office networks or whole data centers to our VPCs, I understand that this is a frequent requirement for many AWS customers, with which Solution Architects are frequently confronted. Nevertheless, I felt the need to position myself (maybe a bit more than really needed) on the other extreme: less Direct Connect, more container networking, from ECS to EKS as well as cross-account connectivity. This is where I typically see challenges at EMnify.

On the less networking-technical side, it was an interesting experience to observe, how a certification is built. The entire process is facilitated by Nadine McBride, PhD, a Psychometrician. With her background and expertise, she very well explained to us networkers, what we do, what we should avoid etc. Overall, a very much data-driven approach to take decisions.

Fast forward a few weeks, all holders of the certification (which did not opt out from AWS communication) received an email with an invitation to the “Blueprint survey.” This survey was aimed at gathering their feedback on the content outline with the expected knowledge and skills that we created. Further, the feedback (aka. data) of survey participants helped to define the weight of different domains and thus the number of questions on design, operations, security, etc.

After having received feedback through hundreds of responses, we had two follow-up sessions to adjust and finally confirm our agreement with the created content outline.

What’s next?

First, existing questions must be evaluated regarding their fit for the new exam guide. Multiple Item Writing Workshops will take place to also create new questions. Finally, a new beta version of the networking specialty certification will be announced on the AWS Certifications page in a couple of months from now.

For myself, before my next workshop , I have it on my plate to renew my own Advanced Networking certification in September. Statistically, it is very unlikely that I would have to answer one of my own questions. Luckily, my peers have ensured that if this does happen, I will be able to understand my own words.

Final Words 

In case you want to learn more about what it means to be an SME for AWS certification, you can find some information as well as the training material online.

I want to thank EMnify for giving me this opportunity to contribute to AWS certifications.

I want to thank Nadine McBride, PhD, and Tim Barnosky from the AWS Certification team for their feedback on this blog post.