AWS is considering adding three new types of questions to its certification exams – “Ordering”, “Matching”, and “Case Study”. But these are in early stages of testing and feedback so don’t expect any immediate changes yet. Good to be aware of what might be on the horizon though. So here goes.
You may be interested to know that AWS is experimenting with three new types of questions to add to their certification exams. They are tentatively calling them the following –
- Ordering
- Matching
- Case Study
These currently appear to be at an early stage of testing. Not so nascent that this is completely internal to AWS. They are willing to test them out publicly in the community. But these are not too ready for primetime either – they are not yet showing up in the real or beta exams, not even as “unscored test content”.
Having said that, below are some glimpses of what’s in store – should these become adopted as is.
Note that the very purpose of this public testing and survey is to gather feedback so that adjustments can be made, including shelving the plans if necessary. So take these with a pinch of salt – these may never appear on the real tests or show up in a substantially modified form. Only time will tell. Till then here a sneak peek to whet your appetite.
1. “Ordering” type questions
The name for this type of question is rather intuitive – you are presented with a set of options and your task is to pick the right ones from the list and then place them in a specific correct ORDER, as per the question statement.
Here are a couple of examples of how those questions might look like:
My Thoughts
- Choosing the right answers from a given list and then ordering them correctly – demands slightly more from the candidate than merely choosing the right options (multiple answer questions of today). This may be a good test of candidate knowledge but the questions will need to be designed all the more carefully and unambiguously. The same knowledge check could also be made using the current question format by re-wording the question, for instance, without needing this complexity.
- This may vary based on actual testing software used but the way they were presented to me on this tool – the longer answer choices are not fully visible in the dropdowns once we make a selection. Theoretically more than once option can have the same few words in the beginning – making it very confusing for candidates and needlessly wasting their time / causing errors unrelated to the subject matter.
- For the above reasons I felt that this new question type might bring more challenges than value – and may not be a significant net positive addition to the certification exams.
- However, as a way of comparison they showed one traditional way of working these ordering type knowledge checks using traditional multiple choice question format (shown below) which did seem tedious for the test-taker in its own way – being too verbose and repetitive.
2. “Matching” type questions
This type of question expects you to place various premises into matching categories from the choices given. These are also presented as drop-down choices, at least in the pilot testing tool I used.
Here are a couple of examples:
My Thoughts
- I thought calling this type of question “Matching” was not the most apt – not that it matters much to test-takers during the exam. I thought “ASSOCIATION type question” maybe a better name.
- Drop-downs may not be ideal – but not sure if there are better alternatives. The same challenge with long choices, and choices with similar initial wordings, etc. – could make the UI needlessly taxing for test-takers.
- In some other certification exams I have seen “drag-and-drop” or other UI means of posing a “matching” type of question. But I can see how that may not be great from accessibility perspective.
3. “Case Study” type questions
This is the simplest one to understand - where a scenario is described in a paragraph or two, followed by more than one standard types of questions all of which refer to the same scenario or “case study”.
Here are is an example illustrating this type of question set:
My Thoughts
- I felt that calling this (at least this example) as “case study” is a stretch. Usually case studies are multiple paragraphs describing a stuation in some detail before questions are posed. Something as simple as this example should simply be called a “Scenario” in my opinion.
Conclusion
Overall it is good to see that AWS is actively looking to keep exams up-to-date an relevant – not just in subject matter and job role coverage, not just in bringing new certifications to the collection – but also experimenting with various types of question formats.
But I am also aware that the last experiment with “labs” style questions that were introduced in the SysOps Administrator associate – remain on hold. So these are not always easy to get right. But the willingness to keep looking for various ways of improvement is commendable.
There were questions in the survey about the devices we used, and if we used any assistive technologies, etc – all of which are relevant. New question formats should not unintentionally disadvatage a section of test-takers for reasons unrelated to AWS knowledge and job-role related skills.
I am hoping that many diverse and representative test-takers from the community have been choosen to participate in this pilot and that they have provided thoughtful feedback to AWS that will elevate the testing experience and value for everyone.
Don’t ask me how I stumbled across this. I am guessing it was pure random chance.
The practice exam with sample questions of new type along with a survey questionnaie asking for feedback on those new question types – was available at this following URL (benchprep is also the platform that AWS now uses to offer official practice exam question sets for all its certification exams). No guarantee you will see it live there though – unless you were also randomly picked to participate: