Are Questions Ambiguous or Not?
The objection below expresses the sentiments of a number of correspondents:
Many questions are ambiguous, despite circulars and reports stating that the exams are not designed or intended to be so. I object to this and to the choice of many answers being so similar to the one correct answer.
There are several aspects to be borne in mind when considering this.
In a document providing an overview of the REs which was published in March this year, the FSB set out the development process as follows:
Appointment of Question Developers
The Examination Bodies appointed subject matter experts to develop the multiple-choice questions for the examinations. The question developers were not required to hold any specific qualification(s), but it was a requirement that they had to be regarded as subject matter experts in the field for which they were required to develop questions.
The Independent Examination Board (IEB) is an assessment body that is accredited by Umalusi, the South African statutory body responsible for quality assurance for school and adult assessments. The IEB training section, ASSET, provides training in assessment. They conducted training for all of the question developers of the various Examination Bodies. The purpose of this training was to ensure that the question developers used the same correct methodology when developing the multiple-choice questions.
Moonstone was tasked with developing the questions for the level 1 REs. To ensure that our developers were properly equipped, we arranged for additional training in the development of multiple choice questions with experts of the University of Stellenbosch.
The FSB provided the following background to the steps taken to quality assure the questions accepted for the examinations:
Quality assurance of questions
The IEB provided guidance in terms of the quality assurance processes that would be required when developing an examination. They were also appointed as the external moderator by the FSB.
They then provide a schematic overview of the process that was followed in the development of each and every question. Please see page 8 of the FSB Overview.
A very important step in the development process involves trials aimed at “field testing” the questions once it passed the internal moderation process. Each question has to be trialled in a group of FSPs not smaller than 30. Their feedback is then incorporated and the questions and possible answers amended where appropriate.
The next step in the process involves reviews and analysis. After a statistical analysis, the questions are sent to the Independent Examination Board who consolidates all the information gathered and makes recommendations based on its findings. Those questions found wanting have to be trialled again and again until it pass the test, or be withdrawn.
The reason for setting out the above in such detail is to show that actual FSPs were involved in the development process, and their feedback used to avoid ambiguity.
From dozens of conversations with candidates who wrote the exams, it appears that one likely cause for the perceived ambiguity stems from study material written in layman’s language while the questions are developed using the actual legislation.
Many correspondents indicate that they achieved high pass marks in the study material mock examinations, only to find that the questions in the REs were set at a substantially higher level.
It does appear that one’s preparation is not complete without spending sufficient time working through the Qualifying Criteria. These documents list all the knowledge and skills required, as well as where in the relevant legislation, one can find the answers.
Making and taking time to study the latest FSB Preparation Guide can save one a lot of unnecessary work and effort.
Extract from: Moonstone Investment Indicators 5 September 2011
