 |
The CMA Examination: Focus on Quality
by Roland L. Madison, Ph.D., CPA
Since 1990, I have made the trek 11 times from Cleveland,
Ohio, to Montvale, New Jersey (where the Institute of Management Accountants
is located), in order to participate in grading the CMA examination.
Upon reflection, I have seen a number of changes made in the CMA program,
intending to stimulate interest in becoming a Certified Management Accountant
and to enhance the quality of the CMA examination.
The Examination: Format and Content
The CMA exam is given semiannually in the months of June and December.
It is divided into the following four parts: Economics, Finance, and Management;
Financial Accounting and Reporting; Management Reporting, Analysis, and
Behavioral Issues; and Decision Analysis and Information Systems.
Currently, each part of the exam contains 25% objective response-type
questions, with the remainder having either computational, narrative-discussion,
or both types of response formats. Each of the latter-type questions (non-objective)
may vary from 12 to 21 points in value. Candidates are permitted to use
calculators, and may readily obtain booklets containing copies of prior
CMA examinations, including the unofficial answers, from the IMA's Publication
Department.
Finally, while the CMA exam is presently an open exam, the ICMA Board
of Regents has approved (May 1995) making the examination non-disclosed,
effective with the June 1997 exam. This action means that, after the December
1996 CMA Examination, candidates can no longer receive their exam booklets
after the test has been administered. This also means that the publication
of the Question and Unofficial Answer booklet ended with the December
1996 examination.
The Regents give several good reasons for this action, including cost
savings and permitting certain basic concepts to be repeatedly tested
without requiring the development of totally new questions. However, this
author is not a proponent of their actions. First, the ICMA will lose
the opportunity to provide potential CMA candidates, their employers,
and the general financial community with a full appreciation for the real
quality of the Institute's professional examination program. Providing
interested parties with a booklet containing a few sample questions simply
will not convey the true merits of this professional examination. Also,
the members of the academic community, who contribute to the CMA program
in a variety of manners, will not receive the same degree of recognition
for their efforts when the Question and Unofficial Answer booklet ceases
to be published. This may decrease the incentive to participate in the
development of exam materials, unless the fee structure for this service
is modified.
Quality Control: Exam Preparation
The process to create an examination is quite extensive. First, an outline
of the exam that is consistent with the Content Specification Outline
is prepared by the ICMA staff. Next, a question-concept (idea) outline
is developed for the specific questions on each part of the exam. These
are then sent to the Board of Regents (twelve members mainly at the executive
level in management accounting) for review and comment at least a year
prior to the scheduled exam date. If the general outline and question-concept
outline are approved, the ICMA staff begins the detailed preparation of
questions.
To digress briefly, both the concept for questions and fully developed,
detailed questions have three potential origins: they may be ICMA staff-prepared;
unsolicited questions may be submitted to the Institute; or the development
of questions, based on concepts approved by the Trustees, may be privately
contracted by the managerial staff of the ICMA. In recent years, greater
than half of the exam questions have come from sources beyond the ICMA
staff.
The contributor submits a question along with a suggested solution and
sources for the answer. The ICMA staff evaluates the question; and, if
it is selected for potential use, edits the question and assigns it to
one of the four examination parts. Next, the question is taken to an Examination
Review Committee (ERC), consisting of 3 regents and 3 current senior-level
management accountants in industry. The question is evaluated at a detailed
level for inclusion in the examination. After this detailed review, the
questions for each exam part are tested on a small sample of confidential
persons. After the test responses to the sample are evaluated, the ICMA
staff prepares a revision of the part for an additional--and hopefully
final--evaluation by the ERC. When a question successfully passes this
point, it will be scheduled for use on a future exam, where it meets one
of the specific requirements of the Content Specification Outline.
A Question is Accepted for the Exam
At some point in this evaluation process by the ERC (the exact point is
confidential), contributors normally receive a stipend and a letter of
thanks from Ms. Terri Funk, director of the CMA Examination. With the
exception of contracted questions, this tentative acceptance could be
a year or more after the question was submitted. In no case are contributors
told if or when their questions will appear on a future exam.
Prior to printing an examination, a question is sent to one or more outside
reviewers, in addition to an ICMA staff person, for one last review for
contemporary appropriateness and the reduction of any final ambiguities.
Quality in Exam Grading
Before production grading of the non-objective response-type questions
begin, both a narrative-solution outline of each question, along with
a point-specific grading guide must be developed. Using the solution prepared
by the contributor of the question as well as the solutions prepared by
the blind referees of the question, the ICMA management team develops
a grading guide and narrative solution for each question. Next, a group
of senior grades and grading supervisors use the staff-compiled solutions
to evaluate a sample of 100 papers from the current exam. The purpose
of this sample grading is to deal with any potential problem areas, and
to finalize the narrative answer sheets and grading guides to be used
by the production graders.
To become a CMA grader, a person must complete an application that includes
proof of their academic credentials, work experience, and professional
certifications held. Production graders are then placed into three levels
of experience--it takes about four years to reach the senior level. Graders
usually work in three- or four-person teams to deal with the increasing
number of papers to be evaluated. The ICMA staff tries to balance the
experience levels of each team, giving consideration to the complexity
of the question and the number of papers to be graded. Finally, every
effort is made to assign a grader to working in their first or second
preference area of expertise.
Prior to starting production grading, each grading team is introduced
and is given an orientation to the grading program by the Director. Next,
all graders, regardless of seniority level, review all procedural guidelines
(e.g. confidentiality, secure handling of papers, scoring, and reconciling
differences in point evaluations) before proceeding to the quality-control
pre-grading process.
Grading-Team Communications Ensures Consistency and Quality
The latter item is a very important change, recently implemented by the
ICMA, to ensure consistency and quality in the evaluation of all exam
papers. Each member of the grading team independently evaluates and scores
a selected group of exam papers. The team then meets to discuss their
evaluations in relation to the grading guide and to reconcile any differences.
Next, the entire team meets with part supervisors to discuss the results
of their evaluations. The supervisor compares their composite results
with the projected results that were developed during the sample grading
process. It is during these two meetings that a consensus is to be reached
among the graders and the supervisor that will ensure the uniform and
consistent evaluation of all CMA candidates' papers.
After all of this preparatory effort toward quality has been applied,
it is time for production grading to commence. During the grading process,
consistent quality is maintained by the supervisors, who continuously
review the work of graders. In addition to grader reviews by the supervisors,
the ICMA also carefully monitors the grading patterns of all co-graders
to see that a statistically significant difference does not develop during
the examination. If one does begin to develop, it will quickly be detected,
and a meeting with a supervisor, project manager, and the grading team
(or grader) will be held to analyze the cause of the disparity. The ICMA
may require a regrading of some group of papers if a disparity of more
than one point between graders persists during the first 30% of the exams
graded. By this point in the process, most co-graders have developed a
solid and consistent rapport in terms of the credit to be granted under
the majority of possible circumstances.
Serious Consideration Given to Marginal Papers
Much like the CPA exam, the ICMA follows a practice of not assigning marginal
scores within a certain point range. (Could we state what this range is?)
For this reason, every single question on each exam that received a grader
score of ZERO is regraded for the potential oversight of credit. Finally,
and perhaps of greatest significance, all papers with an exam-part score
of 50 percent, but less than 70 percent ( a passing score), are regraded
at least once, and potentially two times, before a final score is assigned.
Every effort is made to give the candidate complete credit in the grading
process for knowledge demonstrated.
Thoughts for Potential Graders to Consider
To efficiently process candidate's examinations, the ICMA staff requires
that production grading is completed during a period of less than a month.
For both graders and staff, this requires a significant commitment of
time and involves a lot of effort and concentration. While time commitment
and fatigue may be a factor for some persons doing production grading,
I found the ICMA staff to be most supportive of graders and cooperative
in setting schedules, allowing for personal breaks to relax, and creating
an overall comfortable working atmosphere.
From a personal standpoint, serving as a grader provides an opportunity
to make new friends and learn about our profession, as well as make a
commitment to serve it in some different fashion. Also, working with the
IMA provides one the opportunity to become more aware of the programs
the Institute has to serve its members.
Economically, graders from the local area may fare rather well if their
schedules are carefully planned. Even those experienced graders from out
of town may fare reasonably well if they work as a team. While not part
of the formal agenda, graders often do some sightseeing while visiting
the Big Apple area.
As potential CMA candidates, you should feel extremely confident that
the Institute takes great pride in the quality preparation of an examination
that is designed to evaluate your fundamental skills as a management accountant.
CMA candidates should feel most comfortable to know that the ICMA utilizes
all of the quality controls described herein to ensure the accurate evaluation
of your skills.
Conclusion: Take the Opportunity to Make a Commitment to Quality
I have a very positive view about the quality of the CMA program sponsored
by the Institute of Management Accountants. I would urge both practitioners
and academics to consider preparing a written contribution to the exam,
applying to serve as a CMA exam grader, or sitting for the CMA examination
as a candidate. Persons who hold the CPA certification are automatically
granted credit for Part 2 of the exam, and those who are employed in academia
may enjoy other benefits from involvement with the CMA program.*
Roland L. Madison is a professor of Accountancy at John
Carroll University, University Heights, Ohio. A member of the Cleveland
East Chapter of the IMA, Dr. Madison has served as a consultant, grader,
and contributor to the CMA examination program since 1987. This author
gratefully acknowledges the technical contributions that Terri L. Funk,
CMA, Director--Examinations, has made to this article.
Footnote, to go along with asterisk
* Persons interested in the CMA program may contact the ICMA at its toll-free
number, 1-800-638-4427, to obtain a CMA Information Kit, which contains
registration materials and the Content Specification Outline with a list
of suggested reading materials (e.g. texts, authoritative bodies, and
periodicals), to help the candidate prepare for the exam.
|
