NCCAOM Cancels February Exams, Takes Steps to Improve Security

On Feb. 8, 2006, after an extensive confirmation process revealed that the security of hundreds of test questions in its exam database had been compromised, the National Certification Commission for Acupuncture and Oriental Medicine (NCCAOM) took the unprecedented step of cancelling its entire series of February exams. In response, the NCCAOM has announced a series of steps designed to improve its database and examination procedures, ranging from new methods of exam administration to the retirement of all existing exam questions, which will be replaced with an entirely new set of questions for future examinations.

In an interview with Acupuncture Today, Dr. Kory Ward-Cook, the NCCAOM's chief executive officer, revealed that the commission learned of a breakdown in security not as the result of a computer-based attack or theft of the organization's database, but via an anonymous e-mail sent to the NCCAOM, and at least one other organization, the American Association of Oriental Medicine, in early February. More than 300 questions were included in the e-mail, with the majority of items coming from the NCCAOM's Biomedicine and Chinese Herbology exams.

"We feel it came from people memorizing questions, getting their lists together and circulating them," Ward-Cook said.

While the questions in the e-mail were not comprised of questions and answers identical to those on NCCAOM exams, they were written in such a way that the correct answer would be easy to determine. "It would give you enough information such that the person studying for the exam would be able to know what the question is and what the correct answer would be," Ward-Cook said.

Upon receiving the e-mail, NCCAOM staff began comparing the items in the e-mail with questions in the NCCAOM's exam database, a process that took two days to complete. "There was a huge percentage that overlapped," said Ward-Cook. "We couldn't be sure that this (e-mail) was just the first wave of questions coming to us. We had a very short window to respond, because the exams were originally scheduled to be February 18^th and 19^th. We had about seven or eight business days to deal with it."

After realizing the degree to which the NCCAOM's exams could have been compromised, the organization made the decision to cancel its regularly scheduled exam modules for the first time in the organization's history. The NCCAOM also announced that it would cease using all existing questions from its exam modules.

"We're going to retire thousands of (questions), when you total all of the exam questions from those modules," Ward-Cook explained. "We figured that if we were going to clean this up, we'd better just do it now and really go for it."

Over 1,000 candidates for the examination, ranging from current students to recent graduates to practitioners taking exams to obtain Diplomate status, were affected by the cancellations. As soon as the NCCAOM decided to cancel the February exams, it immediately began contacting candidates and members of the profession to inform them of the decision.

"We sent a blast e-mail to everybody who was scheduled to take an examination in February, we mailed a letter to everybody the second day, and then we called every single person on our list of those registered for the examinations," Ward-Cook said when asked about the NCCAOM's efforts to inform candidates. "If they weren't home, we left a voice mail."

For the people who may have been negatively affected by the February cancellations, the NCCAOM will provide assistance on a variety of levels, including:

• Writing letters of support. Many candidates had already made plans to travel to an exam site; in some cases, the combined costs of airline tickets and hotel reservations could amount to hundreds of dollars. To ameliorate some of those costs, the NCCAOM has offered to write letters on the candidates' behalf. "If they make a request, we'll write a letter to the airline, explain the situation, and try to help them get a refund," Ward-Cook said. She added that the NCCAOM would write letters to prospective employers and financial institutions to help defer student loan payments.
• Creating practice tests from previous questions and making them available through the NCCAOM's Web site. "We're going to build practice tests and have them up in April. They'll be online. It's something that candidates can go online and do," Ward-Cook said. "We're going to give candidates questions that have been in the bank that we're not going to use anymore."
• Waiving recertification fees. Diplomates are responsible for renewing their NCCAOM certification every four years. To renew their certification, diplomates must pay a recertification fee of $200 per certification program, but for candidates affected by the exam cancellations, those fees will be waived for the first recertification cycle. "We're going to waive the recertification fees when they come due," Ward-Cook said. "We won't be charging their recertification fee. They're going to get a voucher; we'll also keep a record in our database, too, in case they lose it."
• Offering free exam reports. Depending on a diplomate's status, fees for exam results can cost between $25 and $100. Similar to the waiver of recertification fees, the NCCAOM will provide free score reports for people originally scheduled to take their exams in February.
• Working with the Council of College of Acupuncture and Medicine to educate students about the consequences of revealing test questions. "Students sign a code of ethics on their application that they will not divulge the contents of the questions to anyone," Ward-Cook noted. "The colleges are going to work very closely with us to get that message out to the students and remind them."

To ensure that the new exam questions will differ substantially from the retired questions while still demonstrating relevance to a particular area of study, the NCCAOM will expand the number of people who serve on its exam committees. Committee members will also "get new training on how to write even better questions on the same content," Ward-Cook said.

In addition, the NCCAOM will hold a series of item-writing events throughout the country through March and early April. "We're going to be doing at least 12 item-writing workshops between now and the end of April," said Ward-Cook. "We're going to bring in a lot of draft materials to the examination committee members for them to review and edit."

The exact make-up dates for the February exams have yet to be scheduled. According to Ward-Cook, the NCCAOM expects to be able to hold the exams in early June, depending on how soon new exam questions can be written and validated.

"We're doing the exams as soon as we can in June," Ward-Cook said. "If it falls the last week in May, we would definitely do it in May. That's why we're not announcing the exact date yet. As we get closer into the process of getting these exam questions in house, then we can announce when we can give the examinations."

One thing the NCCAOM has announced is that it will switch from its current testing agency, Applied Measurement Professionals, to a new vendor, Pearson Vue, to administer all future exams. Asked why NCCAOM had decided to contract with a new testing agency, Dr. Ward-Cook noted that Pearson Vue has one of the higher standards of security and test administration standardization in the industry.

"AMP does not own their test centers. While they have protocols for the facilities, which are reasonably good, they don't have dedicated test centers that are owned and operated by their company. Another thing is that the new vendor employs their own proctors - they train them and they certify them. That means that at every test site, no matter where it's located, you can be assured that it's the same test center design and it will have proctors with standardized training."

"Pearson Vue has a higher level of service and security, and there's a systematic, consistent standard of proctors and training that is received," Ward-Cook added. "If someone's testing in Oregon, it's going to be the same as in New York, and the site's going to look identical, because the vendor owns those sites."

Perhaps the greatest advantage that Pearson Vue holds over AMP is the number of test sites available for candidates.

"We're going to go from approximately 40 domestic sites to over 200 domestic sites," Ward-Cook said. "One of the complaints that we have heard from candidates is that candidates have to travel far. My experience has been with Pearson Vue is that usually no one has to travel more than 50 miles to get to a test site ... There are going to be many more sites for candidates to choose from."

Asked to comment on the way the exam cancellation has affected her organization, and how the acupuncture profession has responded to the NCCAOM's decision, Dr. Ward-Cook offered the following observations.

"I think what this particular episode has done is bring us closer to the community," she said. "The staff and I have been able to talk to a lot of people personally and we have been able to know our candidates and our Diplomates better. It has also accelerated my getting to know some of the college officials a little bit better. Candidates are now getting a lot of one-on-one attention by staff.

"What I'm seeing is a pulling together of the whole community ... I'm pleased that people are responding and understanding and giving us the support that we need - all of the professional societies, the Council of Colleges, and many of the students and state agencies."

Profession Responds to NCCAOM's Decision

In light of the seriousness of the security breach and its impact on the acupuncture profession, Acupuncture Today contacted the American Association of Oriental Medicine, the Council of Colleges of Acupuncture and Oriental Medicine, and the National Acupuncture and Oriental Medicine Alliance to comment on the situation. Each organization's statement is printed below, in the order in which they were received.

Statement From Council of Colleges of Acupuncture and Oriental Medicine The unexpected cancellation of NCCAOM's February examinations was an unfortunate occurrence that created great concern on the part of NCCAOM, the Council's member colleges, and particularly those students that were scheduled to take the exams. The Council understands that the Commission cancelled these modules because it felt it had no other choice in light of its public mission as a national certifying agency for the AOM profession, the evidence that the questions for the February exam modules were compromised, and the Commission's significant uncertainty about the extent to which other questions in the Commission's exam database might also have been compromised. In addition to the Commission's immediate direct response to this situation, the Council has been working with the Commission to take additional steps to address the concerns of the colleges and students. These include the earliest possible announcement of the exact dates of the next test administration (no later than June of this year), Council participation on a task force of the Commission to provide input for the June exam, efforts by the Commission to assist graduating students directly with student loan issues, and an upcoming joint NCCAOM/CCAOM teleconference to provide further information about the June exam. The Council's executive committee has also been working closely with NCCAOM concerning the content of the Commission's FAQ Web domain and in ensuring that member colleges are kept informed of ongoing developments relating to the June exam. The Council believes that further investigation concerning the cause of the breach of exam security is warranted, and will be making a formal request to NCCAOM to this end. Although the cancellation of the February exam modules is very regrettable, it is important for all CCAOM colleges and their students to remain as patient as possible and to work through this challenge with the Commission in the most collegial manner possible. The Council believes that the Commission will make every effort to rectify the situation as soon as possible. As distressing as this situation has been, particularly for graduating students, it is a temporary challenge that can be overcome if everyone involved works together in a constructive and positive way. While the most expeditious administration of the next exam modules is critically important, so too is the collaborative and professional spirit in which that outcome is achieved.

Statement From the National Acupuncture and Oriental Medicine Alliance The board of directors of the AOM Alliance joins with many in our community in expressing its dismay at the need for cancellation of the NCCAOM examinations scheduled for February. We recognize the significant hardship this poses for those who had spent months preparing for these examinations. We offer our sympathy to these new professionals who stand poised at the threshold of practice and are inconvenienced by the delay in their certification examinations. We deplore the actions of those who compromised test security and caused this regrettable situation. It is with such individuals that blame and responsibility must lie. The board also offers its condolences to the leadership and staff of the NCCAOM as they address this situation. We encourage a vigorous effort to find and hold responsible those who have precipitated these events. We express our confidence that the ultimate outcome will be an improved and modernized testing procedure that will continue to ensure that our profession maintains the highest standards. Finally, the board offers its appreciation for the work of the NCCAOM and the patience of those who have been significantly inconvenienced. We recognize that a strong, secure and fair testing procedure remains an important guarantee in protection of the public health. Maintenance of appropriate standards of practice requires the certainty that practitioners have genuinely reached a level of competence that ensures both the safety and the efficacy of their practice. Reliance on the NCCAOM certification process has been an important component in winning public confidence and assisting the acceptance of acupuncture and Oriental medicine by the American public.

Statement From the American Association of Oriental Medicine The AAOM considers the recent breach of NCCAOM examination security to be very unfortunate. Yet, these examinations are essential to the legislative efforts that we work toward around the country. They assure entry-level standards when combined with accredited clinical training programs that provide the requisite clinical competencies. For example, the recent legislative victory in Michigan that creates a registered acupuncturist title relied heavily on the NCCAOM as a means of assuring public safety in addition to letters of support from the AAOM leadership. AAOM President Will Morris immediately contacted the NCCAOM's executive director in support of both the NCCAOM and the affected examinees. The AAOM is satisfied that the NCCAOM is doing everything in their power to rectify the situation quickly and to the best of their ability.

---

Editor's note: The quotes attributed to Dr. Ward-Cook in this article are excerpted from an interview conducted with Dr. Ward-Cook and Mina Larson in February. A complete transcript of the interview is available online.