After learning through an anonymous e-mail that the security of its testing database had been compromised, the National Certification Commission for Acupuncture and Oriental Medicine (NCCAOM) began a process of updating the security and procedures for certification exams.1
Within four months, new test questions were written, security measures were put into place, and a new testing agency was commissioned.2 After the June 2006 testing had concluded, AT contacted NCCAOM CEO Dr. Kory Ward-Cook for an update on the situation.
Acupuncture Today (AT): After all the changes with the testing procedures, how are you adapting?
Dr. Kory Ward-Cook (KW): Over 1,450 candidates successfully took the June 2006 exams. The Pearson VUE and NCCAOM staff worked together to register candidates from February's cancelled test administration along with candidates for the June examinations. As a result, when compared to the October 2005 administration, there was a 100 percent increase in the number of examinees for June 2006.
Our staff was fully prepared and trained for the new testing administration and registration process. Every staff member was on board to help candidates through the examination process during the week of registration. Prior to registration each candidate received information on our new procedures ranging from continuous updates on our website to personal letters and e-mails with specific details for preparing for and registering for the June examinations. Staff was continuously informed about all events relating to the June examinations. Senior staff was available at all times to answer questions and to handle candidates with special needs. Other stakeholders, including school personnel, were informed about the new testing procedures throughout the process, and we will continue to work with them to obtain feedback about their students' experience with the new registration process and testing on computers. All candidates had an opportunity both during their examinations and later if they chose to make suggestions and to report their concerns to NCCAOM. Our organization is already looking at changes to make the process easier and more user-friendly for future candidates including the elimination of deadlines through the implementation of year-round testing in 2007.
More recently, our staff has been busy developing a new user-friendly NCCAOM Candidate Handbook which will be available next month. In addition, for the first time, the Candidate Handbook will also be available online. Future plans include offering applicants the opportunity to apply online.
We will continue to maintain the special e-mail hotline that we initiated in February for candidates who have any questions or concerns with any component of the application or examination process. This hotline is managed by a senior staff member who responds within 24 hours to the candidates' e-mails.
AT: Were there any complications with the new process?
KW: Overall the transition to computerized testing went very smoothly. Most of our applicants were able to register with a minimum of hassle and had very few problems mastering the new format. The AOM community has always been quick on the uptake and took to the new format very easily. However, as in the implementation of any new program, there are always a few complications (or there is always the unforeseen complication). One complication included a data transfer problem that affected some candidates. As soon as we realized the problem, our staff immediately assisted those candidates who were affected to make certain that they were able to register without incident. Because more than 4,500 exams were given during this two-week window, some testing sites were overloaded. As a result, we worked with Pearson VUE to immediately add more sites. We take pride in the fact that all applicants who wished to test were able to find a date and time, although it may have not been their first choice of date and location.
AT: What type of response did you receive from students who took the tests?
KW: As with every examination administration, we received glowing reports and we received complaints. There was nothing outside of our usual response from candidates, which was a pleasant surprise. Many candidates took the exams and were excited about the opportunity to take the first computerized exams, and there were others who complained about various facets of the new examination process. Our testing staff handled every complaint. Each complaint was thoroughly reviewed and warranted changes were made. The complaint process involves sending a written letter or an e-mail to the candidate support line at candidatesupport@nccaom.org. We very much appreciate those who took the time to comment on the examinations and the registration process.
AT: Were there different variations of the test?
KW: Good security always includes more than one version of an examination. There were different versions of our examinations and the separate versions were also different from the other. However, the content of the examinations did not change. We are bound by our examination content outline which is published on the website, www.nccaom.org/examoutlines.htm, and in the Candidate Handbook. We were pleased with the results of this examination administration and believe that the new system made it extremely difficult, if not impossible, for an individual to cheat.
AT: When will the pass/fail rates be published?
KW: The pass/fail decisions will be available on the NCCAOM website after July 15, 2006. All candidates will be able to log on and check their status via the website. In addition, examination results will be e-mailed to the designated contact for those states who signed up for this option.
AT: Were any areas of focus added or deleted?
KW: As stated previously, we are constrained from changing the focus of our examinations from the published examination content outline which is determined by periodic job task analyses. The content is only changed as a result of this important validity study. Rigorous attention is paid to the content outline so that we can be certain that our examinations are fair, valid and reliable.
AT: What was your process to determine the new test questions?
KW: One of our goals is to educate our candidates, diplomates and school officials about the process of putting an exam item (question) together. We have outlined this process in detail in the summer edition of our newsletter, The Diplomate, and we will also be continuing to inform the AOM community about this process. The process to develop an examination usually takes a minimum of 18 months to complete. However, due to the security breach in February and the necessity to retire every item in our item bank, we were forced to expedite the process. Because of the phenomenal efforts put forth by the testing department staff and our volunteers, we were able to complete this process in less than four months, which we knew we had to do in order to meet the next examination administration date of June 2006.
Our examination development involves a process that all certification agencies must follow. It begins with the Job Analysis process which had been completed previously. The next step in the process is item or question writing, which is done by NCCAOM diplomates. Fourteen item-writing events were held throughout the country, and as a result, we were able to obtain the necessary items for the next step. The commissioners and staff want to thank all the dedicated individuals who contributed items during these events. We are overwhelmed by the contribution and the concerns of the AOM community with regard to this situation. It was amazing how many practitioners, school officials and faculty pulled together to work with us to produce a comprehensive set of examinations for the June administration.
After the itemÐwriting process is completed, each item is reviewed by staff for item construction, then by subject-matter experts (specialists) on Exam Development Committees for content validity. The items are then reviewed again by different subject-matter experts in conjunction with a final review by the Exam Development Committee chair and board liaison.
AT: How smooth was the registration process?
KW: This first registration under the new system was a learning process for us and we were extremely pleased that over 1,900 candidates were able to register successfully. That number represents is a total of 5,500 examination modules registered in one administration. Approximately 400 candidates subsequently changed their registrations to wait for the October exam administration.
We had the largest number of applicants registering in a single administration in the history of the NCCAOM. Since we knew, in advance, that we had over 2,000 candidates eligible to register, we prepared the staff and kept the lines of communications open with the candidates by sending them e-mails and letters informing them of the registration process. We also had the candidate support line available on a 24-hour basis the weekend before registration.
As stated previously, every staff member was on hand to answer questions and we imposed a blackout period for vacations during the registration period in order to have every member of the NCCAOM staff available to answer phones and assist the candidates the first few days of registration.
Pearson VUE had 100-plus representatives ready to register approved applicants. With any new exam administration there are often bumps in the road, but because we had trained staff and good support from Pearson VUE, we were able to immediately respond when a candidate had a problem. Some candidates experienced a delay when they called in that first morning of registration. This was due to the fact that over 1,900 candidates were trying to call at the same time, which jammed the lines, resulting in increased waiting time for some candidates. Those candidates who called were helped immediately by staff that personally assisted them with the registration process, often with a member of the Pearson VUE staff on the phone at the same time. Customer service is one of this year's primary goals, and we believe that time and time again, our staff rose to the occasion. Every member of our staff stopped their normal daily duties to assist the candidates that needed our help.
One area that was new for us and for our applicants was online registration, which was offered for the first time in June. Candidates were given a two-week window to take the exams. In the past, everyone had to take the exams in a period of one to two days and there was no immediate registration confirmation. The new system tells you immediately what, where and when. Although some candidates did have problems with the online registration, most candidates were able to use it successfully.
We recognize that this registration process was a unique one for our candidates in that there were candidates from two administration periods registering at the same time. The next group will not have this experience again, as the number of applicants registering will decrease by 50 percent for the October 2006 examination administration. In addition, since we are planning to move to year-round testing sometime in 2007, approved applicants will be able to schedule the exam anytime during the year upon availability through Pearson VUE.
We believe that our decision to offer our examination through Pearson VUE has already benefited our candidates. The number of testing sites alone has increased from 25 to over 200. This means that the average travel time of 75 miles has been reduced by half. Our international candidates also benefited greatly because for the first time, we had over six international test sites including: Burnaby, BC; Hong Kong, China; London, England; Seoul, Korea; Toronto, Ontario; and Montreal, Quebec. Candidates from these foreign sites did not have to travel to the U.S. to take the exams. We look forward to adding additional international test sites in the near future.
AT: Are there mechanisms in place to stop information sharing and protect the test database?
KW: The first change was to retire all of the items that were in the NCCAOM databases and use those items for our new practice tests (to be available soon). The decision was then made to move to computer-based testing, administered by Pearson VUE. The dedicated testing centers, owned and operated by Pearson VUE, with their well-trained staff, oversee many rigorous security measures, including audio and video taping during examinations, along with fingerprinting and other forms of identification.
We have also begun working with Caveon, a testing security company that in cooperation with our psychometric vendor, Schroeder Measurement Technologies, Inc., does forensic data analyses to help us detect cheating and security violations during and after testing. We are also going to continue to educate our candidates and the faculty of the many prestigious schools on how to help eliminate and avoid cheating.
AT: Will any changes be made for the next round of testing?
KW: Registration will begin in August (candidates will be notified via mail) for the Oct. 2-14 administration dates. And other than having gained some experience and ironing out some problems, there should be no real changes for the October exam.
AT: Thank you.
References
Chronic pain afflicts over 20% of the adult population. Sadly, most MDs have essentially no education in treating pain, beyond offering a few toxic medications. Then they tend to steer people with pain away from those health practitioners who are trained. This puts the acupuncture community on the front lines for addressing this epidemic.
When the body is going through a stressful event, the sympathetic system is activated. This activation will help the body with a life-threatening situation. If this state is activated for prolonged periods of time, it starts to consume energy. These traumatic events that generate stressors in the autonomic nervous system (ANS) will eventually become internalized.
| Digital Exclusive
One of the things that makes acupuncture so difficult to measure is that the medicine actually does contain some esoteric aspects. There have been numerous studies that demonstrate the neurochemical reasoning behind why acupuncture points stimulate the autonomic nervous system to enact change However, not all the benefits of TCM can easily be measured by randomized, controlled trials. But does that make them fundamentally and definitively untrue?
| Digital Exclusive