Acupuncture Today's Interview With NCCAOM Chief Executive Officer Kory Ward-Cook, PhD, and NCCAOM Director of Communications and Marketing Mina Larson

Acupuncture Today (AT): We understand that a security breach had occurred regarding the exam questions. Exactly what happened, and can you provide some further details?

Kory Ward-Cook (KWC): The security breakdown was really a result not of any break-in or Internet hacking or anything like that. We received an anonymous e-mail that arrived from a Yahoo account that contained about 300 descriptions of questions. These descriptions of the questions were not the exact questions in the sense that they were a "stem" of a question and then four "distractors," which is how we normally set up the questions on our test bank. Instead, it was a descriptor of the item, or the stem of the question, that was close enough to many of the questions that were going to be on the current exam, as well as a "here is the correct answer." I'll give you an example for a biomedicine question: "Elevated lipase and amylase = pancreatitis." It would give you enough information such that the person studying for the exam would be able to know what the question is and what the correct answer would be.

When we received this e-mail, the most pressing thing was to find out whether that content was something that was really in our exam. The header in the message said, "Cancel NCCAOM Exams Now: These Questions Are Out There." Our staff that oversees the testing department gathered all of the items from our current examination set of questions and compared them side-by-side. It took two days to go through this process. There was a huge percentage that overlapped. The next thing we did was to immediately call the NCCAOM Board of Commissioners together and they made their decision.

We believe the breakdown came from people memorizing questions, getting their lists together and circulating them. Coupled together with the fact that we got it, I think it's important to note that the same day, Rebekah Christensen, Executive Director of the AAOM also received the same e-mail that was sent to the AAOM on the same day. She was kind enough to send it to me and ask if we had received it, because she was concerned. The NCCAOM knew that not only did we receive the anonymous e-mail, but at least one other professional society had received it..

AT: Which exams were compromised?

KWC: The bulk of the items were coming from two exams. The most compromised ones were the biomedicine and Chinese herbology exams. Because that list was so comprehensive we didn't know the depth of it. It may have been a sampling of other questions that other people might have, which also played into our decision to cancel all the examinations. We couldn't be sure that this was just the first wave of questions coming to us. We had a very short window to respond, because the exams were originally scheduled to be February 18^th and 19^th. We had about seven or eight business days to deal with it.

AT: Your press release stated that as a result, you are going to retire all of the existing exam questions. How many questions are we talking about?

KWC: We're going to retire thousands of them, when you collect all of the exam questions from those modules. I haven't added them up, but each exam did have other questions as backups. The problem is that when exams are put together, there's sampling. We have an item bank, and sometimes things are sampled from the bank to produce new forms of the exam. If an item had been exposed, we didn't want to take a chance. We figured that if were going to clean this up, we'd better just do it now and really go for it.

Some of those items can be shuffled to be used on different forms of the exam. When you add that up, each exam probably has a backlog of 200 to 500 questions. It's just that some of those items have been used before. We really don't want to take a chance; we just want to make sure that some of those haven't been memorized either.

AT: We understand that while you're going to retire all of the old questions, you're going to be using the same source materials, and the people who wrote the previous questions are going to be writing the new questions. How will the new questions be substantially different from the old questions?

KWC: We've expanded our committees. We were planning on doing this anyway, so it wasn't anything we had to modify our plans. We are adding more people to our committees and bringing new people on board. That's number one.

Number two, one of the things I've wanted to do is look really closely at how questions are written and start delving into that. There are different ways you can write questions. You can have recall questions, applied questions and judgment questions. Some questions are much easier to memorize. We're going to be doing not only content validity - in other words, matching up what we need to do with our exam blueprint - but we're also looking to make sure we have representative numbers of applied questions, too. And these exam committee members that will be overlapping - in other words, some of the members that had been here before - are going to get new training on how to write better questions on the same content.

AT: The process of putting an exam together is rather lengthy. Questions not only have to be written, but validated, cross-checked and so on. How will the NCCAOM be able to accomplish all of this within the three or four months necessary for the questions to be ready for the June exams?

KWC: We're doing a couple of things. We're going to have a lot more item writing workshops. We've always had item writing workshops in which we get practitioners and educators from the community and invite them to come to a session. We usually do it at the AAOM meeting and the AOM Alliance meeting, and sometimes at the AOBTA meeting. Fortunately, we had already scheduled an additional one this year. We decided we would be going to the CAM Expo in New York City, and had planned an item writing session. That one is scheduled for March 3, and as of today (February 17), I think we're up to 60 people who are registered. Pacific College in New York is going to be sending a number of their faculty to the event.. We're going to be doing not only that item writing workshop, but we're also mapping out everywhere around the country where there's an acupuncture and Oriental medicine college. Where we see a large concentration of schools, we're going to be doing at least nine item writing workshops between now and the end of April. We're going to bring in a lot of "raw materials"

Also, we still have some items that have not been used that we're going to use. We have items in there that we haven't put out there for public exposure yet. They just need to be reviewed.

AT: Do you foresee any situation occurring that would cause you to have to postpone June exams as well?

AT: How many people were affected by the February cancellations?

KWC: We had a little over 1,000 candidates. They were in various stages of their certification process. Some of them had taken a few of their exams to be OM practitioners. Some had not taken any yet. Some were taking just the acupuncture module. Some of them are actually practicing already as acupuncturists and then they were coming back to take other exams.

AT: Do you foresee any situation occurring that would cause you to have to postpone June exams as well?

KWC: No. Let me clarify something. This is something we're learning as we talk to the students and candidates that were planning to take the exams in February. June is when we normally do our next exam cycle, so I think people were thinking that we were just cancelling this and waiting until June. It isn't that at all. We're doing the exams as soon as we can in June. If it falls the last week in May, we would definitely do it in May. That's why we're not announcing the exact date yet. As we get closer into the process of getting these exam questions in, then we can announce when we can do it. We'll have a better idea. What we're saying is that it's going to take us at least until June. The other candidates who were scheduled for June can then take the exams in June, too.

The new vendor that we're working with, the staff and I have had a lot of experience with. That should go pretty smoothly. I'm used to working with them.

AT: What's been the general reaction based on the phone calls and e-mails you've received?

KWC: It's very interesting. We took a lot of care in the fact that we sent a blast e-mail to everybody; we mailed a letter to everybody within the second day; and then we called every single person. If they weren't home, we left a voice mail. People then started calling back, or they got their blast e-mail before we got to them. The staff is reporting to me that at least 95 percent of the people, once we take the time to explain everything that happened, are very appreciative. That's wonderful. Some of them call in; they're very upset and worried about their personal circumstances. After the staff talks to them about all of the support we're going to offer, they get really turned around. When we go through all of the things that this new approach is going to speed up and some of the important things we're going to do, they're much more relieved and happier.

We're at a stage now where 95 percent to 97 percent of the people are pretty calm. It's not their first choice, but they're moving on, and they know that we're going to stay in communication with them. We have a few people that do still want us to try to get exams earlier and don't understand, so we have some more education to do.

AT: Could you provide us some details about the support you're going to offer?

KWC: We have a number of things that we're doing. First of all, for those that are suffering a hardship because of the timing and the delay, we're offering to write letters of support. They're out financially for an airport ticket if they had planned to travel to a site. Right now, with the current vendor we're under contract with, they only have between 40 and 50 sites around the country. It varies a little bit each time, because they don't own the sites. Many students have to get plane tickets, and some of them have to come in internationally. We are writing letters. If they request, we'll write a letter to the airline, explain the situation, and try to help them get a refund.

Also, for those people who want a really fast turnaround with their state license, Betsy Smith, our Associate Deputy Director and Director of State Relations, has called every single state official. The state officials been very supportive, and they're trying to rise to the occasion and work with us to get the licenses turned around very quickly. We're letting the candidates know that they're working closely with the states. We hope that the turnaround time is much shorter now.

A lot of the colleges were concerned about student loans. Some students are going to be trying to pay off loans. They're getting ready to graduate. They may be a couple of months away from getting income that they expected. So, we're going to be writing letters to employers - anything we can possibly do. Basically, we're customizing it for each person.

Here's another thing we're doing. A few of the candidates have said that they've never taken an exam on a computer, and that it's going to be harder for them. What we're going to do is take all of those items in our data bank that have been exposed, or that we think have been exposed, and post them as practice tests. We're going to build practice tests and have them up in April. They'll be online; it's something that they can go online and do. A lot of them were relieved to hear that. They can practice. A lot of them were concerned about the new questions and were wondering if the questions would be similar. We're going to give them things that have been in the bank that we're not going to use anymore.

Of course, we're working very closely with the CCAOM. David Sale and Lixin Huang have been wonderful. We're trying to work with the colleges to spend more time with the students and talk about cheating. Students sign a code of ethics on their application that they will not divulge the contents of the questions to anyone. Students say that people talk, but we have it right on their application that it's a violation, and that they are responsible for reporting anyone sharing these questions. The colleges are going to work very closely with us to get that message out to the students and remind them. We're going to be working on that phase.

We just settled a lawsuit against a Mr. Chen who was doing these review courses which contained questions form the NCCAOM examinations; we had a press release on that. We've been working with our attorneys, and we finally settled that case. He had to pay, and he now has a letter of apology that he signed and that we're getting out to the community. Basically, he lost his certification by getting students to come and give him test questions that he would publish and then teach in his review course. We're trying to have the colleges talk about those things, too, and what the consequences are for people who do that sort of thing.

Also, for any candidate that was affected this cycle, we're going to waive the recertification fees when they come due the first time they recertify. We won't be charging their recertification fee. They're going to get a voucher, and we'll keep it in our database, too in case they lose it. Also, we have students who are in the process of converting to the Oriental medicine certification. They've taken their acupuncture module already. Normally, they would complete all of the other tests that go with the Oriental medicine module, such as Foundations, Biomedicine, and Point Location. They would have to complete those to get their OM certification. We're going ahead and convert them over because they've already met all of the requirements for acupuncture, and without them applying for it, charge them at no cost and give them their certification in acupuncture. Some candidates will be really happy about that, because they won't have their ability to practices delayed. That was something the staff came up with. It was very creative, and I thought it was a wonderful gesture. This will allow people to at least practice as acupuncturists.

AT: Has the NCCAOM considered refunding or waiving exam fees?

KWC: We thought about that, but we're doing some other things. We still have to pay the vendor. If someone really wants a refund and could use the money now, we will certainly give it back to them. We will refund it to them if they want. I don't know if we've had any requests for that - maybe a couple.

Again, there are some other things that we're doing. We're going to be giving free score reports. Normally, they'd pay for that. There are a lot of things that they're going to get some value from. When we put these practice tests up, they can do one round of practice tests for free. We'll have it up as a link to the vendor that will be hosting the test for us. We're going to again accelerate that process so that we can have them do that before they have to take their exams.

People haven't really said that they want a refund or want their money back. It really hasn't come up. They just want to practice, so anything that we can do to get them there quicker and make them feel more comfortable - right now, we're trying to care about them and listen. My staff has been prepared. If it takes a half hour on the call, that's what they're doing.

AT: Tell us about the new testing company. Are they going to be the ones administering the tests in June?

KWC: Yes. They're not a brand new company, but they're going to be new to us. We're pretty excited about that. I've worked with them and have had a lot of experience with this particular company.

We had several companies make a presentation to the testing staff last summer.. We sent out a request for proposals (RFP), even our current vendor responded. Vendors could bid on either one, two or three services: the test administration, the psychometrics, and the item writing software that handles the database. Those are the three things we did proposals for.

AT: What were the reasons for dropping AMP and switching to Pearson Vue?

KWC: A lot of it had to do with security. AMP does not own their test centers. While they have protocols for the facilities, which are reasonably good, they don't have that. Another thing is that the new vendor employs their own proctors. They train them, and they certify them. That means that at every test site, no matter where it's located, you can be assured that it's the same design. The nursing profession uses the same vendor we're going to use; a lot of big organizations use this vendor for large certification programs. They designed these sites so that the proctor can sit in there and see every test station. There are a lot of cameras and video cameras around the test centers.. There's biometrics; people come into the door and have to give their thumbprints. There are high levels of security. We're paying for that extra service and the standardization of it.

We have this higher level of service and security, and there's a systematic, consistent standard of proctors and training that our candidates for the examinations will receive. If someone's testing in Oregon, they will have the same experience as someone testing in New York; the sites are identical, because the vendor owns those sites.

In addition to that, we're going to go from approximately 40 sites to over 200 sites. Some of the complaints that we heard is that candidates have to travel far. With our new test administration vendor practically everyone will only have to travel no more than 50 miles to get to a test site. In other words, if you were in Alexandria like I am here, I would probably have a choice of going to several sites in Washington, D.C. or Maryland. There are going to be many more sites for candidates to choose from.

AT: There are a couple of questions we would like to ask about some of the recent staff changes at NCCAOM.

KWC: I'm glad you asked about that. We're pretty excited about that.

I've been here about a year. I took the first year to get the lay of the land, to see what was going to work and to see people's skill sets. I spent a lot of that time seeing where people fit and where they want to fit, what they want to do, what they are doing well, and what the organization needs.

AT: It seems that a lot of the people who are no longer with NCCAOM are no longer associated with the exam program - for example, Debra Persinger, Mark Smith, Dion Coward and a few others. Why are these people no longer with NCCAOM?

KWC: We started adding some new staff shortly after I got here. Debra Persinger originally held the position of Executive Director of Operations and Interim CEO before I arrived at NCCAOM in November of 2004. We did some recruiting and brought Mark into the organization in February of 2005. Meanwhile, Debra was serving as Executive Director of testing.

That staff arrangement continued for a year. Even when I arrived I knew that we had a staff person in testing that had been working with Debra, would be leaving the organization in June 2005, so we knew we had to bring some new staff into the testing department. We brought in a young lady who had a lot of experience doing committee management and we were really excited about her, but she had no testing experience. We felt that we could work with her, but as time went on, I realized that we needed people that had good testing experience and who had worked with vendors; people that could do database management and practice tests. We decided that we had some of that talent in-house. One of the staff members who had joined our staff in April 2005, I'd been working with here, and she had a lot of experience in this area.

As far as Dion - he just left about four days ago. Essentially, his job was going away, because we won't be doing the test administration enrollment piece anymore. Dion was wonderful as a staff person, and we were working with him trying to figure out what would be the best place for him in the organization; however, he decided that he would probably move on and go ahead and do something where he would be better suited, and he went to work with another college. We are still having Dion do some project work for us.

We had a new person, Yvonne Sanders, start just this week. She just started on Tuesday, and she has 25 years in the testing industry doing database management. I have another person who will be starting the second week of March. She's going to be in the assessment area and the exam development management piece. Yvonne has a master's in database management and computer science; she's good with the management databases. Our incoming staff person Pam Fromelt's strength is as a health care professional and in working with committee management, in particular test development and maintenance.. She's been a director and a manager, and has 20 years of experience as a health care professional; she has 10 combined years in certification testing and educational programs.

Mark Smith was never involved in testing. Mark has an operations background. He used to work for several certifying agencies in the past. He's a really good guy. The reason he left is that we really didn't have enough for him to do. It was a downsizing. It was the same thing with Debra. We had talent already here in the organization that matched what she had, plus we had new people that we needed to bring in at a director level that really knew the testing business and the database field. Mark was more on the business side, and helped with human resources and performance reviews. I've actually outsourced that function for the organization.

AT: Did these people resign, or were they terminated?

KWC: We eliminated two positions. We eliminated the two executive director positions, and I haven't replaced them. I was trying to flatten out the organization and go with directors only. I named a deputy director and an associate deputy director, but basically, we did away with the executive director positions. The titles were confusing, anyway. We've just eliminated those two positions, and they weren't replaced.

AT: Were financial reasons involved?

KWC: No. In fact, we are increasing the staff positions in other areas that are needed for the organization. Our PDA programs are growing. We are really spending a a frequent amount of time approving a lot more courses, so that whole area is growing. There are other areas that have grown considerably. Take Mina Larson, for example. She was doing the marketing piece and the public relations piece, and yet, became my executive assistant. Our organization is growing. We just had to get the right people with the right skill sets in the right jobs. She's now totally devoted to communications and marketing, which is her background. One of my other staff members was promoted to be my executive assistant

Mina Larson (ML): Kory has really gone to the staff and made sure that everyone is doing what they're trained to do and what they'd like to do, and that they're still matching up what they're doing in the office. I'm trained as a journalist. I worked for Gov. Pete Wilson for many years in his press office, so when I came here, I was doing many things, and sometimes doing things that I wasn't really skilled to do. I think Kory saw my passion for marketing and communications, and now I've found my niche. She mentioned that marketing and communications have grown tremendously. I'm sure I'll continue to do that. We're really rolling out a lot of wonderful media marketing campaigns the last year.

KWC: We're really trying to spend a lot more time out at the meetings, not only just the ones in the AAOM community. We're reaching out. We went to the CAM Expo in New York last year. We have a wonderful booth, and we've put some resources into that. We're putting resources into some other areas.

AT: We understand that there was a "parting of the ways" between NCCAOM and a person who had been elected to serve on the board of commissioners. Can you provide us with some details on why that person decided not to serve?

KWC: It was really her choice. We were in our retreat, which we started doing last year as a time to get together with the commissioners, exchange information, and have an orientation. It was the first time with the new commissioners that we decided to do a full-day orientation. We got into what it means to be a commissioner, what your roles and responsibilities (duties of obedience, loyalty and care) are, how you can function vis a vis going out and being proactive, and what you can and cannot do when you're serving in that capacity. Once you're in the seat of a commissioner, you're there to serve the public, look out for the best interests of the organization in the sense that we're producing valid and reliable exams, that our decisions have to be surrounding public safety, and not necessarily promoting one thing over another. It's a very important distinction, and we spent a lot of time on that on the orientation.

We talked about this a lot with the newly elected commissioner that was planning to take a seat on the Commission in January. She has a real zeal and energy in the AOBTA community, and wanted to be able to do some of those things, and it precluded her from being able to do some of those things in this capacity. It was sort of a conflict of interest that we had to worry about.

AT: Did she choose to leave the board, or was she asked to leave?

KWC: It was more like a coming to an understanding that there wasn't a fit here, especially related to the board responsibilities. She offered that, and everybody was seeing that. She decided that what she would prefer to do was to stay connected with the NCCAOM in other capacities, i.e. through committees. She's been on the ABT Examination Development Committee. We've been working with her in particular in writing frequently asked questions for the ABT exam and things like that. Ruth Dalphin, one of the commissioners who was going to go off the board was available to serve another year.

The person who went off is still very involved. We've been having a lot of conversations back and forth with articles that she's writing, and she's going to help us with public relations. She's still going to be involved with the ABT committee. It's just that it wasn't a fit, and she didn't want to get into a situation where she would feel like she had a conflict of interest. She has a real interest in serving and going out to the states, and while we can do testimony if we're asked, we can't be real proactive in promoting. There's a fine line as a certifying agency and she came to an understanding of it in the orientation.

AT: Do you have any final comments for the profession regarding what has happened at NCCAOM over the past few months?

KWC: I think we're going through a renewal. I'm really excited about it. I think that this particular episode with the exam cancellation just accelerated some things we'd already had on our docket. We were going to roll this out within the next six months and take a little slower pace, but I'm excited in a way, because now we're going to get there faster, and it's going to benefit everyone.

We've had this on our plate to do. I'm excited about it. We're moving ahead. I think you've got a sense from what Mina just told you that the staff is happy. People are in the jobs they want to be in. We're growing. And we've made a deliberate attempt to make a good set of exams better, and in the long run, it will be a much better situation for the community.

I think what this particular episode has done is bring us closer to the community. I've been able to talk to a lot of people personally and gotten to know people better. It's accelerated my getting to know some of the college deans a little bit better. Candidates are now getting a lot of one-on-one attention with staff. We're learning little things about them, and how they feel - we've had literally some of these people crying on the phone and then thanking us for listening, and understanding that we had to do this for the betterment of the profession.

What I'm seeing is a pulling together of the whole community. Yes, there are a few outliers there. You'll have that in any situation; you're going to have the majority moving through in a crisis that comes up. I'm pleased that people are responding and understanding and giving us the support that we need - the professional societies, the Council of Colleges, and many of the students and state agencies.

AT: Thank you.