Why doesn't the SoA just do this when it comes to grading...

[Double Down Trent]

Why don't they just set the pass mark, for MC exams, so that the pass rate is as close to 40% as possible?

I would definitely agree to this if it meant we could have our grades back within 2-3 weeks.

If we have 1000 people take Course ~ (that happens to have 20 questions on it) and the following distribution of correct answers:

0 - 0
1 - 9
2 - 19
3 - 45
4 - 72
5 - 95
6 - 138
7 - 156
8 - 141
9 - 100
10 - 60
11 - 57
12 - 45
13 - 20
14 - 14
15 - 9
16 - 12
17 - 4
18 - 3
19 - 1
20 - 0

If you set the pass mark at 9, 32.5% pass. If you set it at 8, 46.6% pass. So, since 46.6 is closer to 40 than is 32.5, the pass mark should be set to 8. There, grading finished, send out the grades.

I don't understand why they have to fiddle around with this report and that one. Why do they have to spend weeks dealing with how difficult they thought the exam was? Shouldn't that be known long before the exam is given?

They do all this micromanagement of the grading process, and yet, for every exam I have ever seen (other than May 2000 EA-1B, where I think I was among the 78% or so of the passers), my process would probably come up with the same pass mark as theirs. I mean, isn't the sample of data points large enough at this point to cut a few corners in order to get the exam results out in a reasonable time frame? Especially if a new grading process would be supported by the vast majority of students and (I would imagine) members?

[Cynic]

A few reasons that I can think of:

1) Politics. They want people to think that they take this exam thing seriously. (While they may in fact just toss our exam papers in a corner for two months).
2) Tradition. "It's been done that way for a long time, so why change now?"
3) Revenge. "We all had to suffer before, so why let those suckers have any easy time?"
4) Laziness. Any change requires some explanations, some memos, some letters, etc.
5) Entertainment. It's always fun to hear some arguments, complaints, whinings. Being an actuary, chances are, these are the only entertainment you can enjoy.

[josie]

The one reason I can think of which would extend the time it takes to grade the exams is for defective questions. I think they have to determine if there were any defective questions and how to handle them. Since volunteers are involved in this process, I can see why a 2 to 3 week turnaround is quick. However, I don't know if that justifies a 2 month turnaround.

[Quasi]

I think that a big part (maybe 1/2) of the justification of the 2 month turn-around is that they have to give people time to complain about defective questions.

[Cynic]

I don't think defective questions justify the delay. The grading process is based on a curve (despite what they say), and the defective questions--if there are any--hardly affect the curve. But this is a pointless discussion. Enough said.

[The Drunken Actuary]

I think that a big part (maybe 1/2) of the justification of the 2 month turn-around is that they have to give people time to complain about defective questions.
That will be hard to do now that exams are not being released. Besides, the exams are full of defective questions that never get thrown out.

[Actuary321]

I think that a big part (maybe 1/2) of the justification of the 2 month turn-around is that they have to give people time to complain about defective questions.
That will be hard to do now that exams are not being released. Besides, the exams are full of defective questions that never get thrown out.

And the turn-around time was 2 months way back when they didn't regularly release exams.

[KnightsPG]

I was thinking about your distribution. You said that if you set the pass mark at 9, 32.5% pass. If you set it at 8, 46.6% pass. Since with these results we really our looking at a normal curve, a lot of time the SOA is going to run into this problem. Does anyone knew what they do about it? I have a conspiracy theory and keep in mind that this isn't fact or anything, but I was hoping to hear from someone on the exam committee to let me know what they do in these situations. You assume that they either pick 8 or 9. But what if the SOA is dead set on passing 40%. Do you think that they take the people with 8 and 9 and go then go and weight the different questions on difficulty. Therefore, questions 1-5 might be worth 1 point, questions 6-10 worth 2 points, questions 11-15 worth 3 points and questions 15-20 worth 4 points. Then the graders could differentiate these people and come up with a solid 40%.

I guess I'm suspicious of the way the SOA is so concerned about not giving out exams, pass marks, exam scores, or letting you take your answers out of the exam room. I was trying to figure out why and it didn't make sense to me. I was thinking though that it has to be tough to decide who gets to pass if you have a normal distrubution curve with a huge sample size, and since a lot of people are going to fall pretty close to eachother, let's assume a low variance. Then you would have lots and lots of people on the borderline (8 or 9 in the previous example). My conspiracy theory is that two people get the same score and one passes and the other fails. They are friends taking the exam and if their results were given they would both realize they got the same score and someone would be upset.

Keep in mind that I'm skeptical of the theory, which is why I'm hoping someone from the SOA that is part of the exam process will fill me in on what goes on when grading. That is, as long as you use an anonymous name so the SOA doesn't hunt you down for telling their secrets. I'm just a little weirded out by the secrecy of the process.

Can anyone help me out?

[Actuary321]

Can anyone help me out?

Do you have a middle name or just an initial?

[ASA_Woman]

:lol: Actuary321, I'm having some trouble understanding your posts today. First you're laughing at I don't-know-what at people discussing tea in the cubicle science thread. And now you're asking KnightPG's middle name here??? What gives? :-? :)

[Dr T Non-Fan]

Your idea assumes a constant average quality and distribution of candidates' adequate knowledge.
It also assumes that 40% is some magic number. Numbers are magic only to the innumerate.

[Steve White]

Without getting into the question of how the passing score is determined, I will say that to the best of my knowledge, multiple choice questions are always weighted equally. (In situations like Course 3, Nov 2000 and many exams before the current course structure, questions could explicitly at the time of the exam be given different point values; then those points were weighted equally.) It has been 8 years since I was involved in deciding where the pass line would be drawn, but in several administrations of 150 and course 3 I have positive knowledge of the equal weighting, and can't believe there could have been unequal weighting in other sessions without my knowing.

[KnightsPG]

Thanks Steve White, I appreciate your response. I figured that was the case, but I just don't understand the process of setting the pass mark. Thanks for enlightening me a little bit.

Dr. T. Non-Fan - You're exactly right on the assumptions, but isn't that reasonable to assume. With only forty questions you're going to have a lot of people who are pretty darn close to eachother in the middle. We're all studying the same questions for three months, I think it is reasonable to assume a low variance on people's knowledge. (I'm not saying everyone will fall in this category. People can differentiate themselves by being in the, let's say top 10%, but overall, depending on the exam, I'm guessing the SOA runs into the problem that Bobby Bowden outlines.

I do agree that the SOA isn't set on a pass mark of 40% either, but I do think that they have a number close to 33% or so that they would like to get close to. My question was intending at understanding what happens in these situations. I was assuming that they merely picked (after long review) a number, 7 or 8 in our example and allowed a larger or smaller number to pass than they intended. However, I never found where in the exam catalog it explicitly states that the problems aren't weighted so I just threw out the question, so I could understand a little more.

Thanks for both of your responses.

[Dr T Non-Fan]

If anything, all weighting of questions is disclosed on the exam. For example, the WA questions on 5 and up are weighted. The T/F MC questions are/were worth half the points of a regular MC question. On Risk Theory (old 151), the questions were explicilty weighted.

It's nice to assume equal distributions of knowledges, but the Exam Committees know better, especially when a whole bunch of people were attracted to the profession due to its top ranking in the late 1980's.
I think that the exam committees would prefer a two-prong distribution instead of a normal one. (One for the adequate-knowledge folk, one for the rest of the pack.) Less hand wringing is better.

[E. Blackadder]

Here's how it used to be done, for test #s < 200. (so I was told -- and greatly simplified)

1: Heap big conference call.
2: Argue about the relative difficulty of each question in multi-hour session. Determine % of effective students who should be able to get question right. (old exam results were used to maintain norms)
3: Add up %ages -- get a pass mark.
4: Fudge the result.

Permission / justification was required to pass too many or too few. (don't know what range was OK)

Note that step 4 made steps 1 - 3 meaningless, or nearly so.

[Lucy]

In my more cynical moments, I agree with Edmund. I can add a little, though.

1) I have recently been involved with tabulating and reporting results of joint exams for the CAS board. I have certain knowledge that all the exams I looked at had a fixed passmark, and all weights of questions were disclosed to candidates. (So if they don't say anything, every question counts the same.)

2) Oddly enough, it takes about 3 weeks to score the multiple choice sheets. The SoA hires an outside company to do this, and that's their turnaround. Then another week or so is spent checking anything that looks weird - either because the scores look weird on that question, or because someone complained to the SoA. I suppose if they just decided to always pass X%, they could release results in about a month, but not in 2 weeks.

[Griffin 1]

Without getting into the question of how the passing score is determined, I will say that to the best of my knowledge, multiple choice questions are always weighted equally. (In situations like Course 3, Nov 2000 and many exams before the current course structure, questions could explicitly at the time of the exam be given different point values; then those points were weighted equally.) It has been 8 years since I was involved in deciding where the pass line would be drawn, but in several administrations of 150 and course 3 I have positive knowledge of the equal weighting, and can't believe there could have been unequal weighting in other sessions without my knowing.
Many of the 100-level exams were not always equally weighted. 120, 151, and 160 come to mind. I don't remember if the others ever had unequal weights.

[Steve White]

Many of the 100-level exams were not always equally weighted. 120, 151, and 160 come to mind. I don't remember if the others ever had unequal weights.
You are just trying to agree with

(In situations like Course 3, Nov 2000 and many exams before the current course structure, questions could explicitly at the time of the exam be given different point values; then those points were weighted equally.)aren't you?

[Dr T Non-Fan]

Thank you as well, Mr White.

[The Drunken Actuary]

Many of the 100-level exams were not always equally weighted. 120, 151, and 160 come to mind. I don't remember if the others ever had unequal weights.
You are just trying to agree with

(In situations like Course 3, Nov 2000 and many exams before the current course structure, questions could explicitly at the time of the exam be given different point values; then those points were weighted equally.)aren't you?
I'm sure. He's a very agreeable person.

[Take 2]

When EA exams began, the SoA had a tough problem -- the Joint Board set the pass mark much higher than the SoA wanted it. Passing for FSA would not be good enough for EA. The arm-wrestle has never quite ended. :-?

[Agtuary]

Has the SOA ever given a reason for the secrecy of the grading process? Is there a reason the SOA does not tell us on which subjects we performed poorly?

The lack of communication can only deter individuals from pursuing this designation and help them choose a different path.

[Mr. BoH]

Is there a reason the SOA does not tell us on which subjects we performed poorly?

The lack of communication can only deter individuals from pursuing this designation and help them choose a different path.

I think you just answered your own question.

Less actuaries = More $$ / FSA

[Moderator2]

Check the official SOA bulletin board. Someone very recently posted a question about the secrecy issue. It will be interesting to see who replies and what (if anything) they offer by way of explanation.

Dilbert punchline quote:"I don't buy, I shovel."

[Steve White]

Is there a reason the SOA does not tell us on which subjects we performed poorly?

Check the official SOA bulletin board. Someone very recently posted a question about the secrecy issue. It will be interesting to see who replies and what (if anything) they offer by way of explanation.

I have consistently been in favor of publishing all the Course 3 exams promptly afterwards, with answer key. In addition to the benefits for candidates on future exams, that would provide great performance feedback. For each question, you could decide whether you thought you knew how to do it, and (if you care), you could try to rework it and verify whether your thoughts were right. (If solutions were also published, you could eyeball the solution to decide if you understood the problem.) This doesn't tell you if your guesses were right, but on numeric problems if you recognize that you were guessing you already have the feedback that you didn't know how to do it.

Unfortunately, that isn't going to happen. Without it, I see two big barriers to telling candidates where they did poorly.

First, the problem of guessing. Suppose there were 8 problems on topic X, and the candidate thought he knew how to do 5 and guessed at 3. If we tell him he got 5 right, should he conclude he got the 5 he expected, and was an unlucky guesser? Or perhaps he got only 3 of those 5, and was lucky enough to guess 2 correctly? Usually, the feedback would be very ambiguous.

Second, and more fundamental, the problem of classification. If you look at the published exams, you'll see that quite a few problems cross whatever logical breakdowns you would try to make. Simulation is the best example. I would bet that more than half the simulation questions in the four published exams involve other topics, e.g. simulating time of death with some information about probabilities of death. If someone misses such a problem on a multiple choice exam, did he miss it because he didn't understand the simulation aspects, or the time of death aspects?

I admit to being an advocate of not trying to give performance breakdowns on Course 3, for those two reasons. The information would be of little value, and might well mislead candidates as to where they should devote their efforts. I would be glad to rethink it if someone had a concrete suggestion as to what the feedback should be. Don't put too much work into your proposal: with my record of success at convincing the SOA to publish the exams, convincing me on feedback wouldn't help you much anyway.

Classifying problems would be much easier on Course 2, but the problem of guesses would remain.

[Agtuary]

Steve, thank you very much for the answer. I understand your reasoning and think it makes a great deal of sense. I can see not publishing the exams in order to develop a question bank, though. When enough questions have been developed that the Society considers "fair" the administration of the MC exams could become much easier. It would also offer the possibility of offering the exam more times per year, 4 maybe. Unfortunately it leaves open the possibility of some groups collaborating to recreate the exam. (A topic of another thread)

But what about my other question? What is the purpose of keeping the grading process and the pass mark secret? It doesn't seem to make sense. I can only assume the pass mark changes enough each sitting the Society would be worried about excessive complaints of unfairness, but publishing the exams would make it very simple to determine the pass mark.

[Steve White]

The grading process has already been explained in this thread.Here's how it used to be done, for test #s < 200. (so I was told -- and greatly simplified)

1: Heap big conference call.
2: Argue about the relative difficulty of each question in multi-hour session. Determine % of effective students who should be able to get question right. (old exam results were used to maintain norms)
3: Add up %ages -- get a pass mark.
4: Fudge the result.

Permission / justification was required to pass too many or too few. (don't know what range was OK)

Note that step 4 made steps 1 - 3 meaningless, or nearly so.
He doesn't have steps 1-3 exactly right, nor is the extent of 4 such as to make 1-3 nearly meaningless, but there is certainly judgment involved.

I'm extremely proud of Course 3's record at avoiding defective questions, but would not be nearly as proud of our record at predicting in advance how hard specific questions or even an entire exam will be. IMO, it would be an unmitigated disaster to commit to a pass mark before we see how candidates performed on the questions. So after the fact, you adjust your opinion of the difficulty based on a host of statistics. Let's say that before you know anything about results the committee would expect a passmark of X. After seeing how median candidate score compared to median candidate score in the prior exam, you might be inclined to go with X+1, because you conclude the exam was easier than you expected. But, suppose you were to consider the mean candidate score instead of the median, and that suggests X+2. Conflict. Throw in a host of other statistics, most suggesting X+1 or X+2, with maybe a few suggesting X or X+3. (I was very careful to say "suggest" each time because even looking at a single statistic leaves room for judgment about what that statistic implies about the pass mark.) If you reach that stage, it's clear the passmark should be X+1 or X+2, and people (none of which is me, BTW) discuss how clear the various "suggestions" are; which are most important, etc; and make what has to be a borderline call between X+1 and X+2. With many candidates having scores of exactly X+1, there are a lot of unhappy candidates if the decision comes down at X+2, ready to second guess a decision that was reasonable, made by humans trying to be as objective.

Oops. I oversimplified. All that discussion has referred to how the statistics on this exam related to statistics on the prior exam. By now, there are many prior course 3's. Suppose (hypothetically; these don't correspond to actual course 3) you concluded that Nov 2000 was .7 question easier than May 2000, so you raised the passmark .7 rounded, or 1. Suppose then you concluded May 2001 was .7 easier than Nov 2000, so you raise it another 1 rounded, or 2. Now you've got an exam 1.4 easier than May 2000, with an passmark 2 higher. You have not done your best to keep May 2001 standards comparable to May 2000, even though you did your best each time to keep standards as close as possible to the prior exam.

You might feel: "If it's reasonably clear that the passmark should be X+1 or X+2, and not overwhelmingly clear which, why not just give the candidates a break and declare it is X+1." For any one session, that might be feasible. Over a several sessions, it would represent a weakening of standards.

The above description gives a good flavor of why I don't think it's feasible to try to explain exactly why the passmark was set where it was. It's been at least 8 years since I had any vote on what the passmark for an exam was, so some of the statements may not be exactly right.

[Cynic]

I agree with Mr. White on the issue of giving failing candidates a break-down of their score. When I got the break-down for C2, it said that I did okay on the Finance part. What actually happened was that I guessed a lot of those questions right.

With regard to the grading process, I don't think it makes sense at all. What they assume is that every group of candidates perform at the same level. What if for some reason a group of candidates perform better than normal so they all got better scores as a result? Why doesn't the SOA make things easy and fast by setting a fix pass rate (say 25 correct out of 40) every time? They can re-evaluation this from time to time. I don't think any particular exam is easier than others. Besides, easy or hard is kind of subjective, isn't it?

[Agtuary]

Steve, thank you again. I think the lack of explanation as to why and the seeming lack of explanation from anyone ever involved in the process is my reason for complaining about the lack of communication. I have seen the process described before, but never from one willing to admit to being involved. You are the only one I have seen willing to say I was involved in the process at one point and this is how it was. Most explanations have been second hand. Maybe I have not been around long enough to see other first hand accounts.

I still do not understand why the SOA seems unwilling to provide an "official" explanation. Maybe I have not tried hard enough, but I would hope it would be easy to come by. Few people will be willing to try unless they are already involved in the process.

Cynic, what it appears Steve is saying is the SOA has made the assumption that each sitting the group taking the exam has the approximate distribution as every other sitting. When creating a new exam basically from scratch each sitting this is may be a better assumption, and I would hope the statistcs back this up, than letting the exam committee try to evaluate the difficulty level. Evaluating the difficulty level of each exam in total prior to the exam would end up being a much more subjective process than making the previous assumption and evaluating the difficulty level of a few individual questions based on actual results.

I guess the SOA has decided the drawbacks from this process outweigh the benefits from any other approach discussed. It unfortunately does seems to ignore the possibility that one group is more prepared than another. Maybe this is part of the "judgement" involved and one would hope intelligent people might consider this possibility. I do not know how they could determine this, other than from the percentage of retakers. Although this doesn't necessarily guarantee a more prepared population.

Making this assumption can inadvertantly raise the standards as time passes. Let's assume the population continually improves over time. In Steve's example the pass mark gets higher and higher becuase the exam committee, errouneously in this case, assumes the exams are becoming easier and easier. So rather than continue to make the pass mark higher next time they make the exam more difficult. Then keep the difficulty level the same for 2 more sittings and the same thing happens. This may or may not be good, but it gives some credibility to the argument that the process, if not the SOA, keeps the level of actuaries low.

Note: I am not saying the exams are more difficult than they have been in the past, but they are!

[Agtuary]

The most recent post from the link Mod 2 referred to:

I think there was a reason why exams are no longer released. It has always been the policy not to release them. After the transition in 2000 they decided to release them, and they gave some long explanation. It was supposed to help students get used to the longer exam format. Alas, it was only temporary.

Unfortunately, things will get worse, not better. The scoring on the written exams are a complete mystery, and the score breakdowns that failing candidates receive only confuse them more. The questions are written so vaguely that candidates debate the correct way to do the problem for months after the exam. The "model solutions" that the SOA releases are "not intended to be perfectly correct solutions," and indeed they are not. So one never knows exactly what the question writer was asking for.

I don't mean to be cynical, but there is absolutely nothing you can do about while you are a student rather than an FSA. All you can do is focus on passing the exams.



My guess is this is a student and he holds what I believe to be a pretty common opinion among students.

[retaker]

I was afraid that was the way they came up with the pass mark.
I agree with cynic.
It should be easier and much more reliable for persons who know the material well enough to make an exam on it to judge whether any particular exam is harder or easier than another.

Depending on the candidates perfarmance is unreliable and unfair.

[Steve White]

Hi, Guys. I checked in at lunchtime and see I've been unclear. I'll write more this weekend.

Briefly, I said to rely on a passmark predicted by the Exam Committee before the exam would be unsatisfactory. Believe me, it's far easier to determine whether a problem is right and on the syllabus than exactly how difficult it is. A difference of 1 or 2 in the passmark affects a lot of people.

The Committee has a lot of involvement after the fact. We see some of the statistics, and can, if we choose, revise our views about the difficulty of the problems. The current Course 3 chairman does get a vote on the passmark and participates in any conference calls discussing what the passmark should be. For each Course 3 given so far, in addition a former Course 3 or former Course 150 chairman has had a vote.

More later.

[retaker]

"Note: I am not saying the exams are more difficult than they have been in the past, but they are!"

rock on :love:

[Steve White]

What they assume is that every group of candidates perform at the same level. What if for some reason a group of candidates perform better than normal so they all got better scores as a result?

No, they do not assume every group performs at the same level. With that assumption, pass mark setting is trivial: set it to preserve, as closely as possible, the percentage passing from the previous session(s).

The actual procedure may not produce results much different.

Why doesn't the SOA make things easy and fast by setting a fix pass rate (say 25 correct out of 40) every time? They can re-evaluation this from time to time. I don't think any particular exam is easier than others. Besides, easy or hard is kind of subjective, isn't it?Most people would think some particular exams are easier than others, and would feel it unfair that they had to score 25 on their set when someone else faced an easier set.

Yes, there is subjectivity involved. Is there less subjectivity in telling the committee: "give us an exam for which you think the passmark should be 25" than "give us the exam you want to offer, and tell us what you think the passmark should be?"
Cynic, what it appears Steve is saying is the SOA has made the assumption that each sitting the group taking the exam has the approximate distribution as every other sitting.

I would not say that assumption has been made. You reevaluate difficulty in light of performance. There will be some problems where the candidates performed significantly better and others where the candidates performed significantly worse than I would have expected under the "same ability as before" assumption or even a "5% better or 5% worse than before" assumption. So I'll reexamine my judgment. I'm not necessarily going to decide "what was I thinking? It should have been whatever matches the actual performance", thus relying strictly on the equal groups assumption. Nor am I necessarily going to conclude "OK, I'm infallible on difficulty. These guys are superstars/klutzes."

[Me]

My guess is this is a student and he holds what I believe to be a pretty common opinion among students.

Yes, he passed course 6 in the spring but is not yet an ASA. Isn't lack of anonymity in the SOharA great?

[Cynic]

Thanks, Mr. White, for your explanations. Even though I'm not any happier with the grading process, I sure appreciate some explanation of how it works.