Throwing Out Questions

[brown bear]

After sitting around and not having anything new on here, I wanted to see if anyone had an answer to something I've been thinking about. Is there any thought as to what the exact criteria is for throwing out questions? I guess I would hope that Bruce might be able to comment, or possible Abe if he knew, but there were a few scenarios that I thought of.

1) What criteria exists for a question to be thrown out? I guess I understand if they forget to put a key assumption in the title, but is there a cut-off for % of people who got the problem right? Say 30% get it right, or 20%, or 10% or even to the extreme 0% get it right...
2) On some past exams, there have been instances where two answers were accepted. What criteria exists to determine if multiple answers will be accepted, or if the question will be thrown out?
3) When a question appears to be thrown out, what happens to that exact question on everybody's exam? Does everyone immediately get it considered correct, or is it just completely removed from consideration?
4) Is throwing out a question ever considered an alternative to reducing the pass mark? I guess the answer to 3 really would impact this one, as say without throwing out a question, the pass mark would be 18/30, but with throwing it out, they decide to raise it to 19/29 or whatever the case may be. If you throw out the question and someone had 19/30 but then gets negatively impacted by throwing out the question (now has 18/29 and doesn't pass), how would this get handled.

These are the main things that I could think of, and any response would be greatly appreciated. I hate to sound ungrateful, but tf you are going to answer, I'd prefer it be concrete evidence, not "I would guess it would be..." or "I heard from my co-worker that..." Thanks for understanding and if no one really knows, I guess that's an acceptable answer too.

[Actuarialsuck]

This should help as to how they decide if the question is defective: http://www.soa.org/education/general-info/grading/edu-defective.aspx. I don't think the % of people getting it right has an impact, it might force them to look at the question if only 10% of the people got it correct but I don't think it's a criterion for throwing a question out. As far as lowering the passmark and causing people to pass/fail, I think there was a thread in the CAS forums regarding this (since this is a joint exam, it probably has an impact) where Arlie addressed the situation I believe.

[Vorian Atreides]

The link Actuarialsuck is a good starting point, I'll offer the following more specific comments:

1) What criteria exists for a question to be thrown out? I guess I understand if they forget to put a key assumption in the title, but is there a cut-off for % of people who got the problem right? Say 30% get it right, or 20%, or 10% or even to the extreme 0% get it right...
I believe the criteria might be called "actuarial judgment". When they look at the distribution of responses for a given question, if the majority of answers are not on the identified correct response, the question will likely be looked at closer.

2) On some past exams, there have been instances where two answers were accepted. What criteria exists to determine if multiple answers will be accepted, or if the question will be thrown out?
If someone on the committee (or a student submitting a valid argument for their response) can support the "alternate" answer from the problem statement, then you'll find a situation where both answers are accepted. I think the only time a problem is "thrown out" is when there is overwhelming evidence that the problem is too ambiguous (e.g., distribution is fairly flat).

3) When a question appears to be thrown out, what happens to that exact question on everybody's exam? Does everyone immediately get it considered correct, or is it just completely removed from consideration?
I believe it's completely removed from consideration and everyone's score is adjusted accordingly. I don't think the pass mark is set until after each problem has been analyzed. (Could someone from the Exam Committee confirm/clarify this, please?)

4) Is throwing out a question ever considered an alternative to reducing the pass mark?
Why would the societies do this? They set the pass mark where they set it. Those who create the exams questions would be insulted if their question was thrown out to make adjustments to the pass mark.

. . . the pass mark would be 18/30, but with throwing it out, they decide to raise it to 19/29 or whatever the case may be. If you throw out the question and someone had 19/30 but then gets negatively impacted by throwing out the question (now has 18/29 and doesn't pass), how would this get handled.
I believe questions that are going to be thrown out are thrown out before the decision to set the pass mark is made.

These are the main things that I could think of, and any response would be greatly appreciated. I hate to sound ungrateful, but tf you are going to answer, I'd prefer it be concrete evidence, not "I would guess it would be..." or "I heard from my co-worker that..." Thanks for understanding and if no one really knows, I guess that's an acceptable answer too.
I doubt that you're going to get a lot of "concrete" examples given. My responses are based on doing this sort of process for six years (common final exam at different universities while teaching at said universities). What actually takes place with the Exam Committee might be different than what I posted above.