Do certain exams get harder after time?

[Steve White]

Originally Posted by wsa

Okay, maybe I am irked here by an apparent prejudice against 'math machines' (being one myself ), and maybe since I am so green in the actuarial profession I am speaking from ignorance here, but I have to take issue with this idea.

I passed exam P this summer, and I used Marcel Finan's study guide (which is very good, I think) and the SOA 150, and I did, in fact, notice a trend towards deliberately ambiguous language in defining problems, and did notice that the exam questions reproduced in Finan's guide that were clearly worded tended to be the older ones.

Now, I have to say, if the objective is not merely to test whether someone is good at math, that is fine -- good, even. But the question remains what, then, that P is designed to test for. When I got into the exam room I was able to tackle the deliberately obtuse questions, but not because I was 'good at thinking on my feet' (which I think I am), but because I had worked through a similarly badly worded question before, and I knew (or could hazard a guess) what the question 'really meant', even though the meaning was deliberately obfuscated.

So, what is it that being deliberately vague tests for? Is it the ability to divine the meaning of poorly formulated instructions? Is it designed to test whether a candidate has used the appropriate study materials? Does the exam committee want to stand up and say, 'these are the qualities that we are looking for in a candidate?'



I can totally understand wanting less people to pass; the more people that pass exam P the less meaningful it is as a signifier of being a competitive candidate. I think, though, that the exam committee should therefore very clearly understand and define what is looked for in an entry-level candidate, if it is something more than simply being good at math, and design exam P to test for that. It doesn't have to be vague in order to be difficult.

Should I thank you? I was on the equivalent of the equivalent Course P committee from around 1978-1984, when you say the questions were clear. Still, I disagree with your premise that they are writing deliberately vague or ambiguous questions. You say the later ones are worse than the earlier ones. I've been involved with many of the preliminary exam committees (just MLC for the past 10 or more years) and every committee I worked with spent a great deal of time trying to make sure the questions are not ambiguous. Sure, they/we occasionally write ambiguous or defective questions, but it's not intentional, and on MLC is very rare. MLC has released a lot of post-2000 questions; the number that are ambiguous is quite low. I haven't looked as carefully at P, FM or C, but when I have looked most of the questions are clear.

Which of the published sample questions on P are the ones you consider "deliberately ambiguous"?

[bigb]

I don't believe the exam committee purposefully writes vague or ambiguous questions. I think it just goes to show the gap between those who have been in the profession for years and those of us young bucks who are still wet behind the ears.

I enjoy the challenge, I just don't care for the stress of an exam. My guess is that if you were to remove me from the situation, namely the pressure of the exam, and asked me if the questions were vague, there's a good chance I would say they weren't.

At the end of the day, if these exams haven't caused me to be a more critical thinker, then I've chosen the wrong profession.

[wsa]

Ultimately I don't think it does any good to bitch and moan about the SOA . . .

I actually would agree with this. I, for one, am very happy that these exams exist and that the SOA and the CAS exist to administer them. I'd like to echo bigb's sentiment and say that I am also enjoying the challenge very much.

To clarify, I am not accusing anyone of intentionally writing defective questions. I had hoped, if asked, to be able to present these ideas with some questions on the exam that I took this summer; unfortunately I don't see it over at http://www.soa.org/education/exam-re...oice-exam.aspx, so these are taken from edu-2010-spring-p-ques.pdf.

Here is question 41, with comments:

A study is being conducted in which the health of two independent groups of ten policyholders is being monitored over a one-year period of time.

It seems clear enough so far; I have two independent groups of insureds and my spidey-sense is telling me that the lives of the insureds are also going to be independent, as, sure enough, they are:

Individual participants in the study drop out before the end of the study with probability 0.2 (independently of the other participants).

Well, wait a minute -- these are two IID binomial random variables . . .

What is the probability that at least 9 participants complete the study in one of the two groups, but not in both groups?

This is where it falls apart IMO. Is the question asking about participants, or groups of participants? By the assertion in the first sentence, you would think that the question is about groups of participants, and you'd be right. The question means to ask, 'what is the probability that exactly one group has at least 9 participants at the conclusion of the study?' But that is not what the question says -- it asks, 'what is the probability that at least 9 participants complete the study in one of the two groups, but not in both groups?' which pretty clearly implies to me that it is possible for one individual to be involved in both groups, and that the question is asking what the probability is of at least nine people not dropping out if they are in one, dropping out of exactly one if they are in two, etc.

My opinion is that the wording of the last sentence could lead one down an erroneous line of thinking, especially considering the similarity of the two groups. Knowing that two binomial experiments with n = 10 isn't quite the same as one with n = 20 (which successes are in which group?) will tell you which part of the language needs to be interpreted literally and which needs to be interpreted creatively, but: is that a 'bug', or a 'feature'?

Moving on to question 64:

A probability distribution of the claim sizes for an auto insurance policy is given in the table below:

 
Claim Size | Probability
-----------|-----------
      20   |    0.15
      30   |    0.10
      40   |    0.05
      50   |    0.20
      60   |    0.10
      70   |    0.10
      80   |    0.30

What percentage of the claims are within one standard deviation of the mean claim size?

Now, this seems like a no-brainer, right? Find the mean (E[X] = 55), find the standard deviation (\sqrt{E[X^2] - (E[X])^2} = 21.79449472), and then add up those rows with claim sizes within 21.79449472 of 55, to get 0.05 + 0.20 + 0.10 + 0.10 = 0.45 claims, which is the right answer.

Except, of course, that you have (correctly) answered a completely different question than the question asked.

The correct answer to the question 'What percentage of claims are within one standard deviation of the mean claim size?' is, of course, 'I don't know.' You're not given a frequency distribution for claim sizes at all; in order to have enough information to finish the problem, you have to assume that the law of large numbers applies and the probability is the incidence. Now, in the earlier questions, the ones having to do with counting arguments, inclusion/exclusion, the text I used says pretty clearly that the naive probability is defined as the incidence, so questions that use the two terms apparently interchangeably must have some license to do so. But: is this a counting problem, with the counting part left out? I don't know.

Question 72:

An investment account earns an annual interest rate R that follows a uniform distribution on the interval (0.04, 0.08). The value of a 10,000 initial investment in this account after one year is given by V = 10,000e^R.

Determine the cumulative distribution function, F(v), of V for values of v that satisfy 0 < F(v) < 1.

The actual problem here is pretty straightforward -- I have a random variable V in terms of another random variable R and I want a df of V, so I blindly manipulate some symbols and voila, I have the right expression.

The main weird thing is that strange condition at the end -- 'for values of v that satisfy 0 < F(v) < 1' -- is it asking me to set up, or test, some boundary condition? Well, no -- it's just there so that the answer can be technically correct without being a piecewise function over the whole real line, and the multiple-choice section takes up less vertical space on the page. So a candidate is maybe tricked, here, into thinking he doesn't understand the question, when in fact he does. ('What does (0,1) have to do with (0.04, 0.08)? Oh wait, that's the range of F . . . but, hmm, isn't that practically just F? What am I missing?) It's not technically vague at all, but, subjectively, it feels like one is jumping through hoops in order to be both concise and technically correct, and ends up with what is on the page just confusing someone who might even be smart enough to do the problem. My opinion is that the question could be worded differently while retaining its typographic brevity, and yet be less likely to throw a candidate off of the right trail.

The swath of questions involving joint distributions typically employ some natural-language statement involving logical connectives that tell the candidate what support to integrate over, and it seems to me that these statements should be especially vulnerable to poor wording, but in the example problems I can't find one that I disagree with.

Does the exam committee want to stand up and say, 'these are the qualities that we are looking for in a candidate?'

This sounds an awful lot like an indictment, and I really didn't mean it that way. I suppose that since having passed and continuing to look at these kinds of problems I can feel my perspective shifting from the pure-Math-mode where anything less specific than 'Calculate the expected bonus payment from the insurer to the 1000 policyholders in one year' (question 142), or 'Calculate the expected value of this piece of equipment after three years of use' (question 130), is 'deliberately ambiguous' -- and yes, as the questions get to be more complicated, I will admit that the language in the SOA 150 does get sharper and more direct.

Also, 3 << 150, and even in these three, if you 'get it', you will set the problem up correctly and arrive at the right answer. I suppose I would do well not to confuse 'deliberately ambiguous' with 'offensive to my aesthetic sensibilities.' I would be embarrassed, but someone else might say with equal validity, 'well, this isn't an English test.'

At this point, I can't help but really wish that the exam that I actually sat for was being disseminated, so that I could check my naive impressions against the actual wording of the questions, to test bigb's hypothesis that it is more the stress of sitting for an exam than it is the exam itself, in addition to being able to make one or two comments I know I would like to make.

I think, though, that the exam committee should therefore very clearly understand and define what is looked for in an entry-level candidate, if it is something more than simply being good at math, and design exam P to test for that.

The more I dig through these sample questions, the more apparent it is to me that it would be very difficult to do this without re-thinking the whole paradigm. It seems to me that the main things being tested for are: the ability to correctly recognize the concepts outlined on the syllabus, and the ability to set up a problem correctly, given an albeit concocted 'real-world' scenario. This is a 'softer' thing to test for than 'can you perform this kind of computation.' If what one is aiming for is testing whether one would be able to, given tools whose use is prohibited on the exam, apply a certain kind of thinking to less 'concocted' problems, this is indeed a difficult thing to design a test to measure, and would of course be more subject to the vagaries of natural language than testing for a 'harder' skill. So, perhaps what seems like 'a trend towards deliberately ambiguous language,' is actually a shift from testing one kind of skill to another, as technology changes?

I should have used a less invective tone. It is difficult to remove ambiguity from natural language, but the people who wrote these 150 questions seem to have done as well as anyone could reasonably expect.

[bigb]

At least you can't be criticized for not thinking through your opinion.

[wsa]

@bigb: Too much?

So, I guess its just part of what we signed up for, whether we like it or not.

I wanted to address that sentiment, before I saw Mr. White's response, if only to say that if it is, in fact, a 'game,' it is so far, I think, a fun and challenging one. Maybe there is this idea holding over, for me, from school -- a touch of resentment, perhaps, that it has to be a 'game' at all. But I'm not about to abandon my aspirations towards passing my exams and finding an actuarial job just because sometimes the process can fail and in so doing collapses to something I think is difficult for the wrong reasons.

[bigb]

Not at all. I like the way you think. Whether or not someone agrees with your opinion the fact that it is well thought out and you give reasons to back it up, shows that your not just griping about something, but instead you are putting effort into your exams to actually learn the material and take something from it. Kudos to you.

[bigb]

One other thing thinking through your post in regards to the wording of a question causing confusion or ambiguity. It is very frustrating for instance, when looking at the solutions to some problems that you couldn't figure out, that you indeed had the toolset to solve the problem, if only you could have understood it better. This happens many many times for me and is very frustrating.

My guess is there are at least 2 to 3 if not more on any given exam (at least for P, FM, and MFE which I have sat for) that I really did have the knowledge to get, if I could have sifted through the weeds with what it was trying to say.
I think that is part of the challenge the SOA presents to us on these exams, I don't see that changing.

But, I also now believe that failing an exam doesn't mean your not capable or qualified for the position. I'm sure there are people out there that may fail an exam but actually have a better grasp of the material and are more valuable as far as their daily job functions go, but of course that isn't necessarily true.

[Stone Wall Jackson]

Just imagine if English is your second language.

[Actuarialsuck]

Originally Posted by Stone Wall Jackson

Just imagine if English is your second language.

I imagined it and lived it. Not too bad.

[Steve White]

Originally Posted by wsa

I actually would agree with this. I, for one, am very happy that these exams exist and that the SOA and the CAS exist to administer them.

Q41. I agree. Not a poster child for well-written questions.
Q64. I hope and think that if I were involved in setting the question, it would have asked "What is the probability that a claim is within one standard deviation...?" It's just as short, and much more precise.
Q72: I tend to disagree with the objection, at least until seeing a proposed alternative. It's definitely correct as written. That idea has got to be somewhere, in the answer choices if not in the question. Densities are easier: you can just say "where nonzero".

To clarify, I am not accusing anyone of intentionally writing defective questions.

If you're even accusing them of intentionally writing ambiguous questions, I would disagree. Can we agree on Hanlon's Razor:

Never attribute to malice that which is adequately explained by stupidity.

I don't think the the Course P committee is stupid. I don't think the MLC committee is stupid. I know we on MLC occasionally, well under 1 per exam on average, ask defective questions where we later ask ourselves "How did that get by?"

I had hoped, if asked, to be able to present these ideas with some questions on the exam that I took this summer

Due to the move to computer-based testing, the SOA and CAS aren't releasing any more exams, and disclosing questions from the bank is not allowed. P has added a few questions to its sample set over time, withdrawing them from the question bank.