Question 3

[MGN]

So there hasn't been much discussion of the actual content of these exam papers, and I think that's because really getting into the weeds is a fairly onerous exercise. So I suggest we break it down question by question in many threads. This one is for question 3. Here's my take, starting with part b:

The question is an 3pt Adler Kline question, with 2.5 for solving the problem and .5 for giving two advantages of the A-K method. Links for convenience:

http://www.casact.org/admissions/exampapers/exam6Can236.pdf
http://www.casact.org/admissions/exampapers/exam6Can882.pdf
http://www.casact.org/admissions/studytools/exam6/fall08-6.pdf

Part B (.5 pts):

Candidate 236 (failer) has "easily incorporates severity trend", which I think counts, so +.25, and has "independent of development method which allows for another check of reserving methods". This seems a little vague and doesn't match the sample solution, so I'm guessing 0 for that one.

Candidate 882 (passer) has "this method incorporates an independent estimate of inflation (flexibility)" and "this method is independent of the case reserving method (flexibility)". Now, the sample solution basically hits both of these two points, though with slightly more detail and implies slightly more causation, especially for the latter. Still though, given that the question says "briefly describe...", I am guessing he got both of these, so +.5 for this guy. (If I'm wrong about this, then all exam takers should note that these answers were insufficient, and this will be informative.) It's possible that he got .38 for this part, if the graders disagreed about the latter answer.


Ok, so 236 got 1.5pts total for the problem, meaning he got 1.25 for part A, and likewise 882 had 1.88 for the problem, so he got 1.38 to 1.5 for part A. Here's my take:

Part A (2.5 pts):

Candidate 236 projected claims closed and severity properly, but only did it for the latest year, which is insufficient as all six below-diagonal points are required for the projected reserve. He seems to demonstrate sample calculations properly, so it looks like he got half credit for providing only the reserve for the latest accident year.

Candidate 882 projected claims closed properly, but projected severity by applying inflation across development periods rather than across accident years within the same development period. Aside from this mistake, he did calculate the reserve properly. So it looks like he lost roughly 1-1.25 points on this one for botching the severity trend application. His sample calcs also look to be sufficient not to lose credit for that.

Final Notes:
It seems like the rubric on this one might be that half credit should be deducted from part A for a signficant misapplication of the process that leads to the wrong answer. Both candidates mistakes can roughly be described as such, and they both led to the wrong answer. Here's where I have an issue with the grading though: 822's mistake seems significantly worse than 236's. He demonstrated a shortcoming in his knowledge of the Adler-Kline method, whereas 236 could easily have misread the problem as asking for the "Accident Year 2008 projected reserve". Seems like 236 is closer to being the MQC here (for part A at least) than 882, but it seems to me they both got close to the same credit for the problem. Maybe Dan can comment?

Lastly, one additional thing they both did was to interpret the top triangle in the problem: "Incremental Loss and ALAE Payments on Closed Claims ($000)" as being the average severity for a claim. The sample solution noticeably interprets this as total payments on all claims. So it's possible that they both lost the same amount of credit for this. I'm curious to hear the group's thoughts on this. It seems like the wording is sufficiently vague that one could make that mistake, but interpreting that as an average severity would lead you to conclude that the severity trend of 5% was a pretty ridiculous number. (Dividing by closed claims makes the 5% trend pretty obvious, as in the sample solution.) But the 5% is given, so verifying it's reasonableness should be outside the scope of the problem, no? I could go either way here.

So here are my takeaway questions: 1) is botching the actual method worse than mis-reading the problem? 2) is the wording there vague or no? and if so, how much was that worth? 3) did 882 lose any points for part B? If so, how many and does this tell us something about what is required for a "briefly describe"?

I hope people comment or pick apart my analysis here. Let's get the most out of these papers.

[UCSDKID]

Lastly, one additional thing they both did was to interpret the top triangle in the problem: "Incremental Loss and ALAE Payments on Closed Claims ($000)" as being the average severity for a claim. The sample solution noticeably interprets this as total payments on all claims. So it's possible that they both lost the same amount of credit for this. I'm curious to hear the group's thoughts on this. It seems like the wording is sufficiently vague that one could make that mistake, but interpreting that as an average severity would lead you to conclude that the severity trend of 5% was a pretty ridiculous number. (Dividing by closed claims makes the 5% trend pretty obvious, as in the sample solution.) But the 5% is given, so verifying it's reasonableness should be outside the scope of the problem, no? I could go either way here.



I noticed many of the same things you did, specifically the wording of the question. This is the type of question that can be frustrating to some extent. They don't tell you whether the amounts are the average or the total. Now, looking at it closely it should be evident that the amounts are not the average as they are decreasing at later maturities and the severity trend does not seem reasonable if they were.
That being said, in an exam where time constraints are perhaps a bigger hurdle than the material itself it is unfortunate to lose half of the possible points because you mistakenly assumed the problem was giving you average severity instead of total dollars and completed everything else correctly. In my opinion candidate 236 (haven't looked at other candidate's paper yet) knew how to use this method. In "real life" situations there isn't the vaugeness of total vs average and that is what makes these problems frustrating.

Now, to the exam committee's credit (particularly Daniel) they are now writing the exams in a format so that time contraints aren't such a burden as well as the problems testing your knowledge better and not your ability to detect subtle tricks, whether intended or not. Unfortunately this wasn't in place during the last exam 6 as evident by the "disconnect between the exam grading and how it was written" and likely many candidates who knew how to apply the method lost roughly half the possible points with this slight error.

[Daniel Roth]

I won't be able to respond in such detail for all questions, but I'm hoping to help get you started by providing a sense of the general approach. What I show here is a reasonable way to look at it, not necessarily exactly how the graders or the Pass Mark Panel actually approached it.

As Arlie pointed out in the introduction, the intention is open discussion now. Later we'll identify specific questions for which we will go back to the graders to get the actual rubrics and deeper insight.

For question 3, I see the problem in four parts:

- calculate the historical average severities
- calculate the future average severities
- calculate the future claim counts
- sumproduct to get the unpaid claim estimate

It's reasonable to expect .75 available points for each of the first three steps, .25 for the last one (i.e., punching the calculator correctly).

I don't understand why you suggest 882 didn't demonstrate understanding of the method. At least he used the term "average severity", which demonstrates enough knowledge of the method to know this step had to be done. I'm more concerned with 236, who gave no indication that average severities are part of the method. In fact, it's possible a difference in their scores is because of this, with one grader awarding 882 0.25 point simply for mentioning it. As to the calculation itself, neither candidate performed a correct calculation, so 882 likely got either 0 or .25, 236 likely got 0.

For future severities, both applied the correct factor. 882 applied it incorrectly; 236 applied it correctly, but not fully to all the development periods. Could see graders knocking off .50 for either of these substantial errors, so .25 score for each.

Future claim count: looks like full-credit .75 for both (don't double ding 236 for not filling out the square, already hit on the severities piece). Note for "show work", it's sufficient to demonstrate how one of the count values was calculated; we understand all others are calculated the same way. But do need to show that calculation.

sumproduct: both candidates correctly converted their severities and counts to a total value. .25 each

so 882: 0 or .25, plus .25, plus .75, plus .25 = 1.25 or 1.50 for part a
and 236: 0 + .25 + .75 + .25 = 1.25 for part a

On part b, 236 first item essentially says that a benefit of this method is it's not some other method. Good try, but, no, this is not a meaningful advantage of this particular method. Item 2 looks OK. Both of 882's items look good, so:

882: 0.5 on b
236: 0.25 on b

Total: 1.75 or 2.00 for 882; 1.50 for 236

Here's the next question: If you were going to pass or fail a candidate based on their response only to this question, what would your pass mark be? This is one perspective on the Minimally Qualified Candidate standard. (There are others we use.) I don't have the 2008 Syllabus handy, but the 2009 Syllabus has the following Learning Objective:

3. Calculate unpaid claim estimates using
each of the following estimation
techniques:
• Development technique, including
case outstanding technique
• Expected claim technique
• Bornhuetter-Ferguson technique
• Cape Cod technique
• Frequency-severity techniques
• Credibility models

Adler-Kline is not specifically mentioned, so I might suggest that less credit is reasonable than if it were B-F or Cape Cod. For part a., I would view the historical and future average severities as hard to miss (in fact, if 882 and 236 had been paying attention to their units, I hope they would have realized their error. Their answers were $979M and $357M of reserve, respectively, for a company with around $1M claims per year.), so 1.25 / 1.5 available for both history and future. The claim count is a little more involved, so maybe I only expect .25 / .75. Plus calc .25. On part b., Adler-Kline may not be specifically mentioned, but it's generally pretty easy to get the 0.25 available for each "Briefly describe" item. We'll go with 0.5 as a placeholder. This would put the MQC standard at 2.25 / 3.00 (75%). This means that neither candidate met the standard on this question, even if we allow 882 2.0. Do you agree that is reasonable? I think so. I would hate to say either candidate did well enough in demonstrating Adler-Kline with these answers, though 882 was quite close, and clearly would have made it if only he had calculated the average severities he said he was going to.

This is how is essentially how pass/fail works. Each candidate's answer is evaluated against the standard. For any question that falls short, the candidate will need to make it up somewhere. A full credit answer on this question would allow such a candidate 0.75 shortfall on some other. Or, put another way, if there were a second question just like this, and 236 got full credit on it, he would pass (0.75 shortfall on this, but 0.75 above the standard on another). Again, I find that reasonable; his answer here was not so egregious that it shouldn't be made up by a perfect score on an equally challenging question somewhere else.

I hope this helps.

[Daniel Roth]

That being said, in an exam where time constraints are perhaps a bigger hurdle than the material itself it is unfortunate to lose half of the possible points because you mistakenly assumed the problem was giving you average severity instead of total dollars and completed everything else correctly.

If my assumptions are correct, a candidate who uses the losses as severities would only lose 0.75 point and would still meet the MQC standard if all else is done correctly.

[UCSDKID]

If my assumptions are correct, a candidate who uses the losses as severities would only lose 0.75 point and would still meet the MQC standard if all else is done correctly.

Daniel, thank you for that detailed explanation above. That helps to lend some insight into the process of exactly how the grading and MQC standards are approached.

I agree with your observation.

On a side note, does the exam committee look at what percentage of exam takers scored as Minimally Qualified on for each question? The reason that I ask is that both of these candidates missed the fact that the loss amounts were totals and not average severities due to the wording being a little vague. It would not surprise me to see if a significant number of candidates did the same thing. If enough people made this mistake (each costing 0.75 pts) then I would believe the the percentage of minimally qualified candidates to be lower than expected. If such were to occur does the committee give any thought to changing the MQC for this particular question due what may be a vague wording that apparently tripped up many candidates, some of which I am sure demonstrated they knew the Adler-Kline method well?

Thanks again for all your (and Arlie's) work in helping address the candidates concerns and bringing more transparency to the examination process.

[MGN]

Dan,

Great response. Very informative and I appreciate the time.

[Daniel Roth]

If enough people made this mistake (each costing 0.75 pts) then I would believe the the percentage of minimally qualified candidates to be lower than expected. If such were to occur does the committee give any thought to changing the MQC for this particular question due what may be a vague wording that apparently tripped up many candidates, some of which I am sure demonstrated they knew the Adler-Kline method well?

Part of the refinement in the pass mark setting process is post-grading feedback from the graders to help validate or refute assumptions inherent in the Pass Mark Panel's deliberations. If many candidates did this, it would likely be part of that discussion, but it might or might not result in changing the MQC expectation.

[UCSDKID]

Part of the refinement in the pass mark setting process is post-grading feedback from the graders to help validate or refute assumptions inherent in the Pass Mark Panel's deliberations. If many candidates did this, it would likely be part of that discussion, but it might or might not result in changing the MQC expectation.

Thank you Daniel for taking the time to respond.

[typical]

did 882 most likely get 1.88 cause one grader had 2.0 and the other 1.75?

there isn't any way where the grading would actually be so precise in partial credit to be less than 0.25 for deductions, right?