Ask a grader

[campbell]

Howdy y'all -- I'm grading for FET this sitting, and as one who has gone from exam-taker to exam-grader this year, I know that the process is often opaque to people and you want info.

I was going to write an article for The Future Actuary with advice/answers from the grader's point of view -- it's been done before, but that was over 10 years ago, I think. Anyway, lots of people will be going from prelims & modules slap into the written exams as opposed to that halfway written/multiple choice that was Courses 5 and 6 (and the 5 & 6 questions were much simpler than the DP & CSP questions).

So I wanted some input from y'all as to what you want to know, or what you think exam-takers want to know, especially those who have never taken a written exam before. I won't be able to answer these questions in a timely manner here because I need to find out what I can and cannot disclose (e.g., I definitely cannot talk about the passmark, for the simple fact that I have nothing to do with setting the passmark.) I will answer what I can when I can, but that may be some weeks as we're deep into first grading right now.

[Bühlmann]

I was going to write an article for The Future Actuary with advice/answers from the grader's point of view -- it's been done before, but that was over 10 years ago, I think.

Would there be a better publication than The Future Actuary? Most people taking the FSA exams are actuaries with their ASA's and don't get this sent to them. I don't get it, but I used to before I had my ASA.

[campbell]

Well, I'll add it to my advice on written exams page (http://wiki.mathuniverse.com/index.php?title=Studying_philosophy_for_actuarial_ exams#Preparing_for_written_exams) well before it gets published on dead trees. And of course, "publish" it on the AO.

I don't know what other actuarial publications I should target. Any ideas? The Future Actuary is an obvious one, as it's sent to people taking exams. I could send it to the Education & Research newsletter, Expanding Horizons (http://soa.org/news-and-publications/newsletters/education-and-research/pub-education-and-research-details.aspx), but looking at recent issues, it doesn't seem that would be an appropriate venue.

I'm open to any suggestions.

[Dreaming]

How about a list of the most common mistakes graders encounter when grading the essays?
I have never taken an essay exam but I hear FSAs at work complaining that many exam takers simply do not answer the question asked.

[bdschobel]

Would there be a better publication than The Future Actuary? Most people taking the FSA exams are actuaries with their ASA's and don't get this sent to them. I don't get it, but I used to before I had my ASA.The Future Actuary doesn't go to ASAs? Are you sure? I thought that it did, but can't check easily until Monday.

Bruce

[campbell]

How about a list of the most common mistakes graders encounter when grading the essays?
I have never taken an essay exam but I hear FSAs at work complaining that many exam takers simply do not answer the question asked.

That's a good one. If you're asked a yes or no question, make sure you answer yes or no at some point.

Example: "Should you invest in the project? Do blah blah analysis to decide."

Yes, you should do the blah blah analysis and that will be weighted with more points than the yes or no answer. But if you do not then have something that says, "As a result of the blah blah analysis, you should/should not invest" (this is a wordy version) you missed a really easy point.

Also, putting down true but irrelevant items will get you no points. Under "true but irrelevant" are formulas for something vaguely related, but not going to be used in the problem. Also true but irrelevant is what you read in the Wall Street Journal that morning (I did this once myself on a question about Social Security, amongst other things. I failed that exam.)

The grading rubric comes directly from the course of reading, and though it may be adjusted if it's deemed other answers have merit, the point is to check that you've absorbed the course of reading and can apply it appropriately. Graders have access to the relevant portions of the course of reading, and for the problem I'm grading, I read through the original source to make sure that the grading rubric was comprehensive.

[Gandalf]

That's a good one. If you're asked a yes or no question, make sure you answer yes or no at some point.

Example: "Should you invest in the project? Do blah blah analysis to decide."

Yes, you should do the blah blah analysis and that will be weighted with more points than the yes or no answer. But if you do not then have something that says, "As a result of the blah blah analysis, you should/should not invest" (this is a wordy version) you missed a really easy point.
You mean you would give someone a point if they said "As a result of the blah blah analysis, you should invest" (assuming that's the conclusion they should reach), even if they did no analysis whatsover? Without analysis, how can it be right to conclude that you should or should not invest?

I would much rather see you give a point to someone who made a calculation mistake in the analysis and concluded you should not invest, assuming his analysis would support not investing.

[campbell]

You mean you would give someone a point if they said "As a result of the blah blah analysis, you should invest" (assuming that's the conclusion they should reach), even if they did no analysis whatsover? Without analysis, how can it be right to conclude that you should or should not invest?

I would much rather see you give a point to someone who made a calculation mistake in the analysis and concluded you should not invest, assuming his analysis would support not investing.

Let me be more specific with my hypothetical situation. Blah blah analysis gets you some number, which then one would conclude that you should invest.

Let's say the grading rubric says I give 4 points for doing the blah blah analysis right, and 1 point for getting the right conclusion from that analysis.

So say you screw up the blah blah analysis a little, and I give you partial credit for the bits you did correctly (say 2 out of the 4 points). As a result of your screwup, you get a number that says =don't= invest, and you write that you should =not= invest due to that number. I would give you the full 1 point for that part. When you screw up earlier in the problem, I will take your screwup as correct and see if you made a correct conclusion for that.

The person who just says yes or no without doing anything in relation to the blah blah analysis will get no points. The point is for getting from the # from the blah blah analysis and making the correct conclusion.

Suppose you can't actually do the calculation (for whatever reason). Then to still get at least a point, you could say "If the blah blah analysis showed a # >= fweedle then you should invest, otherwise you shouldn't." That would get you the 1 point for making a conclusion.



This is an important point on numerical questions. Many times there are multipart calculations to do, and you might not be able to do one part for whatever reason. Say you have to calculate an interest rate and you just can't do that part, and you need that interest rate to do subsequent parts. Then let the screw-up part go and just make up an interest rate. Say something like "I can't do this part, so I'm just going to assume i = 10% (or whatever) and work from there." You can get full points on the remainder - you'd just be out the points for figuring out the interest rate.

[tommie frazier]

should the candidate liberally use the word "y'all" to curry favor with graders?

[bdschobel]

That's one of the secret words that always merits extra points. Write it often, especially on your multiple-choice answer sheets. :D

Bruce

[Bühlmann]

The Future Actuary doesn't go to ASAs? Are you sure? I thought that it did, but can't check easily until Monday.

I used to get it and now I don't. I'd like to though...even when I'm done with exams to stay current with changes.

[Bühlmann]

I have never taken an essay exam but I hear FSAs at work complaining that many exam takers simply do not answer the question asked.

To those who complain about such things: easier said than done. Sorry to disappoint, but given how much there is to learn (and memorize), how much overlap there is between some topics and how vague and/or awkward some questions are written, and most importantly, the huge time crunch you are under, it is easy to understand why sometimes people don't answer the question. Even knowing in advance going into G&H DP that it is a common mistake students make, I know of several parts of the exam where I put down the "wrong list." Fact is that I was just blazing through the exam. And also to this point is the fact that you get asked about those things that you think "oh, they'll never ask this"...and they do. Well, not wanting to leave it blank, you do your best to bs it and get some points possibly.
Long winded response, I know. But I hate to hear FSA graders complaining about people "not answering the question." These things are tough and most of us work our tails off to memorize 2,000 pages of stuff. :swear:

[Separate Topic]
Some of the above posts address answering numeric questions. I know that you get big points for writing down the formula on an exam like FET. But what about an exam like G&H DP? There are virtually no real "formulas" like the black-scholes forumula or something. Yes, you want to show your work, but you don't want to waste too much time defining every ad-hoc expression you write down (and there are many of these during the course of a numeric G&H DP problem). As an example, suppose I'm calculating a gross premium from net claims. Should you write down, " Gross Premium = (Net Claim Cost + Admin)/( 1- profit - commissions)"? It is not a real equation...it is just a technique. I would just write down "Gross Premium = (220 + 16)/(1 - 4% - 8%)"..to me that would get me the points for "writing down the equation"...but saying "Gross Premium = 216/.88" may not.

[campbell]

To those who complain about such things: easier said than done. Sorry to disappoint, but given how much there is to learn (and memorize), how much overlap there is between some topics and how vague and/or awkward some questions are written, and most importantly, the huge time crunch you are under, it is easy to understand why sometimes people don't answer the question. Even knowing in advance going into G&H DP that it is a common mistake students make, I know of several parts of the exam where I put down the "wrong list." Fact is that I was just blazing through the exam. And also to this point is the fact that you get asked about those things that you think "oh, they'll never ask this"...and they do. Well, not wanting to leave it blank, you do your best to bs it and get some points possibly.
Long winded response, I know. But I hate to hear FSA graders complaining about people "not answering the question." These things are tough and most of us work our tails off to memorize 2,000 pages of stuff. :swear:

Remember that all the graders are FSAs, and know about dealing with huge amounts of reading materials. We are very sympathetic with test takers, which is why we're not complaining so much as trying to give you useful information. When I talk about "answer the question" I mean to look at the actual words to the question and make sure you address that. If you're asked to make a recommendation, make sure you make a recommendation as part of your answer!




[Separate Topic]
Some of the above posts address answering numeric questions. I know that you get big points for writing down the formula on an exam like FET. But what about an exam like G&H DP? There are virtually no real "formulas" like the black-scholes forumula or something. Yes, you want to show your work, but you don't want to waste too much time defining every ad-hoc expression you write down (and there are many of these during the course of a numeric G&H DP problem). As an example, suppose I'm calculating a gross premium from net claims. Should you write down, " Gross Premium = (Net Claim Cost + Admin)/( 1- profit - commissions)"? It is not a real equation...it is just a technique. I would just write down "Gross Premium = (220 + 16)/(1 - 4% - 8%)"..to me that would get me the points for "writing down the equation"...but saying "Gross Premium = 216/.88" may not.

Well, you don't get a huge amount of points just for writing down the appropriate formula in the FET. Not from what I've seen. Putting the bits together nets you more than the formula as you're writing. The example you gave with the (220 + 16)/etc. would count as putting down the formula and its requisite info. If your example is correct, you wouldn't get full points (not sure how many) on the second one because your numerator is wrong (assuming it should be 220 + 16). This is why you should write out as many steps as you can stand -- if you do a simple arithmetic screwup and you don't put in in-between steps, you could be losing out on a bunch of points you'd otherwise get.

[no]

When I started with E&E committees, I thought about something similar. I wanted to
organize a session in one of the SOA spring meetings. Have a speakers from each of
the FSA tracks, try to update what they did about 10 years ago. When they did it, it
was mostly 1.5hrs to 2hr exams. Somebody should do it.

I did enjoy grading my own questions probably for about 3 sittings. I don't like grading
anymore, creating my own question is fun.

[campbell]

How many exam-taking ASAs go to the spring meetings? I really don't know.

[bdschobel]

I don't either. It's not always easy to identify "exam-taking ASAs," either -- at least not when they register for the meeting.

Bruce

[no]

Mary, you can organize a session (or try) in a meeting, if it goes wrong (i.e. very few register),
it will not be the first session with 3+ speakers and less than 3 participants!

In any event, what you are trying to do is admirable, but you already knew that. However,
you might want to check to see a chairperson or a vice-chair is interested in co-authoring.

[campbell]

Thanks for the advice, no. I was going to talk to the head honchos when I get to central grading, with a draft of what I've already written and seeing what they'd like to add. Obviously, I'm a novice grader, so I've only seen one question and ~160 answers so far, and it's a numerical question at that, so I don't know all the ways someone could screw up or increase their odds.

[Dreaming]

To those who complain about such things: easier said than done. Sorry to disappoint, but given how much there is to learn (and memorize), how much overlap there is between some topics and how vague and/or awkward some questions are written, and most importantly, the huge time crunch you are under, it is easy to understand why sometimes people don't answer the question. :swear:


I believe it's easier said than done, trust me. I'm not looking forward to those 6hr exams. Given the volume of material and the stress of trying to finish the entire exam on exam day it's expected that a student might quickly answer a question with the associated list he thinks of first, despite that list not being of the utmost relevance.
If you're asked to make a recommendation, do you get part marks for making that recommendation based on not-so-irrelevant but not quite bang-on information? Say for example that you have some sort of intuition of what the recommended answer should be but forget the relevant details of why. I am guessing absolutely not...

(Maybe intution might work?!)

[campbell]

Again, I'm only a novice grader, and many of these things depend on the actual rubric for the particular question. What I've seen so far is that just making the recommendation is accorded the least part of the points on a problem and the reasons for that recommendation are weighted heavier. You may get some points for the recommendation, if it follows from the reasons you give, even if the reasons are wrong.

The issue is that many times people actually give the right reasons.... and then never make the recommendation! Yes, it may be only a small # of points out of a lot, but try to skew the odds in your favor.

Look, I'm not chiding people here. This is not personal. I did the exams, too; they're tough. I am trying to give info out to people, as it may help.

A big problem I see with the exams is the mental stamina needed -- a lot of people flag in the afternoon session especially, and that may be why they're not paying close attention to what is exactly written in the question and jump on key words without realizing they didn't answer the question.

I am re-emphasizing that you need to make sure you're giving the answer asked for. If you're asked for an interest rate, make sure you highlight that. If you're asked for a recommendation, make sure you do that. If you're asked a yes/no question, make sure you answer yes or no at some point. That's all.

As to lists, if you've got two lists that you think may be pertinent, then as long as they don't contradict each other, put both down. We don't take points off for putting in irrelevant info. We add up points for relevant info (unless you put down contradictory statements, in which case they cancel each other out.)

[yanz]

Some things I've wondered about exam grading (I realize that you probably can't answer many of them, but I'm just throwing it all out there for future reference):

1. The Rubric:
A. Who puts together the grading rubric?
B. How is it put together (eg. does it consider all sources of reading or just the one that the question came from)?
C. Are rubrics for different Qs consistent in how they weigh different steps/aspects (eg. does writing down the formula, plugging in numbers, calculating answer, and drawing conclusion always give you a, b, c, and d% of the grade for a problem?)
D. What percentage credit does one need to get (based on the perfect rubric) to get a 10 on a problem (I know it's typically not 100%)?

2. Grading:
A. How is partial credit awarded (eg. if I make a calculation error or don't finish a calculation or whatever, how much leeway does the grader have in how many points are given/taken off? Or, in other words, how likely/possible is it that 2 different graders give me 2 different scores?)?
B. You mentioned that you went through the sources of reading to make sure nothing is missing. How many graders do you think do that?
C. If, while grading, you notice a bunch of people putting down [the same] info that isn't on the rubric, do you try to figure out why they're doing so (eg. there may be another reading that covers the topic or the question may be misunderstood to refer to a different list)?
D. (related to C) How many similar (but off the rubric) answers would you need to see before investigating a potential rubric change? Is there a fixed policy on this?
E. If you decide that the rubric should be adjusted (I'm infering from one of your comments that you have such power), do you discuss/confer with the other grader (my understanding is that there are 2 graders per Q) to make sure you two are grading consistently? Is there a fixed policy on this?
F. When "grading" a question, what grade do you give it? In other words, assume it's a 6 point question, and the rubric breaks it down into N points. Do you grade it out of 6, out of N, or on a scale of 0-10?

I'll post more as I think of more.

[campbell]

Ok, these I can't answer. In some of the cases, I don't know the answer (not enough experience); in other cases, I know the answer, but don't know what I can disclose. I will check on that when I go to central grading.

[yanz]

Ok, these I can't answer. In some of the cases, I don't know the answer (not enough experience); in other cases, I know the answer, but don't know what I can disclose. I will check on that when I go to central grading.

I figured as much, but wanted to get it out there since you asked for a list of Qs ;)

[The Head Detective]

Do you guys really call it a rubric?

[campbell]

Yes, that's the kind of thing I wanted. I will write something up answering those questions, but will check with the head honchos if it's okay to tell people.

[campbell]

Do you guys really call it a rubric?

Well, I do. I suppose most people call it the grading outline.

[The Head Detective]

Here's a couple serious ones. These are similar to some of yanz's above but I think there's enough of a twist to ask them anyway.

1) When a mutli-part question does not specify points for the sub-parts, how are points allocated? Are they always spread evenly over each part? If not, is there a "trick" to identifying the higher point value pieces?

2) Do graders see what score previous graders have assigned to a particular question?

[ajstudies]

The Future Actuary doesn't go to ASAs? Are you sure? I thought that it did, but can't check easily until Monday.

Bruce

I'm an ASA and I don't believe that I've ever received this publication. I became an ASA over 15 years ago though, don't know if that has anything to do with it.

[no]

Here's a couple serious ones. These are similar to some of yanz's above but I think there's enough of a twist to ask them anyway.

1) When a mutli-part question does not specify points for the sub-parts, how are points allocated? Are they always spread evenly over each part? If not, is there a "trick" to identifying the higher point value pieces?

2) Do graders see what score previous graders have assigned to a particular question?

1) Yes, I think Bruce answered this a while ago.
2) Yes, but I always second grade without looking at first graders scores, and
if it is different beyond the accepted level, I look at first graders scores.

[no]

Some things I've wondered about exam grading (I realize that you probably can't answer many of them, but I'm just throwing it all out there for future reference):

1. The Rubric:
A. Who puts together the grading rubric?
B. How is it put together (eg. does it consider all sources of reading or just the one that the question came from)?
C. Are rubrics for different Qs consistent in how they weigh different steps/aspects (eg. does writing down the formula, plugging in numbers, calculating answer, and drawing conclusion always give you a, b, c, and d% of the grade for a problem?)
D. What percentage credit does one need to get (based on the perfect rubric) to get a 10 on a problem (I know it's typically not 100%)?
.
A), B) Question writers submit questons and grading outlines (GO)s and they go through
at least 4 revisions, before it goes to graders. The revisions are done by vice-chair,
chair, general chair etc.

C) Yes


2. Grading:

B. You mentioned that you went through the sources of reading to make sure nothing is missing. How many graders do you think do that?
C. If, while grading, you notice a bunch of people putting down [the same] info that isn't on the rubric, do you try to figure out why they're doing so (eg. there may be another reading that covers the topic or the question may be misunderstood to refer to a different list)?
D. (related to C) How many similar (but off the rubric) answers would you need to see before investigating a potential rubric change? Is there a fixed policy on this?
E. If you decide that the rubric should be adjusted (I'm infering from one of your comments that you have such power), do you discuss/confer with the other grader (my understanding is that there are 2 graders per Q) to make sure you two are grading consistently? Is there a fixed policy on this?
I'll post more as I think of more.
B. In my experience, all the graders that I had contacts did this
C. Well this was the fun part, in one sitting I had to create a new GO, then get it approved
by chairs, all in about a few hours.

Disclaimer:
I am not grading FET. I've written and graded questions for three different exams.

[bdschobel]

I'm an ASA and I don't believe that I've ever received this publication. I became an ASA over 15 years ago though, don't know if that has anything to do with it.I'm working on fixing this. Identifying "active" ASAs is a bit of a problem.

Bruce