What should the FSA exams look like?

[Colymbosathon ecplecticos]

Originally Posted by JollyRancher

Anecdotal evidence. Are you disagreeing?

How would anyone know? Suppose that passing is a 60 points. You get a "5" --- that means that you had between 54.0 and 59.9 points. How could anyone know that they were 1 or 2 points from passing? And, of course, not every failing candidate gets a "5".

[Gandalf]

Originally Posted by Colymbosathon ecplecticos

How would anyone know? Suppose that passing is a 60 points. You get a "5" --- that means that you had between 54.0 and 59.9 points. How could anyone know that they were 1 or 2 points from passing? And, of course, not every failing candidate gets a "5".

You two need to agree on terminology. He said "a question or two", not "a point or two", so he may mean much more than 2 points. OTOH, with partial credit, a person who knows the subject material for a question well could easily end up with 50% of the points or more, even if he misunderstands what the question intended to ask. Not all questions are the same. Not all ambiguity is the same. Generalizing about the average impact of ambiguity seems pointless. Best to agree that it's bad, that it does lead to some erroneous pass/fail results, and that the SOA should try hard to avoid it.

[Sredni Vashtar]

Originally Posted by JollyRancher

Anecdotal evidence. Are you disagreeing?

Anecdotally, I would disagree, though it's been a few years since my prelims. I remember more difficult questions and fewer 'bad' questions.

Anyway, this raises one suggestion, which is that a test-format with more challenging questions would be less prone to this problem.

[almost_there]

Originally Posted by ishamael

Another skill to add to this list is "communication".
Brings fond memories of a piece of trite in the UK version of The Actuary magazine patting FIAs on the back, arguing that they have superior communication skills compared to members of other actuarial orgs because only FIAs go through a communications exams, concluding that FIAs merit higher salaries than the other Fellows.

Granted, that article was written by a recruiter but it has always mystified me how it made its way through the editorial process.

It's very hard to square that article with claims by IFoA of being equivalent to those other actuarial orgs...

[Puffd]

From a net knowledge gain perspective for candidates has thought ever been given to the idea of two exams failures of high enough score meriting a pass.

By that I mean say you're pursuing the Corporate Finance Track. And you get a 4 on CFE. Take an exam from a DIFFERENT track. Score a 4 or higher and be granted a pass. for CFE.

I'd net much more knowledge from failing say CFE with a 4 or 5 and failing LP or the Valuation exam on the ILA with or a 4 or 5 (or passing it for that matter)

[wat?]

Originally Posted by Puffd

From a net knowledge gain perspective for candidates has thought ever been given to the idea of two exams failures of high enough score meriting a pass.

By that I mean say you're pursuing the Corporate Finance Track. And you get a 4 on CFE. Take an exam from a DIFFERENT track. Score a 4 or higher and be granted a pass. for CFE.

I'd net much more knowledge from failing say CFE with a 4 or 5 and failing LP or the Valuation exam on the ILA with or a 4 or 5 (or passing it for that matter)

No.

I studied for the ILA track (both valuation and pricing), got a 4 on the pricing exam. Switched to CFE and finished up my FSA. I did not deserve to pass that exam, despite doing well enough to pass in another track.

Sample size of 1, but there's your anecdote.

[Sredni Vashtar]

Starting on my last exam now. My where has the time gone!

I feel about the same as I did a year ago. Which is that 1,000 hours of training, worth roughly $50,000 per student, really out to yield more for the profession than the bulk memorization of obscure trivia.

Originally Posted by Chuck

My 2 cents from another old FSA in JMOs generation...

The OP raises good and important questions, even if they are somewhat age old questions that we old farts can reasonably characterize as whining (as we once whined). We should continually consider what's being whined about. I also have to agree w/JMO that creating good tests is extremely difficult but there always seems to be some low hanging fruit to consider.

My first thought is to step back and consider in theory (and practically) what the purpose of the exams are. (There is overlap discussion here with the regular debates about university education vs exams.) One argument being made is that they are to provide useful knowledge and education to the exam taker. Seems reasonable and hopefully they do some of that, but of course a 2nd goal is to evaluate the qualifications of the person being tested. Hopefully, there is some overlap between those two goals when designing the exams (and arguably there are elements of the exam that do neither - eg memorizing clearly useless facts unless we think we need to test the ability to memorize and test the stubborness, fortitude, compliance, etc of the candidates to be willing to memorize useless facts). I have never needed to know about the Wigglesworth mortality table (and it's ilk) or the 7 things to look for in a heart murmur, but they are still indelibly fried into my brain some 40 years later taking up my limited data storage for some more important knowledge.

Thanks this is exactly where I was coming from. It helps to hear an old-time like you express the same concerns.

I agree completely that it would be great to start by asking what the exams are trying achieve, and then talk about how well the process itself addresses those goals, and ask what alternatives are feasible.

I also have to agree w/JMO that creating good tests is extremely difficult but

Yes, exactly. It's must be really difficult to create a fair and reasonable general knowledge exam that is hard enough to fail ASAs. Which is why I wanted to talk about alternatives.

But I think that a major benefit of the exams is to generally have and demonstrate the ability to research and learn on your own (independent of any specific material). The likelihood of my exams covering what I need to know is a very small % and getting smaller with each passing year from the exams. But they can demonstrate the ability to learn what you need to learn. For example, rote memorization of the ASOPs is probably worthless. But demonstrating that you know enough about the ASOPs to know when you better find one and review it, and that you have the ability to do so, may be worthwhile (from a qualification standpoint).

So far the exams have done a good job of telling me what's out there-- whether it be an ASOP or a FASB standard or an ACA reg.

However I haven't had to demonstrate the ability to perform research... Because I've never read the source materials. It's a waste of study-time, better to just use flashcards. Sometimes exams include things that aren't on flashcards (sometimes intentionally) but anything that is so obscure it doesn't fit on 500 flashcards is going to be too obscure to remember anyway.

I would love to see an exam built around the need to do research. Because actually it IS really difficult to make sense of Accounting Standards and government guidelines. But in that case, I think you'd want an open-book exam with a less intense time-limit. The exam could ask an extremely obscure question about a regulation, and rather than expect the student to have memorized the whole thing by heart, they could flip through a booklet, and find the right paragraph and translate it from Legalese into numbers.

All that said, I am sure the exams could be better designed to do whatever their purposes are. But lets not make impractical demands of having them make me knowledgeable about much of what we need to know. I think that this is better left to practical continuing ed (personally driven or otherwise) given that things like CFT, asset adequacy, predictive analytics, spreadsheets etc, etc were mere glimmers in the eyes of the old fart actuaries designing the exams in our day. (yes, spreadsheets - I'd have been tested regarding desktop calculators, proper use of green-bar paper, Fortran/Cobol/JCL/card punch machines, and mainframes, etc!). Your important knowledge will all too soon be obsolete as well.

I agree here too.

Although, I want to add, I think the exams I've taken have gone too far in avoiding technology. The dependence on pencil and paper and BA-II+ calculators creates an artificial test of scribbling in answers and performing Excel-work on a crummy calculator.