Actuarial Outpost
 
Go Back   Actuarial Outpost > Actuarial Discussion Forum > Software & Technology
FlashChat Actuarial Discussion Preliminary Exams CAS/SOA Exams Cyberchat Around the World Suggestions


Not looking for a job? Tell us about your ideal job,
and we'll only contact you when it opens up.
https://www.dwsimpson.com/register


Reply
 
Thread Tools Display Modes
  #1  
Old 11-05-2017, 08:59 PM
Colonel Smoothie's Avatar
Colonel Smoothie Colonel Smoothie is offline
Member
CAS
 
Join Date: Sep 2010
College: Jamba Juice University
Favorite beer: AO Amber Ale
Posts: 44,959
Question Does anyone here use Scala?

I was wondering if anyone here was using Scala, you know, to be Scala-friends with.
__________________
Recommended Readings for the EL Actuary || Recommended Readings for the EB Actuary

Quote:
Originally Posted by Wigmeister General View Post
Don't you even think about sending me your resume. I'll turn it into an origami boulder and return it to you.
Reply With Quote
  #2  
Old 11-07-2017, 12:23 AM
kevinykuo kevinykuo is offline
CAS
 
Join Date: Nov 2017
Posts: 10
Default

Why not R?
Reply With Quote
  #3  
Old 11-07-2017, 01:26 AM
Colonel Smoothie's Avatar
Colonel Smoothie Colonel Smoothie is offline
Member
CAS
 
Join Date: Sep 2010
College: Jamba Juice University
Favorite beer: AO Amber Ale
Posts: 44,959
Default

Quote:
Originally Posted by kevinykuo View Post
Why not R?
The main reason is that we're using Spark which supports 3 languages (Python, Scala, and Java). Our cloud computing service (Azure) offers an out-of-the-box solution for parallel processing - our datasets are a few TB in size which would be too much for R to handle on a single machine.

Azure also offers a parallel solution for R, which I haven't tried yet. I'm mainly using Spark because a big client of ours is using it.
__________________
Recommended Readings for the EL Actuary || Recommended Readings for the EB Actuary

Quote:
Originally Posted by Wigmeister General View Post
Don't you even think about sending me your resume. I'll turn it into an origami boulder and return it to you.
Reply With Quote
  #4  
Old 11-07-2017, 06:59 AM
Incredible Hulctuary's Avatar
Incredible Hulctuary Incredible Hulctuary is offline
Member
Non-Actuary
 
Join Date: Jan 2002
Posts: 23,782
Default

I've used Scala in the past. It's a great language, if you can get your head around some of its concepts that aren't common in most other languages such as functional programming and how it organizes its type system and multithreading. You may be less effective with it if you don't know Java first, as it compiles to Java class files and can seamlessly call any of the standard Java libraries (or any other Java class if you supply the .class or .jar).

One thing I don't like is that it lets you use arbitrary special characters as method names and operators, and does that with many of its own standard libraries, so it has stuff like <:< and :/ which have real meaning but you can't Google them. And if you're working in a team, you may have a doofus co-worker who likes to make up unreadable names like &/&<=>: instead of using a word or phrase that means what it says.
__________________
A lot of people are afraid of heights; not me, I'm afraid of widths. - Steven Wright
Reply With Quote
  #5  
Old 11-07-2017, 10:59 AM
kevinykuo kevinykuo is offline
CAS
 
Join Date: Nov 2017
Posts: 10
Default

Quote:
Originally Posted by Colonel Smoothie View Post
The main reason is that we're using Spark which supports 3 languages (Python, Scala, and Java). Our cloud computing service (Azure) offers an out-of-the-box solution for parallel processing - our datasets are a few TB in size which would be too much for R to handle on a single machine.

Azure also offers a parallel solution for R, which I haven't tried yet. I'm mainly using Spark because a big client of ours is using it.
You can also use Spark with R via sparklyr, which may get you to be productive sooner than learning Scala, especially if you're familiar with dplyr/SQL. It works more or less out of the box with HDInsight and supports kerberized clusters. If you're using the latest devel version you can also build ML pipelines that are interoperable with Scala, if that matters to your client.

Scala is a pretty language and a lot of concepts from R carry over since they're both functional languages, but unless you require specific features only available in Scala, R and its ecosystem will probably get you to results faster.
Reply With Quote
  #6  
Old 11-07-2017, 11:48 AM
ALivelySedative's Avatar
ALivelySedative ALivelySedative is offline
Member
CAS
 
Join Date: Dec 2013
Location: Land of the Pine
College: UNC-Chapel Hill Alum
Favorite beer: Red Oak
Posts: 1,522
Default

What's the occasion?
https://www.scalausa.com/
__________________
1/P | 2/FM | 3F/MFE | LC | ST |4/C | 5 | 6 || 7 | 8 | 9
VEE: Econ, Fin, Stat
OC1, OC2
COP
Reply With Quote
  #7  
Old 11-07-2017, 12:11 PM
Colonel Smoothie's Avatar
Colonel Smoothie Colonel Smoothie is offline
Member
CAS
 
Join Date: Sep 2010
College: Jamba Juice University
Favorite beer: AO Amber Ale
Posts: 44,959
Default

Quote:
Originally Posted by Incredible Hulctuary View Post
I've used Scala in the past. It's a great language, if you can get your head around some of its concepts that aren't common in most other languages such as functional programming and how it organizes its type system and multithreading. You may be less effective with it if you don't know Java first, as it compiles to Java class files and can seamlessly call any of the standard Java libraries (or any other Java class if you supply the .class or .jar).

One thing I don't like is that it lets you use arbitrary special characters as method names and operators, and does that with many of its own standard libraries, so it has stuff like <:< and :/ which have real meaning but you can't Google them. And if you're working in a team, you may have a doofus co-worker who likes to make up unreadable names like &/&<=>: instead of using a word or phrase that means what it says.
What did you use Scala for?

I like it so far. Functional programming is pretty neat...I think a skilled programmer could also make the code concise, clean, and readable. Although, it seems really easy to make your code messy and unreadable if you're not diligent.
__________________
Recommended Readings for the EL Actuary || Recommended Readings for the EB Actuary

Quote:
Originally Posted by Wigmeister General View Post
Don't you even think about sending me your resume. I'll turn it into an origami boulder and return it to you.
Reply With Quote
  #8  
Old 11-07-2017, 12:19 PM
Colonel Smoothie's Avatar
Colonel Smoothie Colonel Smoothie is offline
Member
CAS
 
Join Date: Sep 2010
College: Jamba Juice University
Favorite beer: AO Amber Ale
Posts: 44,959
Default

Quote:
Originally Posted by kevinykuo View Post
You can also use Spark with R via sparklyr, which may get you to be productive sooner than learning Scala, especially if you're familiar with dplyr/SQL. It works more or less out of the box with HDInsight and supports kerberized clusters. If you're using the latest devel version you can also build ML pipelines that are interoperable with Scala, if that matters to your client.

Scala is a pretty language and a lot of concepts from R carry over since they're both functional languages, but unless you require specific features only available in Scala, R and its ecosystem will probably get you to results faster.
Thanks! I'll take a look at that - I'm pretty sure if I had used R for the thing that took me all day to do yesterday, it would have taken me 2 minutes if I were using R. It would be really nice if I could just pop open R Studio and deploy an HDInsight cluster and start working without having to worry about cluster configuration.

I'm new to Azure, so I've been deploying it from my web portal, but I'd rather save a template and use that instead, but I have yet to learn how to do that.

I have been looking into things like GraphX - the book I was reading on it may be a little out of date because it claims that Scala is the only language the library works well with, but maybe things have changed since Spark 1.6. I've found that some of the things that make Spark great are hard to grasp without some experience in functional programming, which encouraged me to learn some Scala since one of the popular books on functional programming uses it.
__________________
Recommended Readings for the EL Actuary || Recommended Readings for the EB Actuary

Quote:
Originally Posted by Wigmeister General View Post
Don't you even think about sending me your resume. I'll turn it into an origami boulder and return it to you.
Reply With Quote
  #9  
Old 11-07-2017, 12:27 PM
kevinykuo kevinykuo is offline
CAS
 
Join Date: Nov 2017
Posts: 10
Default

You'd still have to spin up your cluster via Azure CLI or the Azure portal, but that should be straightfoward (if not, it really ought to be someone else's job to admin the clusters.)

What would you be using GraphX for?
Reply With Quote
  #10  
Old 11-07-2017, 10:15 PM
Colonel Smoothie's Avatar
Colonel Smoothie Colonel Smoothie is offline
Member
CAS
 
Join Date: Sep 2010
College: Jamba Juice University
Favorite beer: AO Amber Ale
Posts: 44,959
Default

Quote:
Originally Posted by kevinykuo View Post
You'd still have to spin up your cluster via Azure CLI or the Azure portal, but that should be straightfoward (if not, it really ought to be someone else's job to admin the clusters.)

What would you be using GraphX for?
I've been doing some social network analysis, so I've used it for calculating things like triangle count, clustering coefficient, triadic closure rate, etc.
__________________
Recommended Readings for the EL Actuary || Recommended Readings for the EB Actuary

Quote:
Originally Posted by Wigmeister General View Post
Don't you even think about sending me your resume. I'll turn it into an origami boulder and return it to you.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


All times are GMT -4. The time now is 11:21 AM.


Powered by vBulletin®
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
*PLEASE NOTE: Posts are not checked for accuracy, and do not
represent the views of the Actuarial Outpost or its sponsors.
Page generated in 0.14494 seconds with 9 queries