Database software

[Polly Nomial]

What software do you use to manipulate multiples files with millions of records?

We have been using FoxPro which is powerful enough but soon will be unsupported. Also FoxPro uses an extrodinary amount of the PC's resources.

We are all fairly proficient in SQL programing. Our IT dept is trying to sell us Access, but that thing crawls like a snail through big database files.

[Cho Da]

If you want to go the µserf way, get MSSQL. Beware of massive security problems.
If you want to do it for less, MySQL (http://www.mysql.com). I would guess that thiss forum runs on MySQL.

[Dr T Non-Fan]

SQL, Oracle Sybase, SAS, etc. As long as a separate server's memory, speed, and storage are handling the database needs, you should be fine.
Oh, and don't tell the rest of the company that you have such information easily available. Otherwise your actuarial department turns into a data source.
Also, try to prevent the unsupporting of FoxPro. Your actuarial department head should have enough clout to make sure the department gets what it needs.

[Obi-Wan Kenobi]

Is it the IT folks or the Evil Empire of Redmond that's not going to be supporting FoxPro?

[glenn]

If you want to go the µserf way, get MSSQL. Beware of massive security problems.
If you want to do it for less, MySQL (http://www.mysql.com). I would guess that thiss forum runs on MySQL.
I do everything in MySQL. Reasonably priced and fast for what I want to do. Never had millions of records though no problems with 10's of thousands.

[Polly Nomial]

.......As long as a separate server's memory, speed, and storage are handling the database needs, you should be fine.......

In FoxPro we have it set up so that the data files reside on a separate server, but any data handling is done on our C drive. (probably explains why we have 37 gig hard drives) I assume you are talking about not using the C drive for this purpose. Is this a setup issue or is it the way the software works?

If you want to go the µserf way, get MSSQL

I don't understand "µserf ." Is that a typo or just a term I don't know?

Is it the IT folks or the Evil Empire of Redmond that's not going to be supporting FoxPro?
It's the Evil Empire of Redmond! :D

[Dr T Non-Fan]

I'm not sure about FoxPro. That sounds just like MS-Access.

I think the server has to be set up in a way that it handles everything, and I'm not sure that FoxPro or Access can do these.
I don't think they were created with that in mind

SQL can be very, very difficult, but if all you do is run queries (a "user") it's very efficient.
I suggest a Sybase or SQL server. And a dedicated DBA (or portion of IT department) to handle it.
Yes, it's expensive, and yes, it's worth it. You need data or you don't work as efficiently. If you, as an actuarial department, don't work efficiently, then the company suffers. You don't want to wait to prove me right. And if your company doesn't treat actuaries with the proper respect, then you should find a company that does, since it's better to be on the buying end than on the selling end.

[NoName]

If you want to go the µserf way, get MSSQL

I don't understand "µserf ." Is that a typo or just a term I don't know?


µserf = Microserf = Microsoft. Cho Da must have this board confused with Slashdot.

[Cho Da]

If you want to go the µserf way, get MSSQLI don't understand "µserf ." Is that a typo or just a term I don't know?Is it the IT folks or the Evil Empire of Redmond that's not going to be supporting FoxPro?
It's the Evil Empire of Redmond! :D

See what you become when you given in to the Evil Empire? You become a µserf!

[Incredible Hulctuary]

Any database churning through millions of records will put a strain on a regular PC's resources.

Like MySQL, PostGreSQL (http://www.postgresql.org) is another FREE database that is quite capable of handling a few million records. PostGreSQL has subqueries and stored procedures, unlike the current version (4.0) of MySQL.

[snafu]

Have had tremendous results with the following setup

Under 500K records, MSAccess running on local machines.

Over 500K records, MSSQL running off of a relatively cheap server (or running on a really powerful shared machine). Have easily run 20 gig databases with 20-30 million premium records in a single table and 10 million related loss records.

The hardest part about using MSSQL is getting the large quantities of data into the database. Recommend some research on this item before hand as it can be tricky.

[IMHO]

The hardest part about using MSSQL is getting the large quantities of data into the database. Recommend some research on this item before hand as it can be tricky.

That's why I would recommend MSSQL Server. It has the best instruments for upload/data manipulation. There is nothing tricky in DBF files upload.
And I have never heard it is expensive. Talking about security issues with people who are happy with FoxPro is a joke itself.

[wtt]

That's why I would recommend MSSQL Server. It has the best instruments for upload/data manipulation. There is nothing tricky in DBF files upload.
And I have never heard it is expensive. Talking about security issues with people who are happy with FoxPro is a joke itself.
MySQL is good. It is not only free, it also has the web admin interface developed by other people.

[vchagas]

If there is budget, I would recomend SAS. It does everything a DBMS do (for actuarial purposes) and much more.

Like many others I followed the Microsoft PATH
ACCESS -> FOX PRO -> SQL SERVER

But eventually ended up with the following configuration:
SAS 9.1 Analytics Server + SAS/ACCESS for ODBC + Enterprise Guide 4.1

No more size limits (except disk capacity), no more needs to compact the databases and endless logs.

[MillyT]

Hello, does anyone know about some of the advantages and disadvantages of moving a mysql system to the cloud using cloud database (http://xeround.com/) solutions (such as this one), any known pitfalls to be aware of, and how do I know that it's time to move to a cloud based solution, or on the contrary that a system isn't fit for cloud environment.
Any thought will be appreciated,

Thanks,
Milly.

[ADoubleDot]

In FoxPro we have it set up so that the data files reside on a separate server, but any data handling is done on our C drive. (probably explains why we have 37 gig hard drives) I assume you are talking about not using the C drive for this purpose. Is this a setup issue or is it the way the software works?



I don't understand "µserf ." Is that a typo or just a term I don't know?


It's the Evil Empire of Redmond! :D
Was 37 GB considered large in 2002? Seems odd.

[atfl]

Was 37 GB considered large in 2002? Seems odd.

I think my largest MP3 back then was ~9MB. Biggest game install was 2 or 3GB. Now my smallest FLAC or game are of those sizes.

I could see how a 37GB database could be considered gigantic.

Heck I remember someone with a terabyte of total hard drive space in '02 was picked on for amassing a collection of pr0n of that size (no other reason one needed that much space).

[AndrewC]

There are a few things that you may want to consider:

- performance
- scalability
- manageability
- security
- extensibility

And today's enterprise grade database offers more than just data storage and access; they also provide features like reports, business intelligence, data migration, data mining, master data management. You don't often find these features from desktop database software such as Access and FoxPro.

If you're using FoxPro, I assume you are not just using its database feature, you may be using its development tool as well. Are you going to migrate the development part to something else as well?

[SpaceActuary]

This was a 9-year-old thread that got bumped...

Hello, does anyone know about some of the advantages and disadvantages of moving a mysql system to the cloud using cloud database (http://xeround.com/) solutions (such as this one), any known pitfalls to be aware of, and how do I know that it's time to move to a cloud based solution, or on the contrary that a system isn't fit for cloud environment.
Any thought will be appreciated,

Thanks,
Milly.

[AndrewC]

Hello, does anyone know about some of the advantages and disadvantages of moving a mysql system to the cloud using cloud database (http://xeround.com/) solutions (such as this one), any known pitfalls to be aware of, and how do I know that it's time to move to a cloud based solution, or on the contrary that a system isn't fit for cloud environment.
Any thought will be appreciated,

Thanks,
Milly.

Advantage and disadvantage are relative! Are you running mySQL on Linux or windows? What is your business continuity plan? What is your current application configuration? Windows application? Web application? ASP.Net? J2EE? Web Services?

You may also want to examine if there is any legal obligation or business requirements, e.g. you have to store the data in your own country.


It is difficult to make a recommendation without knowing the background.


But you may want to compare more than 1 cloud provider, e.g.

http://aws.amazon.com/ec2/
http://www.microsoft.com/windowsazure/

At least Apple choose both Amzon and Microsoft to be its iCloud provider.