What arguments to use to explain why SQL Server is far better than a flat file

Question

The higher-ups in my company were told by good friends that flat files are the way to go, and we should switch from SQL Server to them for everything we do. We have over 300 servers and hundreds of different databases. From just the few I'm involved with we have > 10 billion records in quite a few of them with upwards of 100k new records a day and who knows how many updates... Me and a couple others need to come up with a response saying why we shouldn't do this. Most of our stuff is ASP.NET with some legacy ASP. We thought that making a simple console app that tests/times the same interactions between a flat file (stored on the network) and SQL over the network doing large inserts, searches, updates etc along with things like network disconnects randomly. This would show them how bad flat files can be, especially when you are dealing with millions of records.

What things should I use in my response? What should I do with my demo code to illustrate this?

My sort list so far:

• Security
• Concurrent access
• Performance with large amounts of data
• Amount of time to do such a massive rewrite/switch and huge $ cost
• Lack of transactions
• PITA to map relational data to flat files
• NTFS doesn't support tons of files in a directory well
• Lack of Adhoc data searching/manipulation
• Enforcing data integrity
• Recovery from network outage
• Client delay while waiting for other clients changes to commit
• Most everybody stopped using flat files for this type of storage long ago for good reason
• Load balancing/replication

I fear that this will be a great post on the Daily WTF someday if I can't stop it now.

Additionally

Does anyone know if anything about HIPPA could be used in this fight? Many of our records are patient records...

Answer 1

1. Data integrity. First, you can enforce it in a database and cannot in a flat file. Second, you can ensure you have referential integrity between different entities to prevent orphaning rows.

2. Efficiency in storage depending on the nature of the data. If the data is naturally broken into entities, then a database will be more efficient than lots of flat files from the standpoint of the additional code that will need to be written in the case of flat files in order to join data.

3. Native query capabilities. You can query against a database natively whereas you cannot with a flat file. With a flat file you have to load the file into some other environment (e.g. a C# application) and use its capabilities to query against it.

4. Format integrity. The database format is more rigid which means more consistent. A flat file can easily change in a way that the code that reads the flat file(s) will break. The difference is related to #3. In a database, if the schema changes, you can still query against it using native tools. If the flat file format changes, you have to effectively do a search because the code that reads it will likely be broken.

5. "Universal" language. SQL is somewhat ubiquitous where as the structure of the flat file is far more malleable.

Answer 2

I'd also mention data corruption. Most modern SQL databases can have the power killed on the server, or have the server instance crash and you won't (shouldn't) loose data. Flat files aren't really that way.

Also I'd mention search times. Perhaps even write a simple flat file database with 1mil entries and show search times vs MS SQL. With indexes you should be able to search a SQL database thousands of times faster.

I'd also be careful how quickly you write off flat files. Id go so far as saying "it's a good idea for many cases, but in our case....". This way you won't sound like you're not listening to the other views. Tact in situations like this is a major thing to consider. They may be horribly wrong, but you have to convince your boss of that.

Answer 3

What do they gain from using flat files? The conversion process will be hundreds of hours - hours they pay for. How quickly can flat files generate a positive return on that investment? Provide a rough cost estimate. Translate the technical considerations into money (costs), and it puts the problem in their perspective.

On top of just the data conversion, add in the hidden costs for duplicating a database's capabilities...

• Indexing
• Transaction processing
• Logging
• Access control
• Performance
• Security

Answer 4

Databases allow you to easily index your data to be able to particular records or groups of records by searching any number of different columns.

With flat files you have to write your own indexing mechanisms. There is no need to do all that work again when the database does it for you already.

Answer 5

If you use "text files", you'll need to build an interface on top of it which Microsoft has already done for you and called it SQL Server.

Ask your managers if it makes sense to your company to spend all these resources building a home-made database system (because really that's what it is), or would these resources be better spent focusing on the business.

• Performance: SQL Server is built for storing conveniently searchable data. It has optimized data structures in memory built with searching/inserting/deleting in mind. Usage of the disk is lowered, as data regularly queried is kept in memory.

• Business partners: if you ever plan to do B2B with 3rd party companies, SQL Server has built-in functionality for it called Linked Servers. If you have only a bunch of files, your business partner will give up on you as no data interconnection is possible. Unless you want to re-invent the wheel again, and build an interface for each business partner you have.

• Clustering: you can easily cluster servers in SQL Server for high availability and speed, a lot more than what's possible with text based solution.

Answer 6

You have a nice start to your list. The items I would add include:

1. Data integrity - SQL engines provide built-in mechanisms (relationships, constraints, triggers, etc.) that make it very simple to reduce the amount of "bad" data in your system. You would need to hand code all data constraint separately if you use flat files.
2. Add-Hoc data retrieval - SQL engines, through the use of SELECT statements, provide a means of filtering and summarizing your data with very little code. If you are using flat files, considerably more code is needed to get the same results.

These items can be replicated if you want to take the time to build a data engine, but what would be the point? SQL engines already provide these benefits.

Answer 7

I don't think I can even start to list the reasons. I think my head is going to explode. I'll take the risk though to try to help you...

• Simulate a network outage and show what happens to one of the files at that point
• Demo the horrors of a half-committed transaction because text files don't pass the ACID test
• If it's a multi-user application, show how long a client has to wait when 500 connections are all trying to update the same text file
• Try to politely explain why the best approach to making business decisions is to listen to the professionals who you are paying money and who know the domain (in this case, IT) and not your buddy who doesn't have a clue (maybe leave out that last bit)
• Mention the fact that 99% (made up number) of the business world uses relational databases for their important data, not text files and there's probably a reason for that
• Show what happens to your application when someone goes into the text file and types in "haha!" for a column that's supposed to be an integer

Answer 8

If you are a public company, the shareholders would be well served to know this is being seriously contemplated. "We" all know this is a ridiculous suggestion given the size and scope of your operation. Patient records must be protected, not only from security breaches but from irresponsible exposure to loss - lives may depend up the data. If the Executives care at all about the patients, THIS should be their highest concern.

I worked with IBM 370 mainframes from '74 onwards and the day that DB2 took over from plain old flat files, VSAM and ISAM was a milestone day. Haven't looked back to flat-file storage, except for streaming data, in my 25 years with RDBMSs of 4 flavors.

If I owned stock in "you", dumping it in a hurry the moment the project took off would seem appropriate...

Answer 9

Your list is a great start of reasons for sticking with a database.

However, I would recommend that if you're talking to a technical person, to shy away from technical reasons in a recommendation because they might come across as biased.

Here are my 2 points against flat file data storage:

1) Security - HIPPA audits require that patient data remain in a secure environment. The common database systems (Oracle, Microsoft SQL, MySQL) have methods for implementing HIPPA compliant security access. Doing so on a flat-file would be difficult, at best.

Side note: I've also seen medical practices that encrypt the patient name in the database to add extra layers of protection & compliance to ensure even if their DB is compromised that the patient records are not at risk.

2) Reporting - Reporting from any structured database system is simple and common. There are hundreds of thousands of developers that can perform this task. Reporting from flat-files will require an above-average developer. And, because there is no generally accepted method for doing reporting off of a flat-file database, one developer might do things different than another. This could impact the talent pool able to work on a home-grown flat-file system, and ultimately drive costs up by having to support that type of a system.

I hope that helps.

Answer 10

How do you create a relational model with plain text files?

Or are you planning to use a different file for each entity?

Answer 11

Pro file system:

1. Stable (less lines of code = less bugs, easier to understand, more reliable)
2. Faster with huge data blobs
3. Searching/sorting is somewhat slow (but sort can be faster than SQL's order by)

So you'd chose a filesystem to create log files, for example. Logging into a DB is useless unless you need to do complex analysis of the data.

Pro DB:

2. Transactions (which includes concurrent access)
3. It can search through huge amounts of records (but not through huge blobs of data)
4. Chopping the data in all kinds of ways with queries is easy (well, if you know your SQL and the special "oddities" of your DB)

So if you need to add data rarely but search it often, select parts of it by certain criteria or aggregate values, a DB is for you.

Answer 12

NTFS does not support mass amounts of .txt files well. Depending on how a flat file system is developed, the health of a harddrive can become an issue. A lot of older file systems use mass amount of small .txt files to store data. It's bad design, but tends to happen as a flat file system gets older.

Fragmentation becomes an issue, and you lose a text file here and there, causing you to lose small amounts of data. Health of a hard drive should not be an issue when it comes to database design.

Answer 13

This is indeed, on the part of your employer, a MAJOR WTF if he's seriously proposing flat files for everything...

You already know the reasons (oh - add Replication / Load Balancing to your list) - what you need to do now is to convince him of them. My approach on this would two fold.

First of all, I would write a script in whatever tool you currently use to perform a basic operation using SQL, and have it timed. I would then write another script in which you sincerely try to get a flat text solution working, and then highlight the difference in performance. Give him both sets of code so he knows you aren't cheating.

Point out that technology evolves, and that just because someone was successful 20 years ago, this does not automatically entitle them to a credible opinion now.

You might also want to mention the scope for errors in decoding / encoding information in text files, that it would be trivial for someone to steal them, and the costs (justify your estimate) in adapting the current code base to use text files.

I would then ask serious questions of management - foremost amongst them, and I would ask this DIRECTLY, is "Why are you prepared to overrule your technical staff on technical matters" based on one other individual's opinion - especially when said individual is not as familiar with our set up as we are...

I'd also then use the phrase "I do not mean to belittle you, but I seriously feel I have to intervene at this point for the good of the company..."

Another approach - turn the tables - have Mr. Wonderful supply arguments as to why text files are the way forward. You'll then either a) Learn something (not likely), or b) Be in a position to utterly destroy his arguments.

Good luck with this - I feel your pain...

Martin

Answer 14

I suggest you get your retalliation in first, post on Daily WTF now.

As to your question: a business reason would be why does your boss want to rewrite all your systems. From scratch as you would, effectively, have to write your own database system.

For a development reason, you would lose access to the SQL server ecosystem, all the libraries, tools, utilities.

Perhaps the guy that suggested this is actually thinking of going into competition with your company.

Answer 15

Simplest way to refute this argument - name a fortune 500 company that processes data on this scale using flat files?

Now name a fortune 500 company that doesn't use a relational database...

Case closed.

Answer 16

Something is really fishy here. For someone to get the terminology right ( "flat file" ) but not know how overwhelmingly stupid an idea that is, it just doesn't add up. I would be willing to be your manager is non-technical, but the person your manager is talking to is. This sounds more like a lost in translation problem.

Are you sure they don't mean no-SQL, as if you are in a document centric environment, moving away from a relational database actually does make sense in some regards, while still having many of the positives of a tradition RDBMS.

So, instead of justifying why SQL is better than flat files, I would invert the problem and ask what problems flat files are meant to solve. I would put odds on money that this is a communication problem.

If its not and your company is actually considering replacing its DB with a home grown flat file system off the recommendation of "a friend", convincing your manager why he is wrong is the least of your worries. Instead, dust off and start circulating your resume.

Answer 17

•Amount of time to do such a massive rewrite/switch and huge $ cost

It's not just amount of time it is the introduction of new bugs. A re-write of these proportions would cause things that currenty work to break.

I'd suggest a giving him a cost estimate of the hours to do such a rewrite for just one system and then the number of systems that would need to change. Once they have a cost estimate, they will run from this as fast as they can.

Managers like numbers, so do a formal written decision analysis. Compare the two proposals by benefits and risks, side by side with numeric values. When you get to cost 0 to maintain and 100,000,000 to convert they will get the point.

Answer 18

The people that doesn't distinguish between flat files and sql, doesnt understand all arguments that you say before.

---

The explanation must simple as possible, something like this:
SQL is a some kind of search/concurrency wrapper around the flat files.
All the problems that exist currently, will stay even the company going to write the wrapper from zero.

Also you must to give some other way to resolve the current problems, use smart words like advanced BLL or install/uninstall scripting environment. :)

Answer 19

You have to speak executive. Without saying it, make them realize they're in way over their heads here. Here's some ammunition:

Database theory is hardcore computer science. We're talking about building a scalable system that can handle millions of records and tolerate disasters without putting everyone out of business.

This is the work of PhD-level specialists. They've been refining the field for a good 20 years now, and the great thing about that is this: it allows us to specialize in building business systems.

If you have to, come right out and say that this just isn't done in the enterprise. It would be costly and the result would be inferior. It's exactly the kind of wheel that developers love to reinvent, and in my opinion the only time you should is if the result is going to be a product or service that you can sell. And it won't be.