Databases versus plain text

Question

When dealing with small projects, what do you feel is the break even point for storing data in simple text files, hash tables, etc., versus using a real database? For small

Answer 1

For me, the line is crossed once I have to query my data in ways that involve more than a single relationship. Relating two flat data structures on disk is fairly simple, but once we get beyond that, a set-based language like SQL and formal database relationships actually reduce complexity.

Answer 2

Use whatever persistence technology you're most comfortable with, and scales sufficiently.

YAGNI at least means "Don't add a new technology to your personal stack unless you can't be productive with whatever is already there."

For many (most?) of us, our comfort zone for data persistence is SQL. For some, it might be XML. Just don't write your own until (see paragraph 2).

Answer 3

1) Concurrency. Do you have multiple people accessing the same dataset? Then it's going to get pretty involved to broker all of the different readers and writers in a scalable fashion if you roll your own system.

2) Formatting and relationships: Is your data something that doesn't fit neatly into a table structure? Long nucleotide sequences and stuff like that? That's not really conveniently tabular data.

Another example: Nobody would consider implementing software like Photoshop to store PSDs in a relational format, because the data structures don't really lend themselves to that type of storage or query pattern.

3) ACID (sort of a corollary to #1): If Atomicity, Consistency, Integrity, and Durability are not challenges with a flat file, then go with a flat file.

Answer 4

I think at some point you'll miss the querying capabilities of a database, but you can consider some minimalistic database alternatives:

• SQLite (Great, almost SQL-92 standard compliant)
• shsql
• SQL Server Compact

Answer 5

The problem with small projects is that they become bigger before we know it. And once they do , we start missing the sql capabilities.

Always design such that a db can be utilized later on if required without ripping apart half of the application.

Answer 6

For bioinformatics you may be interested on that: Blast on DB. The guy who is working on that is a friend of mine and has a work on fast similarity sequence search, he found out to make his own binary storage better than using databases at this point.

I don't know specific details about his solution but you probably can exchange one or two ideias mailing the guy, even sharing code.