At what point is it worth using a database?

Question

I have a question relating to databases and at what point is worth diving into one. I am primarily an embedded engineer, but I am writing an application using Qt to interfa

Answer 1

I recommend you to introduce a Database in your app, your application will gain flexibility and will be easier to maintain and to improve with new features in the future.
I would start with a lightweight file based db like Sqlite.
With a well designed db you'll have:

1. Reduced data redundancy
2. Greater data integrity
3. Improved data security

Last but not least, using a database will save you from the Excel import/update/export Hell!

Answer 2

A database is worthwhile when:

1. Your application evolves to some form of data driven execution.
2. You're spending time designing and developing external data storage structures.
3. Sharing data between applications or organizations (including individual people)
4. The data is no longer short and simple.
5. Data Duplication

Evolution to Data Driven Execution
When the data is changing but the execution is not, this is a sign of a data driven program or parts of the program are data driven. A set of configuration options is a sign of a data driven function, but the whole application may not be data driven. In any case, a database can help manage the data. (The database library or application does not have to be huge like Oracle, but can be lean and mean like SQLite).

Design & Development of External Data Structures
Posting questions to Stack Overflow about serialization or converting trees and lists to use files is a good indication your program has graduated to using a database. Also, if you are spending any amount of time designing algorithms to store data in a file or designing the data in a file is a good time to research the usage of a database.

Sharing Data
Whether your application is sharing data with another application, another organization or another person, a database can assist. By using a database, data consistency is easier to achieve. One of the big issues in problem investigation is that teams are not using the same data. The customer may use one set of data; the validation team another and development using a different set of data. A database makes versioning the data easier and allows entities to use the same data.

Complex Data
Programs start out using small tables of hard coded data. This evolves into using dynamic data with maps, trees and lists. Sometimes the data expands from simple two columns to 8 or more. Database theory and databases can ease the complexity of organizing data. Let the database worry about managing the data and free up your application and your development time. After all, how the data is managed is not as important as to the quality of the data and it's accessibility.

Data Duplication
Often times, when data grows, there is an ever growing attraction for duplicate data. Databases and database theory can minimize the duplication of data. Databases can be configured to warn against duplications.

Moving to using a database has many factors to be considered. Some include but are not limited to: data complexity, data duplication (including parts of the data), project deadlines, development costs and licensing issues. If your program can run more efficiently with a database, then do so. A database may also save development time (and money). There are other tasks that you and your application can be performing than managing data. Leave data management up to the experts.

Answer 3

You don't need a database if you have a few thousand rows in one or two tables to handle in a single user app (for the embedded point).

If it is for multiple users (concurrent access, locking) or the need of transactions you definitly should consider a database. Handling complex datastructures in normalized tables and maintain integrety, or a huge amount of data would be another indication you should use a database.

Answer 4

Reasons for using a database:

• Concurrent writes. It's easy to achieve concurrency in databases
• Easy querying. SQL queries tend to be much concise than procedural code to search data. UPDATEs, INSERT INTOs can also do lots of stuff with very little code
• Integrity. Constraints are very easy to define and are enforced without writing code. If you have a non-null constraint, you can rest assured that the value won't be null, no need to write checks anywhere. If you have a foreign key constraint in place, you won't have "dangling references".
• Performance over large datasets. Indexing is very simple to add to an SQL database

Reasons for not using a database:

• It tends to be an extra dependency (although there exist very lightweight databases- I like H2 for Java, for instance)
• Data not well suited to a relational schema. Things that are basically key/value maps. XML (although databases often support XPath, etc.).
• Sometimes files are more convenient. They can be diff'ed, merged, edited with a plain text editor, etc. Sometimes spreadsheets can be more practical (you don't have to build an editor- you can use a spreadsheet program)
• Your data is already somewhere else

Answer 5

To add a negative: not suitable for real-time processing, due to non-deterministic latency. However, It would be quite ample for looking up and setting operating parameters, for instance during startup. I would not put database accesses on critical time paths.

Answer 6

We are also facing a similar situation. We have set of data coming from different test setups and it is currently being dumped into excel sheets, processed using Perl or VBA.

We found out this method had lot of problems:

i. Managing data using excel sheets is quite cumbersome. After some time you have a whole lot of excel sheets and there is no easy way to retrieve required data from it.

ii. People start sending the excel sheets to and fro for comments and review through e-mails. E-Mail becomes the primary mode of managing the comments related to the data. These comments are lost at a later point of time and there is no way of retrieving it back.

iii. Multiple copies of the files get created and changes in one copy are not reflected in the other - there is no versioning.

This is for the same reasons we have decided to move to a database based solution and are currently working on it. Let me summaries what we are trying to do:

i. The database is in a central server accessible by PC in all the test setups.

ii. All the data goes into a temporary location (local hard disk in files) as soon as it is generated. From the files, it is pushed into database by a process running in the background (so even if there is a network problem, data will be present in the local files system).

iii. We have a web based application which allows users to log in and access data in the format they want. The portal will allow them to add comment, generate different kind of reports, share it with other users after review etc. It will also have the ability to export data into excel sheet, just in case you need to take it with you.

Let know if this can be better implemented.