At what point is it worth using a database?

Question

I have a question relating to databases and at what point is worth diving into one. I am primarily an embedded engineer, but I am writing an application using Qt to interfa

Answer 1

What you are describing doesn't sound like a typical business application, and many of the answers already posted here assume that this is the kind of application you are talking about, so let me offer a different perspective.

Whether or not you use a database for 700 items is going to depend greatly on the nature of the data.

I would say that, about 90% of the time at this scale, you will benefit from a light-weight database like SQLite, provided that:

1. The data may potentially grow substantially larger than what you are describing,
2. The data may be shared by more than one user,
3. You may need to run queries against the data (which I don't think you're doing right now), and
4. The data can easily be described in table form.

The other 10% of the time, your data will be highly structured, hierarchical, object-based, and doesn't neatly fit into the table model of a database or Excel table. If this is the case, consider using XML files.

I know developers instinctively like to throw databases at problems like this, but if you are currently using Excel data to design user interfaces (or display configuration settings), rather than display a customer record, XML may be a better fit. XML is more expressive than either Excel or database tables, and can be easily manipulated with a simple text editor.

XML parsers and data binders for C++ are easy to find.

Answer 2

"At what point is it worth using a database?"

If and when you've got data to manage ?

Answer 3

There isn't a specific point at which a database is worthwhile. Instead I usually ask the following questions:

• Is the amount of data the application uses/creates growing?
• Is the upper limit of this data growth unknown (or unclear)?
• Will the application need to aggregate or filter this data?
• Could there be future uses of the data that may not be obvious right now?
• Is performance of data retrieval and/or storage important?
• Are there (or could there be) multiple users of the application who share data?

If I answer 'Yes' to most of these questions I almost always choose a database (as opposed to other options such as XML/ini/CSV/Excel/text files or the filesystem).

Also, if the application will have many users who could be accessing the data concurrently, I'll lean towards a full database server (MySQL, SQl Server, Oracle etc).

But often in a single user (or small concurrency) situation, a local database such as SQLite cannot be beaten for portability and ease of deployment.

Answer 4

Don't forget that the appropriate database can be quite different depending on your requirements (and don't forget that a text file could be used as a database if you're requirements are simple enough - for example, config files are just a specific kind of database). Such parameters might be:

• number of records
• size of data items
• does the database need to be shared with other devices? Concurrently?
• how complex are the relationships between the various pieces of data
• is the database read only (created at build time and not changed, for example)?
• does the database need to be updated by multiple entities concurrently?
• do you need to support complex queries?

For a database with 700 entries, an in-memory sorted array loaded from a text file might well be appropriate. But I could also see the need for an embedded SQL database or maybe having the controller request data from the database over a network connection depending on what the various requirements (and resource limitations) are.

Answer 5

It sounds like your application is running on a desktop computer and simply communicating to the embedded device.

As such using a database is much more feasible. Using one on an embedded platform is a much more complex issue.

On the desktop front I use a database when there is the need to store new information continuously and the need to extract that information in a relational way. What i don't use databases for is storing static information, information i read once at load and thats it. The exception is when the application has many users and there is the need to storage this information on a per user basis.

It sounds be to me like your collecting information from your embedded device, storing it somehow, then using it later to display via a GUI.

This is a good case for using a database, especially if you can architect the system such that there is a data collection daemon that manages the continuous communication with the embedded device. This app can then just write the data into the database. When the GUI is launched it can extract the data for display.

Using the database will also ease your GUI develop if you need to display different views, such as "show me all the entries between 2 dates". With a database you just ask it for the correct values to display with a proper SQL query and the GUI displays whatever comes back allowing you to decouple much of the "business logic" code from the GUI.

Answer 6

When you have a lot of data that you're not sure how they will be exploited in the future.

For example you might want to add an SQLite database in an embedded application that need to register statistics that you're not sure how will be used. Later you send the full database for injection in a bigger one running on a central server and those data can easily be exploited, using requests.

In fact, if your application's purpose is to "gather data" then having a database is a must have.