Where to store SQL commands for execution

Question

We face code quality issues because of inline mysql queries. Having self-written mysql queries really clutters the code and also increases code base etc.

Our code is clu

Answer 1

This is no doubt a million dollar question, and I think the right solution depends always on the case.

Here goes my thoughts. Hope could help:

One simple trick (which, in fact, I read that it is surprisingly more efficient than joining strings with "+") is to use arrays of strings for each row and join them.

It continues being a mess but, at least for me, a bit clearer (specially when using, as I do, "\n" as separator instead of spaces, to make resulting strings more readable when printed out for debugging).

Example:

var sql = [
    "select foo.bar",
    "from baz",
    "join foo on (",
    "  foo.bazId = baz.id",
    ")", // I always leave the last comma to avoid errors on possible query grow.
].join("\n"); // or .join(" ") if you prefer.

As a hint, I use that syntax in my own SQL "building" library. It may not work in too complex queries but, if you have cases in which provided parameters could vary, it is very helpful to avoid (also subotptimal) "coalesce" messes by fully removing unneeded query parts. It is also on GitHub, (and it isn't too complex code), so you can extend it if you feel it useful.

If you prefer separate files:

About having single or multiple files, having multiple files is less efficient from the point of view of reading efficiency (more file open/close overhead and harder OS level caching). But, if you load all of them single time at startup, it is not in fact a hardly noticeable difference.

So, the only drawback (for me) is that it is too hard to have a "global glance" of your query collection. Even, if you have very huge amount of queries, I think it is better to mix both approaches. That is: group related queries in the same file so you have single file per each module, submodel or whatever criteria you chosen.

Of course: Single file would result in relatively "huge" file, also difficult to handle "at first". But I (hardly) use vim's marker based folding (foldmethod=marker) which is very helpfull to handle that files.

Of course: if you don't (yet) use vim (truly??), you wouldn't have that option, but sure there is another alternative in your editor. If not, you always can use syntax folding and something like "function (my_tag) {" as markers.

For example:

---(Query 1)---------------------/*{{{*/
select foo from bar;
---------------------------------/*}}}*/

---(Query 2)---------------------/*{{{*/
select foo.baz 
from foo
join bar using (foobar)
---------------------------------/*}}}*/

...when folded, I see it as:

+--  3 línies: ---(Query 1)------------------------------------------------

+--  5 línies: ---(Query 2)------------------------------------------------

Which, using properly selected labels, is much more handy to manage and, from the parsing point of view, is not difficult to parse the whole file splitting queries by that separation rows and using labels as keys to index the queries.

Dirty example:

#!/usr/bin/env node
"use strict";

var Fs = require("fs");

var src = Fs.readFileSync("./test.sql");

var queries = {};


var label = false;

String(src).split("\n").map(function(row){
    var m = row.match(/^-+\((.*?)\)-+[/*{]*$/);
    if (m) return queries[label = m[1].replace(" ", "_").toLowerCase()] = "";
    if(row.match(/^-+[/*}]*$/)) return label = false;
    if (label) queries[label] += row+"\n";
});

console.log(queries);
// { query_1: 'select foo from bar;\n',
//   query_2: 'select foo.baz \nfrom foo\njoin bar using (foobar)\n' }

console.log(queries["query_1"]);
// select foo from bar;

console.log(queries["query_2"]);
// select foo.baz
// from foo
// join bar using (foobar)

Finally (idea), if you do as much effort, wouldn't be a bad idea to add some boolean mark together with each query label telling if that query is intended to be used frequently or only occasionally. Then you can use that information to prepare those statements at application startup or only when they are going to be used more than single time.