Building SQL update statements using a Bash script

问题

For a recent project I need to loop through rows of a csv (ignore.csv) with 3 columns:

acctnum, errcode, date ← in that order…

(Column naming doesn't matter here, its just for context) and use those variables to build SQL statements.

The CSV may have 500 rows or it might have 20k. My SQL statement should look like this:

UPDATE O.ACCT_ERR SET REC_ACTV_IND='T'
  WHERE BUS_DT='20200603' and ERR_CD='R4442' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM O.ACCT
      WHERE ACCT_SRCH_NBR='10100000011'
  );

Currently my sloppy example in Bash is this:

while IFS=, read -r field1 field2 field3;
do
   echo "UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T' WHERE BUS_DT='$field3' and ETL_ERR_CD='$field2' AND ACCT_KEY in (SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO WHERE ACCT_SRCH_NBR='$field1');" > sqlfile.txt
done < ignore.csv

Which works for just one row as verified in output of sqlfile.txt.

But how can I iterate over it to print all rows of the CSV?

I am still awful at looping logic - lucky to be a Prod Support guy in that aspect.

Any help is much appreciated.

回答1:

As Barmar pointed-out:

You're overwriting the file every time you echo inside the loop.

You can do it this way with including space into the field separators of the CSV.

With some important caution:

Shell cannot reliably parse CSV fields witch may include quotes, escaping of quotes. You should consider using a command able to parse CSV correctly and return values as arguments or null delimited fields like csvtool.
If your SQL database engine knows about PREPARE statements, it would be safer to PREPARE the SQL UPDATE request, and feed it with arguments. You'd prepare the request before the loop, and then feed the arguments and execute the prepared statement in the loop.

#!/usr/bin/env bash

while IFS=', ' read -r acctnum errcode date; do

  cat <<SQL
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='$date' and ETL_ERR_CD='$errcode' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='$acctnum');
SQL
done < ignore.csv > sqlfile.txt

Parsing and converting your `exclude.csv` into SQL queries using `csvtool`:

excludeCSV2sql:

#!/usr/bin/env bash

# This script uses csvtool to parse exclude.csv CSV data from stdin
# and output SQL queries to stdout

# Convert arguments from csvtool call, into an SQL query
to_sql ()
{
  # Double single-quote for SQL string values if any
  local -- \
    acct_num="${1//\'/\'\'}" \
    err_code="${2//\'/\'\'}" \
    date="${3//\'/\'\'}"

  cat <<SQL
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='$date' AND ETL_ERR_CD='$err_code' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='$acct_num'
  );
SQL
}

# Export for use in csvtool call
export -f to_sql

# Process CSV from stdin
csvtool call to_sql -

Testing:

Make the above script executable:

chmod +x excludeCSV2sql

Create a sample test_exclude.csv:

cat >test_exclude.csv <<CSV
foo,bar,baz
here,it's using a single quote, string
this, "has a double-quoted string
with a newline", in it
10100000012, "R4242, has comma", 20200524
10100000042, R1337, 20200525
CSV

Run the test:

./excludeCSV2sql <test_exclude.csv >test.sql

Check the result:

test.sql

UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='baz' AND ETL_ERR_CD='bar' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='foo'
  );
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='string' AND ETL_ERR_CD='it''s using a single quote' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='here'
  );
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='in it' AND ETL_ERR_CD='has a double-quoted string
with a newline' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='this'
  );
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='20200524' AND ETL_ERR_CD='R4242, has comma' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='10100000012'
  );
UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T'
  WHERE BUS_DT='20200525' AND ETL_ERR_CD='R1337' AND ACCT_KEY
  IN (
    SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO
      WHERE ACCT_SRCH_NBR='10100000042'
  );

回答2:

You're overwriting the file every time you echo inside the loop. Move the output redirection to the end of the loop.

while IFS=, read -r field1 field2 field3;
do
   echo "UPDATE ODS.PERF_ACCT_ERR_DTL SET REC_ACTV_IND='T' WHERE BUS_DT='$field3' and ETL_ERR_CD='$field2' AND ACCT_KEY in (SELECT ACCT_KEY FROM ODS.ACCT_PORTFOLIO WHERE ACCT_SRCH_NBR='$field1');"
done < ignore.csv > sqlfile.txt

来源：https://stackoverflow.com/questions/62205458/building-sql-update-statements-using-a-bash-script

标签

sql