Invoking a large set of SQL from a Rails 4 application

Question

I have a Rails 4 application that I use in conjunction with sidekiq to run asynchronous jobs. One of the jobs I normally run outside of my Rails application is

Answer 1

Executing one query is - as outlined by other people - quite simply done through

ActiveRecord::Base.connection.execute("SELECT COUNT(*) FROM users")

You are talking about a 20.000 line sql script of multiple queries. Assuming you have the file somewhat under control, you can extract the individual queries from it.

script = Rails.root.join("lib").join("script.sql").read # ah, Pathnames

# this needs to match the delimiter of your queries
STATEMENT_SEPARATOR = ";\n\n"


ActiveRecord::Base.transaction do
  script.split(STATEMENT_SEPARATOR).each do |stmt|
    ActiveRecord::Base.connection.execute(stmt)
  end
end

If you're lucky, then the query delimiter could be ";\n\n", but this depends of course on your script. We had in another example "\x0" as delimiter. The point is that you split the script into queries to send them to the database. I wrapped it in a transaction, to let the database know that there is coming more than one statement. The block commits when no exception is raised while sending the script-queries.

If you do not have the script-file under control, start talking to those who control it to get a reliable delimiter. If it's not under your control and you cannot talk to the one who controls it, you wouldn't execute it, I guess :-).

UPDATE

This is a generic way to solve this. For PostgreSQL, you don't need to split the statements manually. You can just send them all at once via execute. For MySQL, there seem to be solutions to get the adapter into a CLIENT_MULTI_STATEMENTS mode.