efficient bulk update rails database

Question

I\'m trying to build a rake utility that will update my database every so often.

This is the code I have so far:

namespace :utils do

  # utils:updat         


        

Answer 1

As Larry says, use your DB-specific import utilities if the file comes in the format you want. However, if you need to manipulate the data before inserting, you can generate a single INSERT query with data for many rows, which is faster than using a separate query for each row (as ActiveRecord will do). For example:

INSERT INTO iptocs (ip_from, ip_to, country_code) VALUES
  ('xxx', 'xxx', 'xxx'),
  ('yyy', 'yyy', 'yyy'),
  ...;

Answer 2

You could generate a text file with all INSERTs you need and then execute:

mysql -u user -p db_name < mytextfile.txt

Not sure if this will be any faster but worth a try...

Answer 3

Use the database level utilities for high speed Luke!

Unfortunately, they're db specific. But they are fast For mysql, see http://dev.mysql.com/doc/refman/5.1/en/load-data.html

Answer 4

I'm currently experimenting with activerecord-import, which sounds very promising:

https://github.com/zdennis/activerecord-import

Answer 5

Have you tried to use AR Extensions for bulk import? You get impressive performance improvements when you are inserting 1000's of rows to DB. Visit their website for more details.

Refer to these examples for more information

Usage Example 1

Usage Example 2

Usage Example 3