Importing from CSV in Rails. How to do per batches, not all at once?

问题

def self.import file, organization
  counter = 0
  CSV.foreach(file.path, encoding: 'windows-1251:utf-8', headers: true) do |row|
    name = (row["First Name"].to_s + " " + row["Last Name"].to_s).titleize
    customer = Customer.create(
      name: name,
      phone: row["Main Phone"],
      email: row["Main Email"],
      address: row["Address"],
      repair_shop: repair_shop
    )
    puts "#{name} - #{customer.errors.full_messages.join(',')}" if   customer.errors.any?
    counter += 1 if customer.persisted?
  end
  message = "Imported #{counter} users."
end

This is the code I have so far. I'm importing files with 10,000 rows, so it overwhelms my production server in processing.

How could I do this in batches?

回答1:

Taken from https://satishonrails.wordpress.com/2007/07/18/how-to-import-csv-file-in-rails/

Simply add a periodic explicit garbage collection:

def self.import file, organization
  counter = 0
  CSV.foreach(file.path, encoding: 'windows-1251:utf-8', headers: true).with_index do |row, i|
    name = (row["First Name"].to_s + " " + row["Last Name"].to_s).titleize
    customer = Customer.create(
      name: name,
      phone: row["Main Phone"],
      email: row["Main Email"],
      address: row["Address"],
      repair_shop: repair_shop
    )
    puts "#{name} - #{customer.errors.full_messages.join(',')}" if   customer.errors.any?
    counter += 1 if customer.persisted?
    GC.start if i % 100 == 0 # forcing garbage collection
  end
  message = "Imported #{counter} users."
end

This way you will guarantee that your server will not run out of memory. I have checked it in practice, it really worked.

来源：https://stackoverflow.com/questions/32848049/importing-from-csv-in-rails-how-to-do-per-batches-not-all-at-once