Ruby scraper. How to export to CSV?

问题

I wrote this ruby script to scrape product info from the manufacturer website. The scraping and storage of the product objects in an array works, but I can't figure out how to export the array data to a csv file. This error is being thrown: scraper.rb:45: undefined method `send_data' for main:Object (NoMethodError)

I do not understand this piece of code. What's this doing and why isn't it working right?

  send_data csv_data, 
            :type => 'text/csv; charset=iso-8859-1; header=present', 
            :disposition => "attachment; filename=products.csv"

Full code:

#!/usr/bin/ruby

require 'rubygems'
require 'anemone'
require 'fastercsv'

productsArray = Array.new

class Product
    attr_accessor :name, :sku, :desc
end

# Scraper Code

Anemone.crawl("http://retail.pelicanbayltd.com/") do |anemone|
    anemone.on_every_page do |page|

        currentPage = Product.new

        #Product info parsing
        currentPage.name = page.doc.css(".page_headers").text
        currentPage.sku = page.doc.css("tr:nth-child(2) strong").text
        currentPage.desc = page.doc.css("tr:nth-child(4) .item").text

        if currentPage.sku =~ /#\d\d\d\d/
            currentPage.sku = currentPage.sku[1..-1]
            productsArray.push(currentPage)
        end
    end
end

# CSV Export Code

products = productsArray.find(:all) 
csv_data = FasterCSV.generate do |csv| 
    # header row 
    csv << ["sku", "name", "desc"] 

    # data rows 
    productsArray.each do |product| 
      csv << [product.sku, product.name, product.desc] 
    end 
  end 

  send_data csv_data, 
            :type => 'text/csv; charset=iso-8859-1; header=present', 
            :disposition => "attachment; filename=products.csv"

回答1:

File.open('filename.csv', 'w') do |f|
  f.write(csv_data)
end

回答2:

If you are new to Ruby, you should be using Ruby 1.9 or later, in which case you can use the built-in CSV output which builds in fast csv plus l18n support:

require 'csv'
CSV.open('filename.csv', 'w') do |csv|
  csv << [sku, name, desc]
end

http://ruby-doc.org/stdlib-1.9.2/libdoc/csv/rdoc/CSV.html

回答3:

It probably makes more sense to do:

@csv = FasterCSV.open('filename.csv', 'w')

and then write to it as you go along:

@csv << [sku, name, desc]

that way if your script crashes halfway through you've at least got half of the data.

来源：https://stackoverflow.com/questions/10679058/ruby-scraper-how-to-export-to-csv

标签

ruby

fastercsv

scraper

anemone