How do I scrape HTML between two HTML comments using Nokogiri?

I have some HTML pages where the contents to be extracted are marked with HTML comments like below.

<html>
 .....
<!-- begin content -->
 <div>some text</div>
 <div><p>Some more elements</p></div>
<!-- end content -->
...
</html>

I am using Nokogiri and trying to extract the HTML between the  and  comments.

I want to extract the full elements between these two HTML comments:

<div>some text</div>
<div><p>Some more elements</p></div>

I can get the text-only version using this characters callback:

class TextExtractor < Nokogiri::XML::SAX::Document

  def initialize
    @interesting = false
    @text = ""
    @html = ""
  end

  def comment(string)
    case string.strip        # strip leading and trailing whitespaces
    when /^begin content/      # match starting comment
      @interesting = true
    when /^end content/
    @interesting = false   # match closing comment
  end

  def characters(string)
    @text << string if @interesting
  end

end

I get the text-only version with @text but I need the full HTML stored in @html.

Extracting content between two nodes is not a normal thing we'd do; Normally we'd want content inside a particular node. Comments are nodes, they're just special types of nodes.

require 'nokogiri'

doc = Nokogiri::HTML(<<EOT)
<body>
<!-- begin content -->
 <div>some text</div>
 <div><p>Some more elements</p></div>
<!-- end content -->
</body>
EOT

By looking for a comment containing the specified text it's possible to find a starting node:

start_comment = doc.at("//comment()[contains(.,'begin content')]") # => #<Nokogiri::XML::Comment:0x3fe94994268c " begin content ">

Once that's found then a loop is needed that stores the current node, then looks for the next sibling until it finds another comment:

content = Nokogiri::XML::NodeSet.new(doc)
contained_node = start_comment.next_sibling
loop do
  break if contained_node.comment?
  content << contained_node
  contained_node = contained_node.next_sibling
end

content.to_html # => "\n <div>some text</div>\n <div><p>Some more elements</p></div>\n"

来源：https://stackoverflow.com/questions/18869004/how-do-i-scrape-html-between-two-html-comments-using-nokogiri

标签

ruby-on-rails

ruby

web-scraping

web-crawler

nokogiri