BeautifulSoup (bs4) parsing wrong

问题

Parsing this sample document with bs4, from python 2.7.6:

<html>
<body>
<p>HTML allows omitting P end-tags.

<p>Like that and this.

<p>And this, too.

<p>What happened?</p>

<p>And can we <p>nest a paragraph, too?</p></p>

</body>
</html>

Using:

from bs4 import BeautifulSoup as BS
...
tree = BS(fh)

HTML has, for ages, allowed omitted end-tags for various element types, including P (check the schema, or a parser). However, bs4's prettify() on this document shows that it doesn't end any of those paragraphs until it sees </body>:

<html>
 <body>
  <p>
   HTML allows omitting P end-tags.
   <p>
    Like that and this.
    <p>
     And this, too.
     <p>
      What happened?
     </p>
     <p>
      And can we
      <p>
       nest a paragraph, too?
      </p>
     </p>
    </p>
   </p>
  </p>
 </body>

It's not prettify()'s fault, because traversing the tree manually I get the same structure:

<[document]>
    <html>
        ␊
        <body>
            ␊
            <p>
                HTML allows omitting P end-tags.␊␊
                <p>
                    Like that and this.␊␊
                    <p>
                        And this, too.␊␊
                        <p>
                            What happened?
                        </p>
                        ␊
                        <p>
                            And can we 
                            <p>
                                nest a paragraph, too?
                            </p>
                        </p>
                        ␊
                    </p>
                </p>
            </p>
        </body>
        ␊
    </html>
    ␊
</[document]>

Now, this would be the right result for XML (at least up to </body>, at which point it should report a WF error). But this ain't XML. What gives?

回答1:

The doc at http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser tells how to get BS4 to use different parsers. Apparently the default is html.parse, which the BS4 doc says is broken before Python 2.7.3, but apparently still has the problem described above in 2.7.6.

Switching to "lxml" was unsuccessful for me, but switching to "html5lib" produces the correct result:

tree = BS(htmSource, "html5lib")

来源：https://stackoverflow.com/questions/29954170/beautifulsoup-bs4-parsing-wrong

标签

python

html

python-2.7

bs4