Beautifulsoup, maximum recursion depth reached

Question

This is a beautifulsoup procedure that grabs content within all

html tags. After grabbing content from some web pages, I get an error th

Answer 1

I had the same problem. If you have nested tags with a depth of about 480 levels, and you want to convert this tag to string/unicode, you get the RuntimeError maximum recursion depth reached. Every level needs two nested method calls and soon you hit the default of 1000 nested python calls. You can raise this level, or you can use this helper. It extracts all text from the html and displays it in a pre-environment:

def beautiful_soup_tag_to_unicode(tag):
    try:
        return unicode(tag)
    except RuntimeError as e:
        if not str(e).startswith('maximum recursion'):
            raise
        # If you have more than 480 level of nested tags you can hit the maximum recursion level
        out=[]
        for mystring in tag.findAll(text=True):
            mystring=mystring.strip()
            if not mystring:
                continue
            out.append(mystring)
        return u'
%s
' % '\n'.join(out)