How can I use BeautifulSoup or Slimit on a site to output the email address from a javascript variable

问题

I have this example website: http://www.example.com/whatever.asp?profile=1

For each profile number I have a different email in this Java script code.

<script LANGUAGE="JavaScript">
function something()
{
var ptr;
ptr = "";
ptr += "<table><td class=france></td></table>";
ptr += "<table><td class=france><a href=mailto:exa";
ptr += "mple@email.com>email</a></td></table>";
document.all.something.innerHTML = ptr;
}
</script>

I want to parse or regex the email address. The position of emails depends on length. However with this python code only I can parse mple@email.com and not example@email.com

url=urllib.urlopen('http://www.example.com/whatever.asp?profile=1')
contents= url.read()   
soup = BeautifulSoup(contents)
js_content= soup.findAll("script")[0].text
reg = '(<)?(\w+@\w+(?:\.\w+)+)(?(1)>)'
match= re.search(reg,js_content)
print match.group()

Any help? Thanks.

回答1:

I suggest you to use re.findall instead of re.search because search would return only the first match.

url=urllib.urlopen('http://www.example.com/whatever.asp?profile=1')
contents= url.read()   
soup = BeautifulSoup(contents)
js_content= soup.findAll("script")[0].text
reg = r'<?(\w+@\w+(?:\.\w+)+)>?'
match= re.findall(reg,js_content)

回答2:

#!/usr/bin/env python

from bs4 import BeautifulSoup
import re

soup = '''
<script LANGUAGE="JavaScript">
function something()
{
var ptr;
ptr = "";
ptr += "<table><td class=france></td></table>";
ptr += "<table><td class=france><a href=";
ptr += "mailto:example@knesset.com>email</a></td></table>";
document.all.something.innerHTML = ptr;
}
</script>
'''


soup = BeautifulSoup(soup)

for script in soup.find_all('script'):
    reg = '(<)?(\w+@\w+(?:\.\w+)+)(?(1)>)'
    reg2 = 'mailto:.*'
    secondHalf= re.search(reg, script.text)
    firstHalf= re.search(reg2, script.text)
    secondHalfEmail = secondHalf.group()
    firstHalfEmail = firstHalf.group()
    firstHalfEmail = firstHalfEmail.replace('mailto:', '')
    firstHalfEmail = firstHalfEmail.replace('";', '')
    if firstHalfEmail == secondHalfEmail:
        email = secondHalfEmail
    else:
        if ('>') not in firstHalfEmail:
            if ('>') not in secondHalfEmail:
                if firstHalfEmail != secondHalfEmail:
                    email = firstHalfEmail + secondHalfEmail
            else:
                email = firstHalfEmail
        else:
            email = secondHalfEmail

    print email

来源：https://stackoverflow.com/questions/27682751/how-can-i-use-beautifulsoup-or-slimit-on-a-site-to-output-the-email-address-from

标签

javascript

python

parsing

beautifulsoup