extract unidentified html content from between two tags, using jsoup? regex?

问题

I want to get the names of all those links from between the two h2 tags there

<h2><span class="mw-headline" id="People">People</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Bush&amp;action=edit&amp;section=1" title="Edit section: People">edit</a><span class="mw-editsection-bracket">]</span></span></h2>
<ul>
<li><a href="/wiki/George_H._W._Bush" title="George H. W. Bush">George H. W. Bush</a> (born 1924), the 41st president of the United States of America</li>
<li><a href="/wiki/George_W._Bush" title="George W. Bush">George W. Bush</a> (born 1946), the 43rd president of the United States of America</li>
<li><a href="/wiki/Jeb_Bush" title="Jeb Bush">Jeb Bush</a> (born 1953), the former governor of Florida and also a member of the Bush family</li>
<li><a href="/wiki/Bush_family" title="Bush family">Bush family</a>, the political family that includes both presidents</li>
<li><a href="/wiki/Bush_(surname)" title="Bush (surname)">Bush (surname)</a>, a surname (including a list of people with the name)    </li>
</ul>
<h2><span class="mw-headline" id="Places.2C_United_States">Places, United States</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Bush&amp;action=edit&amp;section=2" title="Edit section: Places, United States">edit</a><span class="mw-editsection-bracket">]</span></span></h2>

neither this

    Elements h2next = docx.select("span.mw-headline#People");
    do 
    {
     ul = h2next.select("ul").first();
     System.out.println(ul.text());
    } 
    while (h2next!=null && ul==null);

nor

    //String content = docx.getElementById("People").outerHtml();

works.

It seems like this guy, has the right idea, but I can't make it adapt to my situation.

Maybe I should just use regex?

Seems wikipedia html is kind of "unstructured" and hard to work with.

From the wikipedia disambiguation page I want to grab the different senses in which Bush (or whatever ambiguous name I'm considering) could be used as a person.

I've tried all kinds of ways to grab this data using jsoup but I've not been able to figure it out.

I tried this:

Document docx = Jsoup.connect("https://en.wikipedia.org/wiki/Bush").get();
Element contentDiv = docx.select("span#mw-headlinePeople").first();
String printMe = contentDiv.toString(); // The result

Since I noticed that the data I want lives in a partition named:

 <h2><span class="mw-headline" id="People">

But that output nothing.

I tried some variation on that based on previous questions like this one:

.select("span#mw-headlinePeople");

but still nothing.

How to get at that info?

Ideally, what I'd like is somehting like this:

George H. W. Bush 
George W. Bush 
Jeb Bush

Though I know I'll probably initially also have to get Bush family and Bush (surname) since they're part of that segment, but I guess I can just remove them later.

Also, is it faster to use this:

Document docx = Jsoup.connect("https://en.wikipedia.org/wiki/Bush").get();

or this:

    URL site_two = new URL("https://en.wikipedia.org/wiki/Bush");

    URLConnection ycb = site_two.openConnection();
    BufferedReader inb = new BufferedReader(
                            new InputStreamReader(
                            ycb.getInputStream()));

    StringBuilder sb = new StringBuilder();

    while ((inputLine = inb.readLine()) != null) 
    {
        //get the disambig
        //System.out.println(inputLine);

        sb.append(inputLine);
        sb.append(System.lineSeparator());
        inputLine = inb.readLine();
    }

I tried using this site, but it turns out to be not very useful. Someone should make a jsoup site like all those regex sites.

回答1:

One possible way is to select both all headlines (span.mw-headlines) and all links (best selector I found wasli > a).

If you select both with one selector by combining them with a ,, they will be in the order they appear on the page. Therefore you can keep track of whether you are in a "People section" or not while looping through the results like this:

Elements elements = docx.select("span.mw-headline, li > a");

boolean inPeopleSection = false;
for (Element elem : elements) {
    if (elem.className().equals("mw-headline")) {
        // It's a headline
        inPeopleSection = elem.id().equals("People");
    } else {
        // It's a link
        if (inPeopleSection) {
            System.out.println(elem.text());
        }
    }
}

Output:

George H. W. Bush
George W. Bush
Jeb Bush
Bush family
Bush (surname)

Regarding the performance, I wouldn't think it makes any difference at all, just go with the simpler version (Although I have very limited Jsoup experience, so don't take my word for it).

回答2:

A simple selector would be h2:contains(people) + ul a, e.g.:

Elements els = doc.select("h2:contains(people) + ul a");

Which gives these elements:

0 <a href="/wiki/George_H._W._Bush" title="George H. W. Bush">
George H. W. Bush
1 <a href="/wiki/George_W._Bush" title="George W. Bush">
George W. Bush
2 <a href="/wiki/Jeb_Bush" title="Jeb Bush">
Jeb Bush
3 <a href="/wiki/Bush_family" title="Bush family">
Bush family
4 <a href="/wiki/Bush_(surname)" title="Bush (surname)">
Bush (surname)

I used try.jsoup.org (see working example) and the selector syntax guide as resources.

来源：https://stackoverflow.com/questions/29794099/extract-unidentified-html-content-from-between-two-tags-using-jsoup-regex

标签

java

html

parsing

jsoup

wikipedia